New Efficient and Practicable Adaptive Designs for Calibrating Items Online

Abstract

When calibrating new items online, it is practicable to first compare all new items according to some criterion and then assign the most suitable one to the current examinee who reaches a seeding location. The modified D-optimal design proposed by van der Linden and Ren (denoted as D-VR design) works within this practicable framework with the aim of directly optimizing the estimation of item parameters. However, the optimal design point for a given new item should be obtained by comparing all examinees in a static examinee pool. Thus, D-VR design still has room for improvement in calibration efficiency from the view of traditional optimal design. To this end, this article incorporates the idea of traditional optimal design into D-VR design and proposes a new online calibration design criterion, namely, excellence degree (ED) criterion. Four different schemes are developed to measure the information provided by the current examinee when implementing this new criterion, and four new ED designs equipped with them are put forward accordingly. Simulation studies were conducted under a variety of conditions to compare the D-VR design and the four proposed ED designs in terms of calibration efficiency. Results showed that the four ED designs outperformed D-VR design in almost all simulation conditions.

Introduction

Computerized adaptive testing (CAT) is an efficient testing mode that allows each examinee to answer appropriate test questions on the basis of his or her latent trait level (Lord, 1968). Compared with the traditional paper-pencil (P&P) test, CAT possesses several virtues such as immediate score reporting, flexible testing time, and more accurate ability estimates (e.g., Wang & Chang, 2011; Weiss, 1982). Therefore, many large-scale educational and psychological assessment programs such as the Graduate Management Admission Test (GMAT) and the Armed Services Vocational Aptitude Battery (ASVAB) have already been implemented via CAT (e.g., H.-H. Chang & Ying, 2009).

Because in CAT each examinee is evaluated by an optimal and individualized set of test items that tailored to his or her provisional ability, the attrition speed of operational items in the item bank is faster than that in P&P test. For test reliability, fairness, and security reasons, the low-quality, overexposed, or obsolete items should be retired from item bank periodically. On the other hand, new items should be developed to substitute the unsuitable ones to keep the usability and vitality of the item bank. Moreover, it is of great importance to precisely calibrate the new items, because the inaccuracy of item parameter estimates may result in biased ability estimates and underestimate the standard errors of ability estimates in the ensuing scoring process (e.g., Cheng & Yuan, 2010; Mislevy, Wingersky, & Sheehan, 1994; Tsutakawa & Johnson, 1990; Yang, Hansen, & Cai, 2012; Zhang, Xie, Song, & Lu, 2011).

Typically, new items are calibrated by administering them to a batch of volunteers that are recruited expressly for item calibration using the form of P&P test. However, this calibration approach is costly and the obtained calibration results may be unreliable to a certain extent, because the test takers know that their responses on the new items are unrelated to their final scores and may not take it seriously as a result. The other promising and widely used technique to calibrate new items is online calibration, in which the item responses are collected by randomly or adaptively assigning new items to examinees during the course of their testing with operational items (e.g., Stocking, 1988; Wainer & Mislevy, 1990). Compared with the traditional offline calibration approach, online calibration is typically applied in CAT scenario, and it is time-saving and cost-effective and collects more reliable data. Moreover, online calibration is capable of automatically placing new items on the same scale as the operational items without post hoc linking/scaling, thereby providing an effective solution to the complex linking/equating issues in the construction of a large-scale item bank (e.g., Chen, Xin, Wang, & Chang, 2012).

How to select a batch of examinees that provides the best estimate for a given new item is a matter of great concern. Many researchers have made efforts to directly apply some of the traditional experimental designs to calibrate new items online. For example, Jones and Jin (1994) used the D-optimal criterion to select individual examinees for a given new item and sequentially updated the item parameters once a subdesign is obtained; Y. C. I. Chang and Lu (2010) suggested using a two-stage D-optimal design to calibrate the two-parameter logistic (2PL; Birnbaum, 1968) model online under the variable-length CAT scenario; Lu (2014) calibrated the 2PL model online using the conclusions derived by the D-, A-, E-optimality. It is important to note that a key implementation element of the abovementioned designs is that the examinee pool should be static, from which examinees with suitable ability values will be selected to calibrate new items. However, in the context of online calibration, the examinee pool is typically not static because examinees are allowed to leave immediately after their testing sessions. In this regard, it is hardly feasible to directly apply traditional optimal designs to online calibration scenario.

To obtain a batch of examinees for calibrating new items online, a practical way is to administer new items via the following two steps: (a) compare a finite set of new items that need to be calibrated and pick out the most suitable one according to some criterion; and (b) assign it to the current active examinee when he or she reaches a seeding location (i.e., the location where new items are assigned to examinees during the adaptive testing session). Several online calibration designs are achieved via this practicable framework. For instance, the random design randomly selects a new item and then stochastically seeds it in the current examinee’s adaptive test (Wainer & Mislevy, 1990). Although it is simple and convenient to implement, the randomly selected new items may be either too difficult or too easy compared with examinees’ abilities and accordingly may make the new items be easily discerned from the operational items (Jones & Jin, 1994; Kingsbury, 2009). To address this issue, Chen et al. (2012) and Kingsbury (2009) proposed administering new items in an adaptive fashion by using the same item selection strategy that used for selecting operational items. Nevertheless, this design, classified into the category of “examinee-centered adaptive design” by Zheng (2014), may not be efficient for calibrating new items, because the primary target of item selection algorithm is not maximizing the calibration precision of new items, but maximizing the estimation accuracy of abilities. In contrast, the adapted versions of D-, A-, E-, c-optimality proposed by van der Linden and Ren (2015) can be categorized into “item-centered adaptive designs” because their primary objectives are to optimize the estimation of item parameters. Among them, the modified D-optimality concurrently considers all item parameters, thus it is selected as the object of this study. For ease of description, the modified D-optimal design proposed by van der Linden and Ren (2015) is referred to as D-VR design henceforth.

From the implementation process of D-VR design, one can note that if the new item $k$ is picked out from the set of new items and administered to an active examinee $p$ at a seeding location, it implies that examinee $p$ is more helpful in accurately calibrating new item $k$ than in calibrating any other new items. Note that in traditional experimental designs, all examinees need to compete to give the best estimate for a given new item. Hence, from the perspective of traditional optimal experimental design, the optimal design point of new item $k$ should be derived by comparing all examinees in the entire examinee pool. In this case, the current active examinee $p$ may not be the optimal examinee for calibrating new item $k$, implying that there is still room for improvement for the D-VR design.

To improve the calibration efficiency of D-VR design, this article incorporates the idea of traditional optimal design into D-VR design and proposes a new design criterion (denoted as excellence degree [ED] criterion) for calibrating new item online. The ED criterion measures the excellence degree of the current active examinee relative to the optimal examinee when calibrating each new item. Besides, four different schemes are put forward to measure the information provided by the current examinee. On that basis, four new ED designs (i.e., ED-o, ED-min, ED-mean, and ED-lw) are proposed by equipping ED criterion with the four schemes. Moreover, the whole procedures for assigning new items online are divided into random and adaptive calibration stages, where the item parameters of new item are updated sequentially in the adaptive calibration stage.

The reminder of this article is structured as follows. First, the original D-optimal design and D-VR design are briefly summarized after some pre-knowledge. Then, the newly proposed online calibration designs are introduced in detail, followed by the elaboration on the four schemes to deal with the uncertainty in ability estimates. Next, the simulation studies regarding the full comparison of the four ED designs with the D-VR design under a variety of simulation scenarios are presented. Finally, this article ends with discussion and suggestions for further research.

Method

Pre-Knowledge

For the 2PL model, let ${\theta }_i$ represent the latent trait of examinee $i$, $y_{\mathrm{ij}}$ denote his or her response to a dichotomously scored item $j$, and $P_j({\theta }_i)$ represent the probability of a correct response, that is

$$P_j({\theta }_i)\equiv P(y_{\mathrm{ij}}=1|{\theta }_i,a_j,b_j)=\frac{1}{1+\exp (-a_j({\theta }_i-b_j))},$$ (1)

where $a_j$ and $b_j$ are the discrimination parameter and difficulty parameter for item $j$, respectively.

Item calibration aims to estimate the item parameters $a_j$ and $b_j$ based on a sample of collected response data. Let $m$ be the number of new items that need to be calibrated, and ${\mathbf{\eta }}_{\mathbf{k}}=(a_k,b_k{)}^T$ denotes the item parameter vector of new item $k$ ($k=1,2,\cdots ,m$). Suppose the $k$th new item has already been answered by $n_k$ examinees with ability vector ${\overrightarrow{\mathbf{\theta }}}_{n_k}=({\theta }_1,{\theta }_2,\cdots ,{\theta }_{n_k}{)}^T$. Using the terminology in experimental design, the desired ability values are termed as design points, and all design points form the design space (denoted as $\mathbf{\Theta }$).

In addition, let $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)$ represent the Fisher information of ${\mathbf{\eta }}_{\mathbf{k}}$ that provided by the $i$th examinee who answers the $k$th new item. Then, the Fisher information of ${\mathbf{\eta }}_{\mathbf{k}}$ provided by the total $n_k$ examinees [referred to as $I({\mathbf{\eta }}_{\mathbf{k}};{\overrightarrow{\mathbf{\theta }}}_{n_k})$] is exactly the sum of those provided by each individual examinee under local independence assumption, that is, $I({\mathbf{\eta }}_{\mathbf{k}};{\overrightarrow{\mathbf{\theta }}}_{n_k})=\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)$. For the 2PL model, $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)$ can be concretely expressed as $\left(\begin{array}{cc}{I}_{\mathrm{ik}}^{\mathrm{aa}}\hfill & I_{\mathrm{ik}}^{\mathrm{ab}}\hfill \\ \multicolumn{1}{c}{I_{\mathrm{ik}}^{\mathrm{ab}}}& I_{\mathrm{ik}}^{\mathrm{bb}}\hfill \end{array}\right)$, where $I_{\mathrm{ik}}^{\mathrm{aa}}=({\theta }_i-b_k{)}^2{P}_k({\theta }_i)(1-P_k({\theta }_i))$, $I_{\mathrm{ik}}^{\mathrm{ab}}=-a_k({\theta }_i-b_k)P_k({\theta }_i)(1-P_k({\theta }_i))$, and $I_{\mathrm{ik}}^{\mathrm{bb}}=a_k^2{P}_k({\theta }_i)(1-P_k({\theta }_i))$.

D-Optimal Design and D-VR Design

One of the commonly used criteria in the literature of optimal design is the D-optimality. The D-optimal design is executed by minimizing the generalized variance of item parameter estimates, or equivalently by maximizing the determinant of the Fisher information matrix (Anderson, 1984). In addition, as indicated by Silvey (1980), the D-optimal design is invariant against the linear transformation of item parameters; thus, it is particularly favored by experiment designers. Formally, the D-optimal design for calibrating the $k$th new item can be expressed as

$${\theta }_{k_o}=\arg \underset{\theta }{\max }\left\{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};\theta )]\right\},\theta \in \mathbf{\Theta }.$$ (2)

As seen from Formula 2, the optimal examinee $k_o$ with ability ${\theta }_{k_o}$ for calibrating new item $k$ is derived by comparing all the examinees in a static examinee pool (i.e., $\mathbf{\Theta }$). However, as alluded to above, the examinee pool is not static in the online calibration scenario, because adaptive testing is continuously administered at frequent time intervals and allows examinees to leave if they have finished their tests. Thus, van der Linden and Ren (2015) proposed assigning a new item to the current active examinee by comparing the expected contribution of the current examinee across all new items when he or she reaches a seeding location. Accordingly, the D-optimal design is adjusted as (i.e., the D-VR design)

$$\arg \underset{k}{\max }\{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)]\},k\in \mathcal{H},$$ (3)

where ${\theta }_p$ is the ability of the current active examinee $p$, and $\mathcal{H}=\{k|1\le k\le m\}$ is the ID set of new items that need to be calibrated.

The D-VR design has been proved to be feasible and practical in the context of online calibration (Chen, 2017), primarily because it does not require the examinee pool to be static when assigning new items to examinees. Regarding the full Bayesian version of D-VR design, interested readers are referred to van der Linden and Ren (2015).

New Designs for Online Calibration

Derive virtual optimal design point

Suppose new item $k$ is selected to be assigned to the current active examinee $p$ based on Formula 3, it implies that the $p$th examinee and the $k$th new item from the set of new items are the best matched according to D-VR design. However, from the view of entire design space, the optimal examinee $k_o$ for calibrating new item $k$ should be derived according to Formula 2.

Note that although the examinee pool is dynamic in adaptive testing scenario, all examinees’ abilities are typically truncated by an interval $[C_L,C_U]$ in practice (e.g., Baker, 2001; Y. C. I. Chang & Lu, 2010). Thus, the ability of the optimal examinee $k_o$ can be obtained via the following formula:

$${\theta }_{k_o}=\arg \underset{\theta }{\max }\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};\theta )],\theta \in [C_L,C_U].$$ (4)

To derive ${\theta }_{k_o}$, one feasible way is to randomly discretize the interval $[C_L,C_U]$. The idea of discretizing parameter space is widely used in experimental designs, and similar treatments can be found in the studies of Berger and Tan (2004) and Zhu and Stein (2005). It is noteworthy that examinee $k_o$ does not necessarily correspond to a specific examinee in the examinee pool, and it is more like a virtual examinee. In the infinite case, when the number of examinees in the examinee pool goes to infinity, each point in the interval $[C_L,C_U]$ will correspond to a specific examinee in the examinee pool, and $k_o$ will become a real examinee as a result.

New design criterion for online calibration

Based on the derived optimal ability ${\theta }_{k_o}$ and the ability vector ${\overrightarrow{\mathbf{\theta }}}_{n_k}=({\theta }_1,{\theta }_2,\cdots ,{\theta }_{n_k}{)}^T$ that has already been obtained, the contribution of the virtual optimal examinee $k_o$ to calibrating new item $k$ can be measured according to the D-VR design criterion, that is,

$$\mathrm{INF}({\mathbf{\eta }}_{\mathbf{k}},{\theta }_{k_o})=\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_{k_o})]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)].$$ (5)

Similarly, the contribution of current examinee $p$ to the calibration of new item $k$ is obtained as

$$\mathrm{INF}({\mathbf{\eta }}_{\mathbf{k}},{\theta }_p)=\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)].$$ (6)

Accordingly, when the current active examinee $p$ reaches a seeding location, his or her contribution relative to the maximum contribution can be evaluated as

$$R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})=\frac{\mathrm{INF}({\mathbf{\eta }}_{\mathbf{k}},{\theta }_p)}{\mathrm{INF}({\mathbf{\eta }}_{\mathbf{k}},{\theta }_{k_o})}=\frac{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)]}{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_{k_o})]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)]}.$$ (7)

The value of $R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})$ varies from 0 to 1, and larger value indicates that the contribution provided by the current examinee $p$ is closer to that generated by the virtual optimal examinee $k_o$. Thus, by taking the virtual optimal examinee $k_o$ as a benchmark, $R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})$ can be used to measure the excellence degree of examinee $p$ for calibrating new item $k$. For the convenience of description, $R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})$ is denoted as “excellence degree (ED) criterion” in this article. Furthermore, the corresponding online calibration design is referred to as ED design and can be formally expressed as

$$\arg \underset{k}{\max }\{R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})=\frac{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)]}{\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)+I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_{k_o})]-\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_i)]}\},k\in \mathcal{H}.$$ (8)

More specifically, when the current examinee $p$ reaches a seeding location, the virtual ability ${\theta }_{k_0}$ is first calculated for each new item $k$ ($k\in \mathcal{H}$) according to Formula 4, then $R_{{\mathbf{\eta }}_{\mathbf{k}}}({\theta }_p;{\theta }_{k_0})$ is computed for each new item based on Formula 7, finally the new item with the largest ED criterion value will be assigned to the examinee $p$.

Dealing with the uncertainty in ability estimates

In reality, the true abilities in Formula 8 are never known, thus ability estimates instead of true abilities will be used to calibrate new items. Considering that the ability of the current active examinee $p$ may not be accurately estimated due to relatively short test length of operational items at some seeding locations, this article uses four different schemes to measure the Fisher information provided by the current examinee $p$, that is, $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$.

Scheme 1

This scheme is used as a baseline for comparison, because it simply uses the estimated ability ${\hat{\theta }}_p$ to calculate $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$.

Scheme 2

In this scheme, examinee $p$’s ability is represented by the worst case ${\hat{\theta }}_p^k$ in $[{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p]$, where ${\delta }_p>0$ is a small neighborhood of the ability estimate ${\hat{\theta }}_p$. It should be explained that the worst case ${\hat{\theta }}_p^k$ refers to the ability point from interval $[{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p]$ that provides the least Fisher information for calibrating new item $k$, which is given by

$${\hat{\theta }}_p^k=\arg \underset{x}{min}\det [\sum _{i=1}^{n_k}I({\mathbf{\eta }}_{\mathbf{k}};{\hat{\theta }}_i)+I({\mathbf{\eta }}_{\mathbf{k}};x)],x\in [{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p].$$ (9)

Then, the Fisher information $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$ is calculated as $I({\mathbf{\eta }}_{\mathbf{k}};{\hat{\theta }}_p^k)$.

The small neighborhood ${\delta }_p$ can be set to be the standard error of ${\hat{\theta }}_p$, whose squared value approximately equals the inverse of test information evaluated at ${\hat{\theta }}_p$, that is, $1/I({\hat{\theta }}_p)$, where $I({\hat{\theta }}_p)=\sum _{j=1}^{L_p}\left[\frac{{(P{\prime }_j({\hat{\theta }}_p))}^2}{P_j({\hat{\theta }}_p)\{1-P_j({\hat{\theta }}_p)\}}\right]$, $L_p$ is the test length of operational items, and $P{\prime }_j({\hat{\theta }}_p)=\frac{d{P}_j({\mathrm{\theta }}_p)}{d{\mathrm{\theta }}_p}{|}_{({\mathrm{\theta }}_p={\hat{\theta }}_p)}$ is the derivative with respect to ability parameter.

Scheme 3

Different from Scheme 2, which uses the worst case in the neighborhood of ${\hat{\theta }}_p$, Scheme 3 replaces $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$ by the expected Fisher information $E[I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]$. That is,

$$E[I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]\approx \frac{1}{T}\sum _{t=1}^{T}I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p^{(t)}),$$ (10)

where ${\theta }_p^{(t)}$ ($t=1,2,\cdots ,T$) are randomly drawn from the interval $[{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p]$, assuming that true ability ${\theta }_p$ is equally distributed in a small neighborhood of ${\hat{\theta }}_p$.

In addition, if a reasonable and proper prior distribution $f({\theta }_p)$ for ${\theta }_p$ is available, the posterior distribution $f({\theta }_p|{\overrightarrow{\mathit{v}}}_{L_p})$ can be obtained as

$$f({\theta }_p|{\overrightarrow{\mathit{v}}}_{L_p})=\frac{f\left({\theta }_p\right)\underset{j=1}{\overset{L_p}{\Pi }}\left[{(P_j({\theta }_p))}^{v_k}{(1-P_j({\theta }_p))}^{1-v_k}\right]}{{\int }_{-\infty }^{+\infty }f\left({\theta }_p\right)\left\{\underset{j=1}{\overset{L_p}{\Pi }}\left[{(P_j({\theta }_p))}^{v_k}{(1-P_j({\theta }_p))}^{1-v_k}\right]\right\}d{\theta }_p},$$ (11)

where ${\overrightarrow{\mathit{v}}}_{L_p}=(v_1,v_2,\cdots ,v_{L_p}{)}^T$ is the response vector of examinee $p$ to $L_p$ operational items. Then, $E[I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)]$ can also be approximated by using ${\theta }_p^{(t)}$ ($t=1,2,\cdots ,T$) that are randomly sampled from the posterior distribution $f({\theta }_p|{\overrightarrow{\mathit{v}}}_{L_p})$.

Scheme 4

In this scheme, $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$ is weighted by the likelihood of response vector ${\overrightarrow{\mathit{v}}}_{L_p}$ (denoted as $L({\overrightarrow{\mathit{v}}}_{L_p};{\theta }_p)$), and then the unknown ability parameter is integrated out of the weighted information matrix, that is

$$E[I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)L({\overrightarrow{\mathit{v}}}_{L_p};{\theta }_p)]=\int I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)\underset{j=1}{\overset{L_p}{\Pi }}\left[{(P_j({\theta }_p))}^{v_k}{(1-P_j({\theta }_p))}^{1-v_k}\right]d{\theta }_p.$$ (12)

In summary, Scheme 1 is relatively simple, and Scheme 2 is conservative due to its use of the worst case in the interval, which is to some extent in the same fashion as the minimax optimal design (e.g., Berger, King, & Wong, 2000; King & Wong, 2000). Compared with Scheme 2, both Scheme 3 and Scheme 4 are less sensitive to the extreme points and are more robust, because they evaluate $I({\mathbf{\eta }}_{\mathbf{k}};{\theta }_p)$ by integrating ${\theta }_p$ out. Besides, Scheme 4 also fully utilizes auxiliary information from the operational items. Because Schemes 1 through 4, respectively, use the original estimated information (o), the minimum information (min), the mean information (mean), and the likelihood-weighted information (lw) to measure the information provided by the current examinee, the ED designs equipped with the four schemes are accordingly denoted as ED-o, ED-min, ED-mean, and ED-lw designs for short.

Dealing with the dependence on unknown item parameters

Another noteworthy point is that the implementations of adaptive designs (i.e., D-VR, ED-o, ED-min, ED-mean, and ED-lw) also depend on the unknown item parameters of new items. To handle this kind of dependency, the idea of sequential design is used. In sequential designs, experiments are designed stage by stage, and the item parameter estimates obtained from the current stage will be used as the initial values in the next stage until the predefined sample size is reached or the item parameter estimates are precise enough.

Sequential designs start with initial item parameter values, which can be estimated by item writers based on subjective judgment (Wainer & Mislevy, 1990) or using data-based methods (e.g., Chen, Wang, Xin, & Chang, 2017; Chen et al., 2012). To be on the safe side, the initial item parameter values are determined based on a batch of examines’ responses in this article. Thus, the entire process of online calibration can be divided into random calibration stage and adaptive calibration stage as in the study of Chen et al. (2012). More specifically, in the random calibration stage, new items are randomly administered to a subgroup of test takers, and the initial values are estimated based on the collected data; then in the adaptive calibration stage, new items are assigned to examinees by using different adaptive design criteria and the item parameters of each new item are sequentially updated once it receives a fixed number of new responses.

For simplicity, each item is retired from the whole calibration process when it receives a fixed total number of $N_T$ responses. The detailed procedures for administering new items online are sketched in the online Appendix A.

Simulation Studies

To fully evaluate the calibration efficiency of different adaptive designs relative to the random design, simulation studies were conducted under various conditions. In addition, the relative efficiency is calculated via the following formula:

$$\frac{1}{\mathrm{rep}*m}\sum _{r=1}^{\mathrm{rep}}\left\{\sum _{j=1}^m\left[\frac{\det \left[\sum _{i=1}^{N_T}I({\mathbf{\eta }}_{\mathbf{j}}^{\mathbf{(}\mathbf{r}\mathbf{)}}\mathbf{;}{\mathbf{\theta }}_{\mathbf{i}}^{\mathbf{(}{\mathbf{j}}_{\mathbf{A}}\mathbf{r}\mathbf{)}})\right]}{\det \left[\sum _{i=1}^{N_T}I({\mathbf{\eta }}_{\mathbf{j}}^{\mathbf{(}\mathbf{r}\mathbf{)}}\mathbf{;}{\mathbf{\theta }}_{\mathbf{i}}^{\mathbf{(}{\mathbf{j}}_{\mathbf{R}}\mathbf{r}\mathbf{)}})\right]}\right]\right\},$$ (13)

where $\mathrm{rep}$ is the total number of replications, ${\mathbf{\eta }}_{\mathbf{j}}^{\mathbf{(}\mathbf{r}\mathbf{)}}$ is the parameter vector of the $j$th new item simulated in the $r$th replication. ${\theta }_i^{(j_{A}r)}$ and ${\theta }_i^{(j_{R}r)}$, respectively, denote the abilities of the $i$th examinee selected by an adaptive design (e.g., D-VR) and the random design for new item $j$ in the $r$th replication. To reduce random errors, the entire simulation process is replicated 100 times (i.e., $\mathrm{rep}$ = 100).

General Setup

For each examinee, his or her true ability was randomly drawn from the standard normal distribution. Because parameter $a$ is typically supposed to follow a lognormal distribution (Baker & Kim, 2004) and parameters $a$ and $b$ are often truncated by intervals in practice (Lord & Wingersky, 1984), the parameter vector $\mathbf{\gamma }=(\log (a),b{)}^T$ was randomly drawn from a truncated bivariate normal distribution with $a$ constrained by 0 and 2 and $b$ truncated between −3 and 3. Moreover, the correlation coefficient between the simulated $a$ and $b$ was set to be 0.25 to mimic realistic scenario, in which parameters $a$ and $b$ are positively related to a certain extent (e.g., H.-H. Chang, Qian, & Ying, 2001). In addition, 1,000 operational items were generated to construct the CAT item bank and a total number of 20 new items (i.e., $m$= 20) were simulated in the same way as the operational items.

Maximum Fisher information (MFI; Lord, 1980) was used as the item selection strategy in CAT, and each test taker’s adaptive test was terminated if he or she finished a fixed number of 30 operational items. Besides, ability was estimated via the expected a posteriori (EAP) method if the current test length of operational items is less than 5 or the response patterns are nonmixed (i.e., all incorrect or all correct); otherwise, ability was updated by using the weighted likelihood estimation method (Firth, 1993; Warm, 1989). What’s more, the total number of new items that each examinee $p$ received was set to be less than 5.

The interval $[C_L,C_U]$ in Formula 4 was chosen as [−3, 3] and the optimal ability ${\theta }_{k_o}$ was derived by comparing a finite of 100 samples that randomly drawn from this interval. To obtain the worst case ${\hat{\theta }}_p^k$ in Scheme 2, a batch of 100 samples were randomly drawn from $[{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p]$, where ${\delta }_p$ was set to be 1/$\sqrt{I({\hat{\theta }}_p)}$ and 3/$\sqrt{I({\hat{\theta }}_p)}$ for the purpose of examining the impact of ${\delta }_p$ on the performance of ED-min design. In like fashion, to explore the robustness of ED-mean design against ${\delta }_p$, the expected information $E[I({\mathbf{\eta }}_{\mathbf{k}};\theta )]$ in Scheme 3 was also calculated by using samples randomly drawn from $[{\hat{\theta }}_p-{\delta }_p,{\hat{\theta }}_p+{\delta }_p]$ under the abovementioned two levels of ${\delta }_p$. Other highly accurate and efficient ways for approaching $E[I({\mathbf{\eta }}_{\mathbf{k}};\theta )]$ may be explored by using posterior distribution in future studies. In addition, the bounds of integral were chosen as −4 and +4 in Scheme 4.

The “one expectation maximization (EM) cycle” (OEM; Wainer & Mislevy, 1990) method was selected to update item parameters in the adaptive calibration stage (see Step 3b in online Appendix A), because OEM works well in terms of item parameter recovery and is unaffected by the measurement errors contained in ability estimates by integrating the latent abilities out in the M step (e.g., Chen & Wang, 2016). The reason why the “multiple EM cycles” (MEM) method that performs better than OEM (e.g., Ban, Hanson, Wang, Yi, & Harris, 2001) was not used is that MEM is much more computationally intensive, which may not be suitable to be frequently used to update item parameter estimates. For other calibration methods, please refer to Chen and Wang (2016), Kim (2006), and Stocking (1988).

Results

Comparison Among Different Adaptive Designs

Summarized in Table 1 are the relative efficiencies of different adaptive designs under three levels of $N_T$ ($N_T$ = 200, 400, and 600) and three levels of seeding location (Front, Middle, and Rear). Note that the Front, Middle, and Rear, respectively, denote that the seeding locations were randomly drawn from $w_p$ out of the first ($w_p$+5), the middle ($w_p$+6), and the last ($w_p$+5) positions in a test of (30+$w_p$) items, where $w_p$ ($w_p\le$ 5) is the number of new items that examinee $p$ will receive. Moreover, both the ED-min and the ED-mean designs were conducted under two levels of ${\delta }_p$ (i.e., 1 SE and 3 SE of ability estimates). In addition, the online calibration procedures depicted in online Appendix A was implemented under $n_r$ = 100 and $n_b$ = 30, where $n_r$ is the total number of examinees participating in the random calibration stage, and $n_b$ is the number of newly received responses for each new item to update its item parameters during the adaptive calibration stage.

Table 1.

Relative Efficiencies of D-VR and Four ED Designs Under Different Seeding Locations.

$N_T$	Sed.	D-VR	ED-o	ED-min (1 SE)	ED-min (3 SE)	ED-mean (1 SE)	ED-mean (3 SE)	ED-lw
200	Front	1.0206	1.0379	1.0686	1.0386	1.0320	1.0572	1.0901
	Middle	1.0374	1.1066	1.1042	1.0196	1.1134	1.0873	1.1093
	Rear	1.0392	1.1108	1.1089	1.0183	1.1169	1.0838	1.1100
400	Front	1.0102	1.0331	1.0731	1.0389	1.0287	1.0590	1.0944
	Middle	1.0255	1.1166	1.1134	1.0222	1.1254	1.0915	1.1179
	Rear	1.0253	1.1188	1.1178	1.0196	1.1314	1.0900	1.1171
600	Front	1.0066	1.0308	1.0729	1.0352	1.0266	1.0606	1.0966
	Middle	1.0168	1.1230	1.1176	1.0209	1.1301	1.0930	1.1220
	Rear	1.0170	1.1254	1.1195	1.0170	1.1340	1.0903	1.1217

Note. One SE and 3 SE represent that ${\delta }_p$ is obtained as one and three standard error(s) of ability estimate, respectively. The results for conditions where the ED-min, ED-mean, and ED-lw designs are more efficient than the ED-o design are in boldface. VR = van der Linden and Ren; ED = excellence degree; Sed. = Seeding location.

As shown in Table 1, D-VR design was less efficient than ED designs under almost all conditions. In addition, except for ED-min (3 SE), seeding new items at the middle and last locations of adaptive test could result in larger relative efficiency values than seeding them at the beginning. Moreover, the calibration results for the “Middle” seeding location were as good as those generated by the “Rear” seeding location. Furthermore, it is found from Table 1 that ED-o design performed well if new items were seeded in the middle or toward the end of adaptive test. To be specific, ED-o design consistently outperformed ED-min (1 SE), ED-min (3 SE), and ED-mean (3 SE) designs at “Middle” and “Rear” seeding locations, and behaved better than ED-lw design at “Rear” seeding location. This is mainly because CAT is characterized as highly efficient and can make ability estimates rapidly approach their true values as the test length of operational items increases.

When new items were assigned to examinees at “Front” seeding location, ED-lw design generated the highest relative efficiency values, indicating that the information from operational items was helpful to improving calibration efficiency when the provisional test length is short.

By comparing the results of the ED-min and ED-mean designs side by side, one can note that the ED-mean design produced consistently larger calibration efficiency values than the ED-min design at “Middle” and “Rear” seeding locations, whereas at “Front” seeding location, ED-min design outperformed ED-mean design when the neighborhood size was small, and the result was exactly the opposite when the neighborhood size was large. In addition, ED-mean design was more robust to the neighborhood size than ED-min design for both “Middle” and “Rear” seeding locations. For example, when the neighborhood size was changed from “1 SE” to “3 SE,” the relative efficiencies of ED-mean design decreased slightly, while the relative efficiencies of ED-min design decreased dramatically and even lower than those produced by D-VR design when $N_T$ = 200 and 400 (see the italics in Table 1).

In addition, the average computation times for the entire online calibration process were also recorded in Table B1 (see online Appendix B). From Table B1, we can find that D-VR design consumed the least time for calibrating new items as a whole, while ED-min design took the longest time. Compared with the calibration time spent by the D-VR design, the additional computational costs associated with the ED-o, ED-mean, and ED-lw designs were acceptable.

In practice, van der Linden and Ren (2015) recommended seeding new items toward the end of CAT, because it profits maximally from the information available in the ability estimates. From the above-presented results, one can notice that the ED-o design was time-saving and performed well at “Rear” seeding locations. Thus, among the four ED designs, the ED-o design with “Rear” seeding location is used for the next studies.

Exploring the Effect of $n_r$ and $n_b$

The implementations of adaptive designs also depend on the values of $n_r$ and $n_b$. To explore their effects on the performance of adaptive designs, the D-VR design and ED-o design were thoroughly compared under five levels of $n_r$ (i.e., $n_r$ = 100 : 100 : 500) and six levels of $n_b$ (i.e., $n_b$ = 5 : 5 : 30). The resulting relative efficiency values were plotted in Figure 1. Note that, the numbers from 1 to 5 on the $n_r$-axis and from 1 to 6 on the $n_b$-axis indicate that $n_r$ increased from 100 to 500 and $n_b$ increased from 5 to 30, respectively.

Figure 1.

The calibration efficiencies of D-VR design and ED-o design relative to random design under different conditions.

Note. VR = van der Linden and Ren; ED = excellence degree.

Consistent with the results in Table 1, all relative efficiency values in Figure 1 were larger than 1 under all simulation scenarios, indicating that both D-VR design and ED-o design were more efficient than that of the random design. Moreover, ED-o design was superior to D-VR design in terms of calibration efficiency, because the relative efficiency surfaces generated by ED-o design were all above over those resulting from D-VR design.

For each level of $N_T$, ED-o design behaved differently from D-VR design as $n_r$ increased. More specifically, D-VR design worked well for relatively larger $n_r$, while ED-o design was more efficient when $n_r$ was relatively smaller. This is primarily because larger $n_r$ produced more accurate initial item parameters and ED-o design was less sensitive to the calibration precision of initial item parameters than D-VR design. Hence, the performance of ED-o design may be further improved by using more examinees in the adaptive calibration stage. In addition, the relative efficiencies of D-VR design reached the peak at $n_r$ = 400 when $N_T$ = 200, implying that there might exist an optimal proportion between $n_r$, $N_T$, the total number of new items (i.e., $m$), the number of new items that each examinee receives (i.e., $w_p$), and so forth, for D-VR design to achieve maximum relative efficiency.

In addition, a little change was observed in the relative efficiency as $n_b$ varied, thus $n_b$ will be set to be 30 henceforth, in order to save calibration time. A possible explanation is that the impact of $n_b$ was not fully reflected by the six levels (i.e., $n_b$ = 5 : 5 : 30) selected in this study. More researches are welcomed to further explore the impact of $n_b$ on the performance of adaptive designs.

Ability Estimation Accuracy by Using Calibrated Items

In practical terms, calibrated items will be put into use to estimate ability values. To determine whether the differences in item calibration have an impact on the subsequent ability estimation, all the calibrated 2000 new items (20 items in each replication multiply by 100 replications) were used to recover 61 different ability levels (i.e., -3 : 0.1 : 3). The average bias values against the true abilities with different calibration sample sizes are plotted in Figures C1 through C3 (see online Appendix C).

As alluded to Figures C1 through C3, abilities of medium level were recovered better than those of extreme level. As expected, items with true parameters worked better than calibrated items, and larger calibration sample size produced more accurate ability estimates. In addition, items calibrated by the ED-o design performed slightly worse than those calibrated by the D-VR design in recovering positive abilities when $N_T$ = 200. When $N_T$ is relatively larger, ED-o design showed better performance than D-VR design. Specifically, ED-o design generated less biased ability estimates at most ability levels when $N_T$ = 400; and when $N_T$ = 600, new items calibrated by the ED-o design performed as well as the true items, while new items calibrated by the D-VR design still behaved clearly different from the true ones.

Conclusion

The following conclusions can be arrived with the specific settings in the simulations:

1. Adaptive designs are consistently superior to random design under all simulation conditions, and the calibration precision has an impact on the scoring process.

2. Relative to the D-VR design, the ED-o design works better in terms of relative efficiency and is less sensitive to the calibration precision of initial item parameter values. Besides, the abilities can be better recovered by using items calibrated by ED-o for relatively large $N_T$ values (e.g., 400 and 600).

3. ED-min design is more efficient when new items are seeded at the beginning of adaptive test, and it is time consuming and vulnerable to the neighborhood size ${\delta }_p$. By comparison, ED-mean design is more robust to ${\delta }_p$ and performs best when new items are seeded at middle/rear locations for small ${\delta }_p$.

4. Among all adaptive designs, ED-lw design shows best performance when seeding new items at front locations.

5. For all adaptive designs, seeding new items at middle/rear locations generates better results than seeding them at front locations as a whole.

Discussion

The quality of item is essential to obtain accurate ability estimates in the ensuing scoring process. The question of how to efficiently calibrate new items has received great attentions in the context of online calibration. This article attempts to improve the calibration efficiency of the adapted D-optimal design proposed by van der Linden and Ren (2015) (i.e., D-VR design), and further proposes the ED criterion for calibrating new items online. Moreover, the ED criterion is equipped with four different schemes for measuring information provided by the current active examinee, and four ED designs (i.e., ED-o, ED-min, ED-mean, and ED-lw) are put forward accordingly. Besides, the problem regarding the dependence of adaptive designs on unknown item parameters is also well solved by sequentially updating the item parameters.

Results showed that the newly proposed ED designs perform better than the D-VR design when calibrating the 2PL model. Note that the ED designs can be readily generalized to other item response theory (IRT) models and other CAT formats. As an example, this article also applied one representative ED design (i.e., ED-o design) to calibrate the 3PL model, which is widely used in many large-scale assessment programs. Results indicated that ED-o is also a promising design when calibrating the 3PL model online. For more detailed information, interested readers are referred to the online Appendix D.

The current study can be expanded in several directions. First, it is worth exploring in the future whether the ED designs still work well when they are extended to multidimensional CAT and cognitive diagnostic CAT. Second, this article derived the ED-mean design by using samples randomly drawn from a small neighborhood size of ability estimates; future researches may explore its performance when samples are directly drawn from the posterior distribution of ability (e.g., Formula 11). Third, the virtual optimal ability is similarly obtained by randomly sampling from an interval. Because true abilities are typically normally distributed in practice, it would be interesting to further evaluate the ED designs by incorporating such useful distribution information. Fourth, this article sequentially updates the item parameters in the adaptive calibration stage with the intent of dealing with the dependence on unknown item parameters. However, in addition to the sequential design, researchers may use other feasible approaches, such as the minimax optimal design and the Bayesian optimal design, to handle this kind of dependence. Last, but not least, this article retires each new item from the calibration process once it receives a fixed number of responses; thus, another line of research worth considering is to compare the number of examinees required by different designs when the calibration precision reaches a predetermined threshold.