A Comparison of Item Selection Methods and Parameter Estimation Approaches for Online Calibration in Computerized Adaptive Testing

Abstract

Online calibration enables continuous replenishment of computerized adaptive testing (CAT) item banks by embedding new pretest items within operational test sessions. However, the relative performance of different calibration components and their interactions have not been fully examined in a unified framework. This study compared three pretest item selection methods (maximum Fisher information [MFI], D-optimal value design, and Bayesian D-optimal design), two parameter estimation methods (a fixed-ability joint maximum likelihood procedure [JML-itpar] and one expectation-maximization cycle [OEM]), three levels of random calibration stage sample size (250, 500, and 1,000), and three levels of calibration sample size per pretest item (250, 500, and 1,000) under the one-parameter logistic and two-parameter logistic item response theory models through a Monte Carlo simulation with 108 conditions and 100 replications per condition. The results revealed a trade-off among item selection methods: MFI yielded the most accurate difficulty parameter estimates, while D-optimal value design and Bayesian D-optimal design produced better discrimination parameter recovery. Bayesian D-optimal design offered the best balance between accuracy and calibration efficiency. For the first time in the standard online calibration context, JML-itpar was systematically evaluated and found to be a viable estimation method. JML-itpar paired with MFI produced lower discrimination parameter root mean squared error than OEM at calibration sample sizes of 500 and 1,000. OEM remained superior for the difficulty parameter. The calibration sample size per pretest item was the most influential factor, with substantial accuracy gains from 250 to 500 responses and diminishing returns beyond 500. The random calibration stage sample size had negligible effects. These findings provide practical guidance for selecting calibration components in operational CAT programs.

Introduction

The integration of item response theory (IRT) and computer technology has shifted educational and psychological testing from fixed linear formats to adaptive approaches. Computerized adaptive testing (CAT) is a prominent application of this shift, designed to select and sequentially administer the most appropriate items to examinees according to their provisional ability estimates. CAT offers substantial advantages for both test takers and test designers: it estimates ability with less measurement error, shortens the test length, helps maintain examinee motivation (although this does not hold generally true; see Frey et al., 2024), provides timely access to results and feedback, and allows considerable flexibility in testing time and location (van der Linden & Glas, 2010; Wainer & Eignor, 2000; Weiss, 1982).

CAT faces an inherent challenge in maintaining the quality of the item bank. Because the system continuously administers items to different examinees at different time intervals, some items in the bank become overexposed or outdated, and the quality of the item bank gradually decreases (Wainer & Mislevy, 1990). These items should be retired from the bank and replaced by new ones. Therefore, item replenishment plays an essential and critical role in managing and maintaining the item bank (Chang & Lu, 2010). As with the initial construction of the item bank, newly developed items must be calibrated before they can be transferred to the operational pool. The precision of item parameter estimation is known to directly affect the validity and reliability of test scores, as well as related psychometric procedures such as test equating and differential item functioning analysis. It is therefore crucial that item parameters can be estimated on a large scale, efficiently, and economically (Chang & Lu, 2010; Stout et al., 2003).

Two methods stand out for calibrating new items (also called field test items or pretest items) for item replenishment in CAT (Wainer & Mislevy, 1990). The first, conventional offline calibration, relies on an anchor item design and is performed using a paper-and-pencil form, similar to the initial development of the item bank. After the parameters of new items are estimated, a linking or scaling procedure is applied to place them on the same scale as the operational items (Wainer & Mislevy, 1990). This method is expensive and time-consuming and carries disadvantages such as low examinee motivation during the field test, potentially unreliable data, and the need for an extra linking process. The second method, online calibration, offers significant benefits over the traditional approach. In this method, new items are carefully embedded among operational items and are seeded inconspicuously to examinees during their CAT session (Ban et al., 2001; Stocking, 1988; Wainer & Mislevy, 1990). This helps maintain examinees’ focus and motivation, leading to more reliable response data (Parshall, 1998). The method also allows for simultaneous estimation of examinee abilities and item parameters, eliminating the need for complex equating and linking techniques. Online calibration is both cost-effective and time-saving, making it a practical choice for item replenishment in CAT (Chen et al., 2012; Makransky & Glas, 2010).

Online calibration seeks to estimate the parameters of pretest items by administering them to an appropriate set of test takers (Ali & Chang, 2014; Zheng, 2014). The procedure begins by randomly assigning a predetermined number of pretest items to each examinee at specific positions within the test (known as seeding positions). Because no item parameter estimates are available at this stage, responses to pretest items are compiled and stored until a defined threshold of responses is reached, marking the completion of the random calibration phase. Once a sufficient sample size has been attained, initial item parameters are estimated. In the subsequent phase, pretest items are allocated to test takers based on predetermined selection rules, typically toward the conclusion of their CAT session. Using new responses from active examinees together with responses recorded from earlier examinees who encountered the same item, the parameters of each pretest item are iteratively updated. This process continues until the specified stopping criterion is met, at which point the final parameters are determined (Zheng, 2014).

The online calibration procedure consists of two fundamental components. The first is the pretest item selection method (also referred to as the online calibration design), which encompasses the rules governing how pretest items are administered to examinees during the testing session. Researchers have addressed this component in several ways, including sequential design, random selection, and adaptive design. Sequential design (Chang & Lu, 2010; Jones & Jin, 1994; Lu, 2014; Zhu, 2006), rooted in traditional optimal design paradigms, aims to select the most suitable examinee from the pool for each pretest item based on specific optimality criteria (L, E, A, or primarily D). While this approach offers theoretical advantages, its practical implementation is challenging because of the dynamic nature of CAT sessions (Guo, 2016; Zheng, 2014). Random selection (Wainer & Mislevy, 1990) provides a straightforward alternative, but it can reduce calibration efficiency because the selected examinees may not be well matched to the informative ability range for a given pretest item (He et al., 2020; Kingsbury, 2009; Zheng, 2014). Adaptive designs represent a practical compromise and can be categorized into examinee-centered and item-centered approaches. Examinee-centered methods select pretest items based on the examinee’s ability level (Zheng, 2014), while item-centered methods prioritize items that contribute the most to parameter estimation. Several item-centered adaptive methods have been proposed, such as D-optimal value design, which selects the pretest item with the highest D-optimal value (Anderson, 1984), the Ordered Informative Range Priority Index (Zheng, 2014), the Suitability Index (Ali & Chang, 2014), and the Bayesian optimal design (van der Linden & Ren, 2015). Among these, D-optimality is the most widely used criterion in the literature, though it can lead to uneven selection of certain items.

The second fundamental component is the parameter estimation method, which deals with how pretest item parameters are estimated using the response data from examinees whose abilities have been measured with operational items. This is a complex task because of the unique characteristics of CAT data, including sparse response patterns, restricted ability ranges for individual items, and relatively small sample sizes compared to traditional paper-and-pencil administration (Ban et al., 2001; Stocking, 1988). To address these challenges, various online calibration methods have been developed: method A and method B (Stocking, 1988), one expectation-maximization cycle (OEM; Wainer & Mislevy, 1990), multiple expectation-maximization (MEM) cycles (Ban et al., 2001), BILOG/Strong Prior (Ban et al., 2001), and maximum likelihood estimation with Lord’s bias correction (MLE-LBCI-Method A; He et al., 2017). Method A is a simple approach that first estimates examinee ability using operational item responses and then estimates pretest item parameters by treating these abilities as fixed (Chen & Wang, 2016). However, it is susceptible to parameter drift because measurement error in the ability estimates is ignored (Ban et al., 2001). Method B addresses this drift by incorporating anchor items but requires additional responses from examinees, making it less practical (Ban et al., 2001). OEM and MEM are iterative methods that compute the expected posterior distribution of ability to maximize the marginal maximum likelihood function. OEM uses a single expectation-maximization (EM) cycle based on operational items only, while MEM applies multiple cycles incorporating both operational and pretest item information, potentially improving estimates but introducing sensitivity to poorly calibrated pretest parameters. In the present study, OEM serves as the EM benchmark because it conditions on the same information as the fixed-ability ML procedure under investigation: both use only operational-item information about examinee ability together with the responses to the pretest item being calibrated, so the comparison isolates the estimation principle rather than differences in the information sets. The MLE-LBCI-Method A incorporates Lord’s bias correction into the method A framework to reduce estimation bias (He et al., 2017). Although joint maximum likelihood (JML) estimation is well established in traditional IRT calibration, its application as a parameter estimation method in online calibration has received virtually no attention.

In addition to these two fundamental components, several design elements have been examined for their impact on parameter recovery. The seeding position, that is, where pretest items are placed within the CAT session, significantly influences the accuracy of parameter estimates. Later positions generally yield better results because ability estimates are more stable toward the end of the test (Kingsbury, 2009; van der Linden & Ren, 2015). Various seeding strategies have been documented, including random assignment (Chen, 2017; Chen & Wang, 2016), fixed positions (Kingsbury, 2009), and placement at specific test segments (He et al., 2020; Zheng & Chang, 2017). The termination rule dictates when sampling for a pretest item concludes and the item is removed from the pretest bank. Although criteria such as standard error and parameter stabilization have been proposed (Kingsbury, 2009), sample size remains the predominant stopping criterion in contemporary research (Ali & Chang, 2014; He et al., 2017, 2020; Ren et al., 2017; van der Linden & Ren, 2015; Zheng & Chang, 2017). Sample size can be defined as a total or cumulative count across all pretest items, or as a count specific to each pretest item. The total approach risks overexposing certain items, whereas the per-item approach promotes more balanced exposure and more consistent calibration outcomes.

The literature on online calibration has grown considerably since Stocking’s (1988) foundational work. Previous studies (e.g., Ali & Chang, 2014; Ban et al., 2001; Chang & Lu, 2010; He et al., 2017, 2020; Kingsbury, 2009; Makransky & Glas, 2010; van der Linden & Ren, 2015; Zheng, 2014; Zheng & Chang, 2017) have generally focused on different combinations of item selection methods, parameter estimation methods, and sample size conditions. However, only a few of these studies (e.g., He et al., 2017, 2020; Zheng, 2014; Zheng & Chang, 2017) have examined multiple components together within a single simulation design in unidimensional CAT. Perhaps more importantly, with the exception of Verschoor et al. (2019), who employed JML within a different adaptive testing design, no study has systematically evaluated JML as a parameter estimation method in standard online calibration. JML is computationally simpler than marginal maximum likelihood approaches because it does not require integration over the ability distribution, and in the online calibration context, examinee abilities estimated from operational items are treated as known values, which may alleviate some of JML’s well-documented limitations in traditional settings (e.g., Neyman & Scott, 1948). Since the completion of the present study, research on online calibration has continued to develop. He and Chen (2020) proposed D-c optimal designs incorporating classical and reformed D-optimality criteria; He et al. (2021) introduced information-per-time-unit designs for continuous calibration; Yuan et al. (2023) extended online calibration methods to polytomous multidimensional CAT; and Bjermo et al. (2025) developed calibration designs that account for uncertainty in ability estimates. In parallel, frameworks have emerged that embed calibration within broader test-design machinery: van der Linden and Jiang (2020) integrated pretest item selection into a shadow-test model that accommodates content and exposure constraints, and fully Bayesian approaches account for the joint uncertainty in ability and item parameters during testing and calibration (Fink et al., 2025; van der Linden & Ren, 2020). These developments highlight the continued importance of refining calibration procedures, yet the systematic comparison of core components and the evaluation of JML in standard online calibration remain unaddressed.

The purpose of the present study was to provide a comprehensive comparison of the core components of online calibration in unidimensional CAT under both the one-parameter logistic (1-PL) and two-parameter logistic (2-PL) IRT models. Specifically, the study examined three pretest item selection methods (maximum Fisher information [MFI], D-optimal value design, and Bayesian D-optimal design), two parameter estimation methods (a fixed-ability implementation of joint maximum likelihood, hereafter JML-itpar, and OEM), three levels of random calibration stage sample size (250, 500, and 1,000), and three levels of calibration sample size per pretest item (250, 500, and 1,000). The accuracy of parameter estimates was evaluated using bias and root mean squared error (RMSE), and calibration efficiency was assessed through the cumulative sample size required to complete the calibration of all pretest items. A further objective was to evaluate the feasibility of JML-itpar as a parameter estimation method for online calibration and to provide practical guidance for practitioners regarding the selection of calibration components. The following research questions guided this study:

1. How do the pretest item selection methods (MFI, D-optimal value design, and Bayesian D-optimal design) compare in terms of parameter estimation accuracy and calibration efficiency?

2. How does JML-itpar compare to OEM as a parameter estimation method for online calibration, and under what conditions is JML-itpar a viable alternative?

3. What is the effect of the random calibration stage sample size on the accuracy of pretest item parameter estimates?

4. How does the calibration sample size per pretest item affect parameter estimation accuracy and calibration efficiency?

Method

Simulation Design

A Monte Carlo simulation study was conducted to examine the effects of four factors on the accuracy of pretest item parameter estimation in unidimensional online calibration. The study employed a fully crossed factorial design with the following factors: (a) IRT model (1-PL, 2-PL), (b) pretest item selection method (MFI, D-optimal value design, Bayesian D-optimal design), (c) parameter estimation method (JML-itpar, OEM cycle), (d) random calibration stage sample size (number of examinees, n_r = 250, 500, 1,000), and (e) calibration sample size per pretest item (N = 250, 500, 1,000). This yielded 2 × 3 × 2 × 3 × 3 = 108 unique conditions. Each condition was replicated 100 times to reduce the influence of random sampling variability, consistent with the replication count used in prior online calibration research (Ban et al., 2001; Zheng & Chang, 2017). Table 1 summarizes the simulation conditions.

Table 1.

Simulation Design.

Factors	Levels
IRT model	1-PL, 2-PL
Pretest item selection method	MFI, D-optimal value design, Bayesian D-optimal design
Parameter estimation method	JML-itpar, OEM
Random calibration stage sample size (number of examinees; nr)	250, 500, 1,000
Calibration sample size (N)	250, 500, 1,000

Note. The fully crossed design yielded 2 × 3 × 2 × 3 × 3 = 108 conditions, each replicated 100 times. MFI = maximum Fisher information; JML-itpar = fixed-ability joint maximum likelihood (item parameters only); OEM = one expectation-maximization cycle; IRT = item response theory; 1-PL = one-parameter logistic; 2-PL = two-parameter logistic.

The three-parameter logistic (3-PL) model was not considered in this study, for two reasons. First, adaptive item selection reduces the occurrence of guessing because operational items are targeted to the provisional ability estimate (van der Linden & Glas, 2010), a rationale adopted in previous online calibration research employing the 1-PL or 2-PL model (e.g., Chang & Lu, 2010; Kingsbury, 2009). Second, and more decisive for a calibration study, estimating a model with a guessing parameter requires very large samples, on the order of 5,000 to 20,000 respondents, and even then, the estimates are often unstable (Akour & AL-Omari, 2013). Such sample sizes lie far beyond realistic online calibration budgets, including the 250 to 1,000 responses per pretest item examined here. Available remedies restrict or fix the guessing estimates or, in a Bayesian framework, place heavy priors on them; these heuristic methods are beyond the scope of the present study.

Data Generation

For each replication, a new operational item bank containing 250 items and a pretest item bank containing 25 items were randomly generated. This procedure follows the design of several previous studies in which item banks were regenerated across replications (He et al., 2020; Kingsbury, 2009; Zheng, 2014). Under the 2-PL model, discrimination parameters were drawn from a log-normal distribution, a ∼ log-normal(0, 0.25), and difficulty parameters were drawn from a standard normal distribution, b ∼ N(0, 1). These distributions have been used in prior simulation work to approximate realistic item characteristics (Fink et al., 2018; Natesan et al., 2016). Under the 1-PL model, the discrimination parameter was fixed at a = 1 for all items, and only the difficulty parameter was estimated. The same generating distributions were applied to both operational and pretest items. Examinee ability parameters were independently sampled from a standard normal distribution, θ ∼ N(0, 1), for each replication. Descriptive statistics for the generated item parameters were consistent with the specified distributions across all replications.

CAT Procedure

Each simulated examinee completed a fixed-length operational CAT consisting of 35 items. The first operational item was randomly selected from items with medium difficulty (b between −0.5 and 0.5), as recommended by Thompson and Weiss (2011). Subsequent items were selected using the MFI criterion. Both interim and final ability estimates were computed using the weighted likelihood estimator (WLE; Warm, 1989), which reduces the finite-sample bias associated with MLE. No content balancing or item exposure control constraints were applied, as the focus of this study was on the online calibration components rather than operational CAT administration features. The operational item responses were not used for pretest item parameter estimation; they served only to generate the ability estimates that were subsequently used in the calibration phase.

Online Calibration Procedure

The online calibration procedure was implemented in two sequential phases. In the random calibration phase, pretest items were administered to examinees immediately following the completion of their 35 operational items. This end-of-test seeding position was chosen because ability estimates are most precise after all operational items have been administered, which in turn improves the quality of information available for pretest item calibration (He et al., 2020; Zheng & Chang, 2017). In the random calibration phase, pretest items were administered to a fixed number of examinees determined by the random calibration stage sample size condition (n_r = 250, 500, or 1,000 examinees). Each of these examinees received 5 pretest items selected at random from the 25-item pretest bank. Under this design, the expected number of random-phase responses per pretest item equals n_r × 5/25, that is, approximately 50, 100, and 200 responses for the three n_r conditions. Because the largest expected random-phase contribution (200 responses per item) remains below the smallest calibration threshold (N = 250), every crossed condition involves a substantive adaptive phase. Because no item parameter estimates were available during this phase, responses to pretest items were recorded and stored without parameter updating. When the random phase ended, initial item parameters were estimated using the assigned estimation method (JML-itpar or OEM), and the adaptive calibration phase began.

In the adaptive calibration phase, pretest items were selected based on one of the three item selection methods (MFI, D-optimal, or Bayesian D-optimal) rather than at random. As new examinees completed their operational CAT session, pretest items were assigned to them using the current parameter estimates from the random phase and all subsequent updates. Item parameters were re-estimated after every 10 new responses, following the update interval adopted in previous research (Zheng, 2014; Zheng & Chang, 2017). The convergence threshold for the Newton-Raphson iterations was set to 0.001, with a maximum of 100 iterations. Sampling for a given pretest item was terminated once it accumulated a total of N responses (250, 500, or 1,000, inclusive of both phases), at which point the item was removed from the pretest bank. The procedure continued until all 25 pretest items had been removed. Responses to pretest items were not included in the scoring of any examinee’s ability. The total number of examinees required to complete the calibration of all pretest items was recorded as the cumulative sample size, which served as an indicator of calibration efficiency.

Pretest Item Selection Methods

Three pretest item selection methods were compared. All three are item-centered methods that aim to select the pretest item whose parameter estimation benefits most from the current examinee’s response. Under the 2-PL model, the item response function for pretest item k is given by

$$P_k\left({\theta }_j\right)={\{1+\exp [-{\mathrm{a}}_k({\theta }_j-b_k)]\}}^{-1}$$ (1)

where $a_k$ is the discrimination parameter, $b_k$ is the difficulty parameter, and ${\theta }_j$ is the ability of examinee j. For the 1-PL model, $a_k$  = 1 for all items.

The Fisher information for pretest item k at ability ${\theta }_j$ is $I\left({\theta }_j\right)={a_k}^2{P}_k\left({\theta }_j\right)[1-P_k\left({\theta }_j\right)]$ . Under the MFI method, the pretest item with the highest Fisher information at the current examinee’s estimated ability is selected. This criterion is straightforward and computationally efficient but considers only the information contributed by the current examinee without taking into account the information already accumulated from previous examinees.

The D-optimal value design uses the full Fisher information matrix of the item parameter vector. Under the 2-PL model, the information matrix for pretest item k contributed by examinee j with ability ${\theta }_j$ is a 2 × 2 matrix $I({\eta }_k;{\theta }_j)$ , where ${\eta }_k=(a_k,b_k{)}^T$ . The elements of this matrix are:

$$I_{\mathrm{aa}}=({\theta }_j-b_k{)}^2{P}_k({\theta }_j)[1-P_k({\theta }_j)]$$ (2)

$$I_{\mathrm{ab}}=-a_k({\theta }_j-b_k)P_k({\theta }_j)(1-P_k({\theta }_j))$$ (3)

$$I_{\mathrm{bb}}={a_k}^2{P}_k\left({\theta }_j\right)[1-P_k\left({\theta }_j\right)]$$ (4)

The total information matrix for item k based on all $n_k$ examinees who have responded to the item is $I_k({\eta }_k;\theta )=\sum I({\eta }_k;{\theta }_j)$ . The D-optimal selection rule identifies the pretest item whose total information matrix, augmented by the current examinee’s contribution, has the largest determinant:

$$k^{*}=\arg \max \left\{\det [I_k({\eta }_k;\theta )+I({\eta }_k;{\theta }_c)]\right\}$$ (5)

where ${\theta }_c$ denotes the current examinee’s ability estimate. Maximizing the determinant of the information matrix is equivalent to minimizing the generalized variance of the item parameter estimates (Anderson, 1984). For the 1-PL model, the information matrix reduces to a scalar, and the determinant simplifies to the summed information.

The Bayesian D-optimal design (van der Linden & Ren, 2015) modifies the D-optimal criterion by selecting the pretest item for which the current examinee provides the largest expected contribution to the total information. Rather than maximizing the augmented determinant, it maximizes the increment, that is, the difference between the augmented and current determinants:

$$k^{*}=\arg \max \{\det [I_k+I({\eta }_k;{\theta }_c)]-\det \left[I_k\right]\}$$ (6)

In Equation 6, I_k denotes the total information matrix accumulated for pretest item k from all examinees who have responded to it so far; its elements are the sums of the expressions in Equations 2 to 4 over these examinees, evaluated at the current parameter estimates. The criterion is Bayesian in the sense of van der Linden and Ren (2015): the information already collected for the item serves as the prior precision of its parameters, and the selection rule maximizes the expected gain in this precision contributed by the current examinee. Under the 1-PL model, the information matrix reduces to a scalar, the accumulated term cancels in the difference, and the Bayesian D-optimal criterion coincides with MFI. This formulation favors items for which the current examinee provides relatively more new information, which can result in a more balanced distribution of examinees across items compared to the standard D-optimal design. The item parameter estimates used in both D-optimal and Bayesian D-optimal selection are the most recently updated estimates, beginning with the initial estimates obtained at the end of the random phase.

Parameter Estimation Methods

Joint Maximum Likelihood

In traditional IRT calibration, JML simultaneously estimates both ability and item parameters through an iterative two-stage procedure (Baker & Kim, 2004). In the online calibration context, however, each examinee’s ability has already been estimated from the operational items using WLE prior to encountering any pretest items. Therefore, the procedure was implemented as fixed-ability maximum likelihood: the WLE ability estimates were treated as known constants, and only the pretest item parameters were estimated via Newton-Raphson optimization. To distinguish this implementation from classical JML, which estimates person and item parameters jointly, we label it JML-itpar (joint maximum likelihood for item parameters) and use this label throughout. This approach retains the computational simplicity of JML (no integration over the ability distribution is required) while avoiding the classical incidental parameter problem that arises in classical JML when the number of examinees grows with the number of parameters (Neyman & Scott, 1948). It should be noted that this fixed-ability formulation is functionally similar to Stocking’s (1988) method A, with one key difference: method A applies a single Newton-Raphson step per update, whereas the present JML-itpar implementation iterates until convergence (threshold = 0.001, maximum 100 iterations). This distinction allows JML-itpar to fully optimize item parameter estimates at each update rather than relying on incremental adjustments.

For the 2-PL model, the item parameters were reparameterized as dk = log(ak) to ensure positive discrimination estimates, and the Newton-Raphson update was applied to the vector (dk, bk)^T. Starting values for the Newton-Raphson iterations were obtained through a root-finding procedure (Brent, 1973) applied to the score equations: the initial difficulty estimate was the root of the first derivative of the log-likelihood with respect to b over the interval [−4, 4], and the initial discrimination estimate was similarly obtained over [0.01, 3]. At each iteration, the gradient vector and the Hessian matrix were computed from the summed contributions of all examinees who had responded to item k, and the parameter vector was updated accordingly. Boundary constraints were imposed to maintain parameter estimates within plausible ranges: discrimination values were constrained between 0.01 and 3.0, and difficulty values between −4.0 and 4.0. If an estimate exceeded these boundaries, it was reset to its initial value. For the 1-PL model, only the difficulty parameter was estimated using the same Newton-Raphson procedure. The gradient and Hessian formulas follow standard maximum likelihood derivations for the 2-PL model (see Baker & Kim, 2004, for a complete treatment).

One Expectation-Maximization Cycle

The OEM method (Wainer & Mislevy, 1990) estimates pretest item parameters within the marginal maximum likelihood framework, using a single EM cycle per parameter update. In the E-step, the posterior distribution of each examinee’s ability is computed using only the operational item responses and the operational item parameters. Specifically, for examinee j who responded to pretest item k, the posterior distribution of θ _j is obtained by combining the likelihood of the operational item responses with a standard normal prior distribution over a set of quadrature points. In this study, 33 equally spaced quadrature points were used over the interval [−4, 4]. The use of a standard normal prior for ability is consistent with both the generating distribution in the simulation and common practice in online calibration research (Ban et al., 2001; He et al., 2020; Zheng, 2014). The expected number of correct responses and the expected number of total responses at each quadrature point are then computed.

In the M-step, the pretest item parameters are updated by a single Newton-Raphson step using the expected counts from the E-step as sufficient statistics. The log-likelihood is expressed as a function of the item parameters and the expected counts, and the Newton-Raphson update is applied in the same manner as in JML-itpar, except that the sufficient statistics are weighted by the posterior ability distributions rather than based on point estimates of ability. As with JML-itpar, the discrimination parameter was reparameterized on the log scale for the 2-PL model. A key distinction between OEM and JML-itpar is that OEM accounts for uncertainty in ability estimation through the posterior distribution, whereas the JML-itpar implementation in this study uses point estimates of ability. The OEM E-step and M-step follow the procedure described by Ban et al. (2001) and Wainer and Mislevy (1990).

Evaluation Criteria

The accuracy of pretest item parameter estimation was evaluated using two criteria: bias and RMSE. For a given parameter (a or b), these criteria were computed by averaging over the 25 pretest items within each replication and then averaging across the 100 replications. Bias was defined as:

$$\mathrm{Bias}=(1/R)\sum (1/m)\sum ({\xi }_{\mathrm{kr}}-{\xi }_k)$$ (7)

where ${\xi }_{\mathrm{kr}}$ is the estimated parameter of pretest item k in replication r, ${\xi }_k$ is the true parameter value, m = 25 is the number of pretest items, and R = 100 is the number of replications. RMSE was defined as:

$$\mathrm{RMSE}=\sqrt{[(1/R)\sum (1/m)\sum {({\xi }_{\mathrm{kr}}-{\xi }_k)}^2]}$$ (8)

For the 2-PL model, bias and RMSE were computed separately for the discrimination parameter a and the difficulty parameter b. For the 1-PL model, only the b parameter was evaluated. In addition to these accuracy measures, the cumulative sample size required to complete the calibration of all 25 pretest items was recorded as a measure of calibration efficiency. A smaller cumulative sample size under comparable accuracy indicates a more efficient calibration process.

Software

All simulations were conducted using a custom program written in R (R Core Team, 2024) with computationally intensive routines implemented in C++ via the Rcpp (Eddelbuettel et al., 2018) and RcppArmadillo (Eddelbuettel et al., 2019) packages. Simulations were executed in parallel across multiple computing platforms to reduce total computation time. The simulation code will be made available as part of an R package currently in preparation.

Results

Results are organized around the four research questions. The presentation begins with an overview of bias, followed by findings on item selection methods, parameter estimation methods, the random calibration stage sample size, and the calibration sample size per pretest item. Calibration efficiency is addressed at the end of the section. Given that the 2-PL model involves the estimation of both discrimination and difficulty parameters and therefore provides a richer picture of calibration performance, the 2-PL results are reported in detail with supporting tables and figures. Results for the 1-PL model, which showed consistent patterns with the 2-PL findings, are summarized in the text.

Bias

Across all simulated conditions, bias values for the difficulty parameter were close to zero under both IRT models and both estimation methods. Under the 1-PL model, bias of b ranged from −0.0050 to 0.0081 for JML-itpar and from −0.0029 to 0.0071 for OEM, indicating negligible systematic error regardless of item selection method, random stage sample size, or calibration sample size. Under the 2-PL model, bias of b was similarly small, ranging from −0.0096 to 0.0104 for JML-itpar and from −0.0092 to 0.0045 for OEM. Bias of the discrimination parameter a, however, was consistently negative across all conditions, indicating systematic underestimation. JML-itpar produced a parameter bias value between −0.0553 and −0.0852, and OEM between −0.0859 and −0.1366. OEM exhibited greater underestimation of the discrimination parameter than JML-itpar across all conditions. This pattern of negative bias in discrimination estimates is consistent with observations reported in prior online calibration research (Ban et al., 2001; He et al., 2017). Because bias was relatively stable across conditions and did not differ meaningfully by item selection method or sample size level, the remainder of the “Results” section focuses on RMSE as the primary accuracy criterion. Table 2 presents the full RMSE values for the 2-PL conditions.

Table 2.

RMSE of a and b Parameters for Different Item Selection Methods Under the 2-PL Model.

			JML-itpar		OEM
nr	N	Method	a	b	a	b
250	250	MFI	0.2074	0.2825	0.2043	0.2531
		DOVD	0.2091	0.3344	0.2123	0.3030
		BDOD	0.2077	0.3176	0.2075	0.2898
	500	MFI	0.1617	0.1922	0.1815	0.1904
		DOVD	0.1694	0.2481	0.1679	0.2151
		BDOD	0.1733	0.2413	0.1628	0.2104
	1,000	MFI	0.1396	0.1616	0.1695	0.1469
		DOVD	0.1533	0.2112	0.1393	0.1752
		BDOD	0.1392	0.1970	0.1394	0.1756
500	250	MFI	0.2121	0.3165	0.2059	0.2645
		DOVD	0.2155	0.3441	0.2126	0.2971
		BDOD	0.2095	0.3167	0.2012	0.2942
	500	MFI	0.1706	0.2282	0.1843	0.1898
		DOVD	0.1785	0.2538	0.1696	0.2248
		BDOD	0.1631	0.2272	0.1618	0.2106
	1,000	MFI	0.1370	0.1561	0.1782	0.1561
		DOVD	0.1516	0.2068	0.1421	0.1728
		BDOD	0.1461	0.1926	0.1376	0.1621
1,000	250	MFI	0.2062	0.3304	0.2016	0.2893
		DOVD	0.2106	0.3170	0.2108	0.2879
		BDOD	0.2085	0.3283	0.2037	0.2810
	500	MFI	0.1701	0.2247	0.1777	0.1867
		DOVD	0.1691	0.2631	0.1642	0.2131
		BDOD	0.1712	0.2601	0.1633	0.2102
	1,000	MFI	0.1423	0.1854	0.1760	0.1433
		DOVD	0.1389	0.2035	0.1417	0.1812
		BDOD	0.1446	0.2125	0.1382	0.1722

Note. The lowest RMSE value among the three item selection methods within each condition is presented in boldface. Between-replication standard errors of the tabled RMSE values (delta method, 100 replications per condition) did not exceed 0.008 for the a parameter and 0.018 for the b parameter. n_r = sample size of the random calibration stage; N = calibration sample size per pretest item; MFI = maximum Fisher information; DOVD = D-optimal value design; BDOD = Bayesian D-optimal design; JML-itpar = fixed-ability joint maximum likelihood (item parameters only); OEM = one expectation-maximization cycle; 2-PL = two-parameter logistic; RMSE = root mean squared error.

Item Selection Methods

The three item selection methods produced distinct patterns of RMSE that varied by parameter, estimation method, and calibration sample size (see Table 2 and Figures 1 and 2).

Figure 1.

RMSE of the discrimination parameter (a) for different item selection methods under the 2-PL model. Rows represent random calibration stage sample sizes (n_r = 250, 500, 1,000); columns represent parameter estimation methods (JML-itpar, OEM).

Note. MFI = maximum Fisher information; D-optimal = D-optimal value design; Bayesian D-optimal = Bayesian D-optimal design; JML-itpar = fixed-ability joint maximum likelihood; OEM = one expectation-maximization cycle; 2-PL = two-parameter logistic; RMSE = root mean squared error.

Figure 2.

RMSE of the difficulty parameter (b) for different item selection methods under the 2-PL model. See Figure 1 for panel layout and abbreviations.

For the discrimination parameter a under the 2-PL model (Figure 1), the relative performance of the item selection methods depended on the estimation method. When JML-itpar was used, no single item selection method was clearly superior: MFI produced the lowest RMSE in four of the nine conditions, Bayesian D-optimal design in three, and D-optimal value design in two. All three methods yielded comparable RMSE values that decreased with increasing calibration sample size. When OEM was used, the pattern was more differentiated. D-optimal value design produced the highest RMSE when the calibration sample size was small (N = 250), but its performance improved rapidly as N increased and it achieved the lowest or near-lowest RMSE values at N = 1,000. MFI produced the lowest or near-lowest RMSE at N = 250 but showed limited improvement at larger sample sizes. At N = 1,000, MFI produced notably higher RMSE values than D-optimal value design and Bayesian D-optimal design. In standardized terms (condition differences divided by the pooled between-replication standard deviation of replication-level RMSE), these were the largest selection-method differences observed in the study: MFI’s disadvantage at N = 1,000 under OEM corresponded to standardized differences of 1.35 to 1.51, or absolute RMSE differences of 0.030 to 0.040. Bayesian D-optimal design showed steady improvement and performed well across all sample sizes under OEM. At large calibration sample sizes, the three item selection methods converged in performance under JML-itpar, whereas under OEM, MFI fell behind D-optimal value design and Bayesian D-optimal design.

For the difficulty parameter b under the 2-PL model (Figure 2), MFI produced the lowest RMSE across nearly all conditions for both JML-itpar and OEM. This advantage was particularly pronounced under OEM, where MFI consistently outperformed both D-optimal value design and Bayesian D-optimal design. D-optimal value design yielded the highest b parameter RMSE under most conditions, and this disadvantage was largest at small calibration sample sizes. Bayesian D-optimal design fell between the two other methods, with performance closer to MFI than to D-optimal value design in many conditions. The superiority of MFI for the difficulty parameter is consistent with its design rationale: by selecting the pretest item that provides the most Fisher information at the examinee’s estimated ability, MFI effectively matches examinees to items near their difficulty level, which benefits b parameter estimation. D-optimal value design and Bayesian D-optimal design, in contrast, optimize the full information matrix including the discrimination component, and this joint optimization appears to come at some cost to b parameter recovery.

When the results for the two parameters are considered together, the findings indicate a trade-off between the item selection methods. MFI provided the best difficulty parameter estimates but was less effective for the discrimination parameter at larger sample sizes. D-optimal value design and Bayesian D-optimal design were better suited for discrimination parameter recovery, particularly under OEM, but at the expense of higher RMSE for difficulty. Bayesian D-optimal design offered a reasonable balance across both parameters. Expressed on the same standardized scale, most differences among the selection methods were small to moderate: differences between D-optimal value design and Bayesian D-optimal design did not exceed a standardized value of 0.28 in any condition, and MFI’s advantage for the difficulty parameter reached at most 0.63. The clear exception was the discrimination parameter under OEM at N = 1,000 noted above.

Under the 1-PL model, where only the b parameter was estimated, D-optimal value design consistently produced the lowest RMSE for JML-itpar, with values ranging from 0.0785 to 0.1589 across conditions. For OEM, MFI and Bayesian D-optimal design produced identical results up to numerical precision, as implied by the equivalence of the two criteria under the 1-PL model, and both outperformed D-optimal value design in eight of the nine conditions. These patterns were consistent with those observed for the b parameter under the 2-PL model for OEM, but the advantage of D-optimal value design with JML-itpar was specific to the 1-PL case, where the optimization criteria simplified to a single parameter.

Parameter Estimation Methods

The comparison of JML-itpar and OEM revealed differences that depended on the item selection method, the IRT model, and the parameter under consideration. For the difficulty parameter, OEM produced lower RMSE values than JML-itpar in most conditions (Table 2 and Figure 2). Under the 2-PL model, this advantage was of broadly similar size across the three selection methods and across calibration sample sizes. Under the 1-PL model, the advantage was most pronounced under MFI and Bayesian D-optimal design and grew with the calibration sample size, but it nearly disappeared under D-optimal value design: OEM produced only marginally lower RMSE than JML-itpar in the nine 1-PL D-optimal conditions, and the differences did not exceed 0.004.

For the discrimination parameter a under the 2-PL model, the comparison was less one-sided (Figure 1). Under MFI, OEM outperformed JML-itpar at N = 250 but JML-itpar produced lower RMSE values at N = 500 and N = 1,000. This reversal occurred because OEM’s a parameter RMSE under MFI showed diminishing improvement at larger sample sizes, whereas JML-itpar continued to benefit. Under D-optimal value design, OEM was slightly better in most conditions, though the differences were small and three conditions favored JML-itpar. Under Bayesian D-optimal design, OEM showed an advantage at N = 250 and N = 500, while JML-itpar was comparable or slightly better at N = 1,000 in some conditions.

These results suggest that OEM is clearly preferable when the primary concern is difficulty parameter estimation, but the choice between the two methods is less straightforward when discrimination parameter accuracy is also important. JML-itpar showed a particular strength under MFI at larger sample sizes for the a parameter. This is notable because JML-itpar is computationally simpler: it does not require numerical integration over the ability distribution and involves fewer tuning parameters (e.g., quadrature points). In contexts where computational efficiency matters or where discrimination recovery is a priority, JML-itpar paired with MFI may offer a viable combination. When overall accuracy across both parameters is the primary objective, OEM paired with Bayesian D-optimal design or MFI yielded the best combined performance.

Random Calibration Stage Sample Size

The random calibration stage sample size (n_r = 250, 500, or 1,000 examinees) had a limited effect on the accuracy of pretest item parameter estimates. Under both IRT models, RMSE values for a given item selection method, estimation method, and calibration sample size remained similar across the three n_r levels. Slight differences were observed at small calibration sample sizes (N = 250), where a larger random stage occasionally produced marginally different RMSE values, but no systematic trend favoring any particular n_r level emerged. As described in the “Method” section, the random phase contributed approximately 50, 100, or 200 responses per pretest item under the three n_r conditions, and the adaptive phase supplied the remainder of the per-item calibration sample. The random-phase share of the total therefore ranged from 5% (n_r = 250 examinees with N = 1,000) to 80% (n_r = 1,000 examinees with N = 250), with the adaptive phase contributing the remaining 20% to 95%. Even in the cells where the random phase supplied the largest share of the data, RMSE differences across n_r levels remained small and showed no systematic direction. Because item parameters were re-estimated after every 10 new responses during the adaptive phase, differences in the quality of the initial estimates were washed out early in the adaptive phase, regardless of how large a share of the data the random phase contributed. This finding is consistent across both IRT models and both estimation methods, and it suggests that practitioners can select a moderate random stage sample size without a meaningful loss in calibration accuracy.

Calibration Sample Size

The calibration sample size per pretest item (N) was the most influential factor in the simulation. Across all conditions, increasing N from 250 to 500 to 1,000 led to substantial reductions in RMSE for both parameters under the 2-PL model and for the b parameter under the 1-PL model. Under the 2-PL model with JML-itpar, the average RMSE of a decreased from approximately 0.21 at N = 250 to 0.17 at N = 500 and to 0.14 at N = 1,000 across item selection methods (Table 2). For b, the corresponding decrease was from approximately 0.33 to 0.26 to 0.21 under conditions where D-optimal value design was used, with similar proportional decreases observed for MFI and Bayesian D-optimal design. Under OEM, the same general trend held, and the RMSE at N = 1,000 reached values below 0.18 for a and below 0.18 for b in most conditions. The improvement from N = 250 to N = 500 was typically larger than from N = 500 to N = 1,000, indicating diminishing returns at higher sample sizes.

Under the 1-PL model, a parallel pattern was observed. JML-itpar’s b parameter RMSE decreased from approximately 0.16 at N = 250 to 0.08 at N = 1,000 under D-optimal value design, and from approximately 0.16 to 0.11 under MFI. OEM showed steeper declines across sample sizes, reaching RMSE values near 0.077 at N = 1,000 across all item selection methods.

The consistent effect of calibration sample size highlights that collecting a sufficient number of responses per pretest item is the single most effective strategy for improving calibration accuracy, regardless of which item selection method or estimation approach is employed.

Calibration Efficiency

The cumulative sample size, defined as the total number of examinees required to complete the calibration of all 25 pretest items, served as the measure of calibration efficiency (Figure 3). This measure was virtually identical for JML-itpar and OEM within each combination of item selection method and sample size conditions, indicating that the estimation method does not influence the speed at which items accumulate responses; item assignment is governed entirely by the selection method.

Figure 3.

Average cumulative sample size required to calibrate all 25 pretest items under the 1-PL and 2-PL models. Solid bars represent JML-itpar (shown as JML in the legend); hatched bars represent OEM. Values are averaged across random calibration stage sample sizes (n_r = 250, 500, 1,000).

Note. JML-itpar = fixed-ability joint maximum likelihood; OEM = one expectation-maximization cycle; 1-PL = one-parameter logistic; 2-PL = two-parameter logistic.

D-optimal value design required the smallest cumulative sample size across all conditions under both IRT models (Figure 3). For example, at N = 1,000 with n_r = 250 under the 2-PL model, D-optimal value design required an average of approximately 5,026 examinees to calibrate all items, compared to approximately 5,367 for MFI and 5,114 for Bayesian D-optimal design. This advantage was consistent across sample size levels but most apparent at larger calibration sample sizes, where the absolute differences were larger. Under the 1-PL model, a similar ordering was observed, with D-optimal value design again the most efficient and MFI the least efficient.

The efficiency advantage of D-optimal value design can be attributed to its tendency to distribute examinees across pretest items in a way that the items complete calibration at roughly similar rates. MFI, being examinee-centered, tends to direct examinees toward whichever pretest item provides the most information at that individual’s ability level, which can lead to uneven exposure: some items may accumulate responses rapidly while others lag, prolonging the total calibration period. Bayesian D-optimal design partially addresses this imbalance by favoring items that would gain the most from the current examinee’s response relative to what has already been accumulated, which explains its intermediate efficiency.

Taken together, the efficiency results present a practical trade-off. D-optimal value design completes calibration with fewer examinees but, as noted above, produces less accurate difficulty estimates than MFI. Bayesian D-optimal design offers a balance: its cumulative sample size is close to that of D-optimal value design, and its parameter accuracy, particularly for b under OEM, is closer to MFI than to D-optimal value design. The choice among methods therefore depends on whether accuracy or efficiency is the higher priority in a given operational context.

Discussion

The present study examined the core components of online calibration in unidimensional CAT through a comprehensive Monte Carlo simulation that crossed three pretest item selection methods, two parameter estimation methods, three levels of random calibration stage sample size, and three levels of calibration sample size under the 1-PL and 2-PL IRT models. The results are discussed in relation to the four research questions, compared with prior findings, and situated within recent developments in the online calibration literature.

Comparison of Item Selection Methods

The first research question asked how the three pretest item selection methods compare in terms of parameter estimation accuracy and calibration efficiency. The results revealed a trade-off between the methods that depended on which parameter was being estimated. MFI produced the best difficulty parameter recovery across nearly all conditions, while D-optimal value design and Bayesian D-optimal design performed better for the discrimination parameter, particularly when paired with OEM at larger calibration sample sizes. This parameter-level trade-off has not been explicitly documented in prior online calibration studies, likely because most previous work (e.g., Ban et al., 2001; Zheng & Chang, 2017) did not separate parameter-level performance across multiple item selection methods within a single simulation design.

The trade-off has a clear theoretical basis in the objectives of each selection criterion. MFI selects the pretest item that maximizes Fisher information at the examinee’s estimated ability, effectively matching examinees to items near their difficulty level and benefiting b parameter estimation. D-optimal value design and Bayesian D-optimal design optimize the determinant of the full information matrix, which simultaneously accounts for both the discrimination and difficulty components. This joint optimization favors a parameter recovery because estimating discrimination requires examinees with abilities spread across a wider range around the item difficulty, whereas estimating difficulty benefits most from examinees concentrated near the item’s difficulty level (Bjermo et al., 2025). Bjermo et al. (2025) provided a formal demonstration of this asymmetry: they showed that D-optimal designs yield greater precision gains for the discrimination parameter than for the difficulty parameter relative to random assignment, because the D-criterion allocates examinees in a way that benefits the estimation of the full item characteristic curve rather than just its location. Our empirical results are consistent with this theoretical analysis.

The finding that adaptive item selection methods produced differentiated performance contrasts with Zheng and Chang (2017), who concluded that random selection was comparable to or more efficient than the adaptive methods they tested. Several design differences may explain this discrepancy. Zheng and Chang (2017) used Bayesian MEM as their estimation method, tested three IRT models including the 3-PL, and employed calibration sample sizes up to 7,500. The present study used OEM and a fixed-ability ML approach (JML-itpar), focused on the 1-PL and 2-PL models, and used calibration sample sizes up to 1,000 that reflect operational settings with continuous item replenishment. Under smaller sample sizes, the efficient allocation of examinees to pretest items becomes more consequential, and differences among the adaptive methods are more likely to emerge. This interpretation is consistent with He et al. (2020), who found that the differences among adaptive designs (their D-VR design, corresponding to our D-optimal value design, and their proposed excellence degree (ED) designs) were modest in terms of calibration accuracy but more pronounced in efficiency. The present findings extend this observation by showing that the accuracy differences among methods, while small in absolute terms, follow systematic patterns that depend on the parameter being estimated and the estimation method being used.

Bayesian D-optimal design, first introduced for online calibration by van der Linden and Ren (2015), emerged as the most balanced method in the present study. van der Linden and Ren (2015) proposed the Bayesian version of D-optimality to incorporate uncertainty about item parameters accumulated from previous examinees when selecting the next pretest item for the current examinee. By weighing the expected contribution of a new observation against the information already accumulated, Bayesian D-optimal design inherently favors items that have received less informative data, which helps distribute examinees more evenly across pretest items than standard D-optimal value design. In our results, Bayesian D-optimal design’s difficulty parameter RMSE fell between MFI and D-optimal value design, while its discrimination parameter recovery was competitive. Its cumulative sample size was closer to D-optimal value design than to MFI, indicating good calibration efficiency. For practitioners who need a single item selection method that performs adequately across both parameters without requiring excessive examinees, Bayesian D-optimal design offers a practical default.

Under the 1-PL model, where only the b parameter is estimated, D-optimal value design consistently produced the lowest RMSE for JML-itpar. This reversal from the 2-PL pattern occurs because when only one parameter is being estimated, the D-optimal criterion reduces to maximizing Fisher information for that single parameter, aligning its behavior with MFI. This observation illustrates how the relative advantage of item selection methods depends on the number and type of parameters being estimated and underscores the importance of evaluating methods under multiple IRT models.

JML-itpar as a Parameter Estimation Method

The second research question asked whether JML-itpar is a viable parameter estimation method for online calibration and under what conditions it competes with OEM. This study represents the first systematic evaluation of fixed-ability ML (JML-itpar) in a standard online calibration framework, and the results demonstrate that JML-itpar is feasible and competitive under specific conditions.

For the discrimination parameter under the 2-PL model, JML-itpar paired with MFI produced lower RMSE than OEM at calibration sample sizes of 500 and 1,000. This result was unexpected given OEM’s established status in the literature. Ban et al. (2001) recommended OEM (and especially MEM) as the preferred estimation method, and subsequent studies (He et al., 2017, 2020; Zheng, 2014; Zheng & Chang, 2017) have consistently used OEM or MEM as benchmark methods. The present finding does not contradict these recommendations for the difficulty parameter, where OEM was clearly superior in our study as well. Rather, it reveals a parameter-specific advantage of JML-itpar that prior work could not detect because JML-itpar had not been included as a comparison method.

The reversal in a parameter RMSE at larger sample sizes can be understood through the bias results. Both estimation methods produced systematic negative bias in the discrimination parameter, but OEM showed larger underestimation (bias from −0.086 to −0.137) than JML-itpar (bias from −0.055 to −0.085). Despite this larger bias, OEM generally achieved lower RMSE at small sample sizes, indicating lower variance. As sample size increased, the variance component of both methods decreased, but the bias component remained relatively stable, giving JML-itpar an advantage at N = 500 and N = 1,000 under MFI. He et al. (2017) reported a similar observation in a different context: they found that OEM’s integration over the ability distribution can introduce systematic error in discrimination estimates, and proposed MLE-LBCI-Method A partly to address this issue. The bias-variance trade-off observed here between JML-itpar and OEM warrants further investigation at the item parameter level, including analyses by item difficulty and discrimination ranges, and is planned as a focused follow-up study.

The implementation of JML-itpar in this study treated examinee abilities estimated from operational items (via WLE) as fixed values and estimated only the item parameters through Newton-Raphson iteration. This fixed-ability ML approach is related to method A (Stocking, 1988), which also treats abilities as known, but differs in that it uses full iterative optimization rather than a single estimation step. The critical distinction from classical JML, which simultaneously estimates both person and item parameters (Neyman & Scott, 1948), is that the incidental parameter problem does not apply here: each pretest item is calibrated against examinees whose abilities are treated as known constants, not as parameters to be estimated. The consistent improvement in RMSE with increasing calibration sample size confirms that this implementation does not suffer from the inconsistency that characterizes classical JML, where parameter accuracy fails to improve as sample size increases (Holland, 1990).

Verschoor et al. (2019) used JML for on-the-fly calibration of entire item pools under the Rasch model. They reported that JML converged faster than MML initially but was outperformed as observations accumulated and proposed combining Elo ratings with JML and using reference items to correct for bias. The present study differs in scope: JML-itpar was applied to individual pretest items within a two-phase online calibration framework, and the 2-PL model was included. Our results complement their work by demonstrating that fixed-ability ML produces competitive discrimination parameter estimates in the standard online calibration setting, a dimension not examined in their Rasch-only design.

The practical recommendation for estimation method selection depends on the testing context. When difficulty parameter accuracy is the primary concern, or when both parameters are equally important, OEM is the preferred choice. When discrimination parameter recovery is a priority, computational simplicity is valued, or the testing program employs a 1-PL model, JML-itpar offers a simpler and competitive alternative. JML-itpar requires no numerical integration over the ability distribution, involves no tuning parameters such as quadrature points, and is straightforward to implement, making it an attractive option for smaller testing programs or contexts where computational resources are limited.

Effects of Sample Size

The third and fourth research questions addressed the effects of random calibration stage sample size and calibration sample size per pretest item. The random stage sample size (n_r = 250, 500, or 1,000 examinees) had a negligible effect on parameter recovery across all conditions, even though the random phase supplied anywhere from 5% to 80% of the per-item calibration sample across the crossed conditions. This finding confirms that once the adaptive phase begins, iterative item selection and re-estimation after every 10 new responses compensate for differences in the quality of initial estimates, whatever share of the data the random phase contributed. Kingsbury (2009) noted that the initial seeding phase primarily serves to accumulate a minimum number of responses, and our results support this view. Practitioners can set the random stage sample size at a moderate level (e.g., n_r = 250 examinees, which yields roughly 50 responses per pretest item under the present design) without sacrificing calibration accuracy. A short random stage is also desirable operationally, because it limits the number of examinees who receive pretest items with poorly calibrated or unknown parameters.

The calibration sample size per pretest item (N) was the most influential factor in the simulation. RMSE decreased monotonically as N increased from 250 to 1,000, with the largest improvement between N = 250 and N = 500. This finding is consistent across the broader literature: Ban et al. (2001) reported that MEM and OEM both improved with larger samples, Ali and Chang (2014) observed similar trends, and Yuan et al. (2023) confirmed the pattern in the multidimensional polytomous context. The diminishing returns between N = 500 and N = 1,000 suggest that for practical purposes, a calibration sample size of 500 per pretest item provides a reasonable balance between accuracy and the time required to complete calibration. Collecting sufficient responses per pretest item is the single most effective strategy for improving calibration accuracy, regardless of the item selection or estimation method employed.

Calibration Efficiency

The cumulative sample size results showed that D-optimal value design required the fewest examinees to complete calibration, MFI required the most, and Bayesian D-optimal design fell between the two. The estimation method did not affect efficiency, confirming that item assignment is governed entirely by the selection method. D-optimal value design’s efficiency advantage stems from its tendency to distribute examinees more evenly across pretest items, so that items complete calibration at similar rates. MFI directs examinees toward whichever pretest item provides the most information at their ability level, creating uneven exposure patterns that prolong the total calibration period. Zheng and Chang (2017) described a similar tendency for their examinee-centered method, and our CSS results provide quantitative support for this characterization. Bayesian D-optimal design partially corrects this imbalance by incorporating the accumulated information for each pretest item into the selection criterion (van der Linden & Ren, 2015), which explains its intermediate efficiency.

Yuan et al. (2023) reported a consistent finding in the multidimensional polytomous context: their adaptive designs outperformed the random design for discrimination parameters but not for intercept (difficulty) parameters when calibration sample size was sufficient. This parameter-specific advantage of item-centered adaptive designs aligns with our observation that D-optimal value design and Bayesian D-optimal design benefited the a parameter more than the b parameter.

The present findings should also be positioned relative to frameworks that embed online calibration within broader test-assembly and uncertainty-handling machinery. The shadow-test approach of van der Linden and Jiang (2020) selects operational and pretest items jointly under content and exposure constraints, and fully Bayesian frameworks treat ability and item parameters through their joint posterior distribution, propagating item calibration error into ability estimation and item selection (Fink et al., 2025; van der Linden & Ren, 2020). Bjermo et al. (2025) pursued a related goal by adjusting optimal calibration designs for the uncertainty in ability estimates. We did not implement these frameworks. Each can be configured in many ways, involving choices of constraint sets, priors, and sampling algorithms, and evaluating them properly would have complicated the presentation and interpretation of a fully crossed comparison beyond what a single study can support. We instead examined a basic CAT configuration in a clean, fully crossed design, which isolates the effects of item selection method, estimation method, and sample size. The resulting benchmarks provide a baseline against which constraint-aware and uncertainty-aware implementations can be evaluated in future work.

The combined accuracy and efficiency results lead to a practical decision framework. If a testing program prioritizes difficulty parameter accuracy, MFI paired with OEM provides the best results, though at the cost of requiring more examinees. If the program values efficiency or needs to calibrate items quickly with limited examinee flow, D-optimal value design paired with OEM offers the fastest calibration with adequate accuracy. If a balance between accuracy across both parameters and reasonable efficiency is desired, Bayesian D-optimal design paired with OEM offers the most versatile combination. A useful benchmark for judging the practical weight of these differences is the accuracy gain from enlarging the calibration sample. The largest accuracy differences attributable to the choice of selection method at a fixed sample size were of the same order as the gain from doubling the calibration sample from 500 to 1,000 responses per item. Selecting well-matched calibration components can therefore achieve accuracy that would otherwise require a substantially larger calibration sample, which is where the operational significance of the reported differences lies.

Limitations

Several limitations should be considered when interpreting these results. First, the study did not include the 3-PL model. In CAT, the adaptive item selection algorithm matches items to the examinee’s ability level, which means that examinees typically respond to items near their ability. Under these conditions, the probability of guessing is substantially reduced because items are not excessively difficult for the examinee, and the practical influence of the c parameter is minimized. In addition, the sample sizes required for stable estimation of a guessing parameter far exceed realistic online calibration budgets (Akour & AL-Omari, 2013). The 1-PL and 2-PL models therefore provide a reasonable approximation for many CAT applications. That said, the 3-PL model remains common in certain testing programs (e.g., multiple-choice tests with low-ability examinees), and the performance of the methods under the 3-PL model remains an open question. Second, all pretest items were seeded at the end of the operational test, where ability estimates are most stable. Different seeding positions may alter the relative performance of the methods, although Zheng and Chang (2017) reported no effect of seeding location in their design. Third, the simulation used a single fixed test length (35 items), a single operational item bank size (250 items), and specific parameter distributions (a from log-normal, b from standard normal). Different test configurations may yield different results. Fourth, operational constraints such as content balancing, item exposure control, and potential differences in examinee motivation between operational and pretest items were not modeled. Fifth, the ability estimates used as fixed values in the JML-itpar procedure carry measurement error that was not explicitly modeled. Bjermo et al. (2025) demonstrated that accounting for this uncertainty can improve calibration design efficiency, and their uncertainty-adjusted approach represents a promising direction for refining the item selection methods examined here.

Future Directions

The findings point to several directions for future research. The behavior of JML-itpar in online calibration warrants further investigation at the item parameter level, including how bias and RMSE vary as a function of item difficulty and discrimination values, and how convergence patterns differ across item characteristics. Such an analysis would clarify the boundary conditions for JML-itpar and help identify when it represents a sufficient alternative to OEM. The development of bias correction procedures for the fixed-ability ML framework, analogous to Lord’s bias correction applied to method A (He et al., 2017) or the reference item approach of Verschoor et al. (2019), could improve JML-itpar’s difficulty parameter recovery. Incorporating the uncertainty-adjusted optimal design framework of Bjermo et al. (2025) into the comparison of item selection methods would refine the practical recommendations. Extending the comparison to MEM and other multiple-cycle EM approaches, which feed provisional pretest item parameters back into the ability posterior, would test whether the additional information outweighs the sensitivity to poorly calibrated parameters under frequent parameter updating (Ban et al., 2001). Extensions to multidimensional IRT models (Chen, 2017; Yuan et al., 2023) and to continuous calibration frameworks that incorporate response time information (He et al., 2021) would broaden the applicability of these findings. The custom simulation code developed for this study provides a foundation for an open-source R package that would facilitate replication and experimentation with online calibration components.

Conclusion

This study provides a systematic comparison of the core components of online calibration in unidimensional CAT and establishes fixed-ability ML (JML-itpar) as a feasible estimation method in this context. The results show that no single combination of item selection and estimation methods is optimal across all parameters and evaluation criteria. MFI paired with OEM yields the best difficulty parameter accuracy. D-optimal value design and Bayesian D-optimal design, first introduced for online calibration by van der Linden and Ren (2015), offer competitive discrimination parameter recovery at moderate to large calibration sample sizes, particularly when paired with JML-itpar. Bayesian D-optimal design paired with OEM provides a balanced combination of accuracy across both parameters and reasonable calibration efficiency. The calibration sample size per pretest item is the most important determinant of parameter recovery, and a minimum of 500 responses per item is recommended for acceptable accuracy under the 2-PL model. These findings offer practical guidance for CAT practitioners selecting among calibration components and demonstrate that computationally simpler estimation approaches can be competitive with established methods under well-defined conditions.