Nonparametric CAT for CD in Educational Settings With Small Samples

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) has been suggested by researchers as a diagnostic tool for assessment and evaluation. Although model-based CD-CAT is relatively well researched in the context of large-scale assessment systems, this type of system has not received the same degree of research and development in small-scale settings, such as at the course-based level, where this system would be the most useful. The main obstacle is that the statistical estimation techniques that are successfully applied within the context of a large-scale assessment require large samples to guarantee reliable calibration of the item parameters and an accurate estimation of the examinees’ proficiency class membership. Such samples are simply not obtainable in course-based settings. Therefore, the nonparametric item selection (NPS) method that does not require any parameter calibration, and thus, can be used in small educational programs is proposed in the study. The proposed nonparametric CD-CAT uses the nonparametric classification (NPC) method to estimate an examinee’s attribute profile and based on the examinee’s item responses, the item that can best discriminate the estimated attribute profile and the other attribute profiles is then selected. The simulation results show that the NPS method outperformed the compared parametric CD-CAT algorithms and the differences were substantial when the calibration samples were small.

Assessments for cognitive diagnosis (CD) are designed to assess students’ mastery or nonmastery of a set of skills in a knowledge domain with the goal of providing immediate feedback for further remediation. In the past several years, CD has become a new paradigm of educational testing, and CD-related research has drawn substantial attention. Recently, researchers went one step further to develop large-scale cognitive diagnostic computerized adaptive testing (CD-CAT), which features in improving estimation accuracy and testing efficiency. A pioneer is the CD-CAT developed for administering an English language proficiency test to more than 120,000 students in China (Liu, You, Wang, Ding, & Chang, 2013). Another ongoing research program, the Dynamic Learning Maps Alternate Assessment System Consortium, supported by the U.S. Department of Education, plans to develop computerized cognitive-psychometric models to analyze large-scale data collected from students with the most significant cognitive disabilities. These programs confirm and support the finding that the development of CD-CAT as a diagnostic tool for assessment and evaluation should be the primary interest in psychological and educational measurement (e.g., Cheng, 2009; Cheng & Chang, 2007; Liu et al., 2013; K. K. Tatsuoka & Tatsuoka, 1997).

In the CD-CAT framework, an effective and efficient item selection algorithm is the key to the success of a CD-CAT program. To date, various parametric item selection methods have been proposed (see “Technical Background” section for a detailed review). These parametric methods follow a common rationale that they first calibrate the item parameters by fitting a large data set with a cognitive diagnostic model (CDM) and then select the best item for each examinee by maximizing an information function, minimizing the expected Shannon entropy, or maximizing the variability in the probabilities of success of the estimated latent classes.

Although model-based CD-CAT is relatively well-researched in the context of large-scale assessment systems, where data for thousands of examinees are available, this type of system has not received the same degree of research and development in small-scale settings, such as at the course-based level, where this system would be the most useful. There are at least two difficulties in adapting the current model-based CD-CAT to a small-scale setting. First, the statistical estimation techniques that are successfully applied within the context of a large-scale assessment require large samples to guarantee reliable calibration of the item parameters and an accurate estimation of the examinees’ proficiency class membership. Such samples are simply not obtainable in course-based settings. In most, if not all, published CD-CAT papers on the topic of parametric item selection methods, the true item parameters are used in simulations without calibration to assess the performance of the methods, and thus, no error from item parameter calibration is taken into account (Cheng, 2009; Kaplan, de la Torre, & Barrada, 2015). This setup is unrealistic, particularly, in a small-scale setting, where the calibration of the item parameters could be unstable and thus, the resulting estimates unreliable. Second, item quality is a built-in criterion for these parametric methods, and only items of the best quality in terms of guessing and slipping parameters are selected. Such a routine guarantees optimal estimation accuracy, but results in high exposure rates for certain items and consequently, sacrifices the test security. The trade-off between estimation accuracy and test security is well-known (Ponsoda & Olea, 2003; Zheng & Wang, 2017); however, in the studies where these parametric methods were developed, the estimation accuracy was evaluated without controlling the item exposure (Cheng, 2009; Kaplan et al., 2015) and therefore, the findings did not reflect the true performance of the parametric methods in practice. Zheng and Wang (2017) had the same concern and developed an item selection method with the binary searching algorithm for item exposure control. However, their method was compared only with the posterior-weighted Kullback–Leibler (PWKL) method, and a thorough evaluation of the degree of item overexposure the other parametric methods could encounter without controlling item exposure remains unavailable.

In response to these issues, the nonparametric item selection (NPS) method that first uses the nonparametric classification (NPC) method (Chiu & Douglas, 2013) to classify examinees and then identifies the best item for each examinee individually by maximizing the discrimination power of the items is proposed in this study to develop a nonparametric CD-CAT. Differing from the existing parametric methods for CD-CAT that require a large sample to secure accurate item parameter calibration, the proposed method is free of any item calibration and thus, is favored in a setting where the calibration sample is small or even unavailable.

The remainder of the article is organized as follows. “Technical Background” section gives a brief review of the CDMs and the methods used in the study, followed by “NPS Method for CD-CAT” section where the key concepts and the algorithm of the NPS method are explained in detail. “Simulation Studies” section includes two simulation studies for assessing the performance of the proposed method in comparison with several existing parametric methods under various conditions. The article is concluded with several suggestions and future research directions.

Technical Background

CDMs

The section begins with the key definitions and notations used in the article. The construct of interest in CD is referred to as “attribute,” which is binary with 1 and 0 indicating mastery and nonmastery, respectively. Suppose at most $K$ attributes are required by the items in the test. The latent attribute profile of examinee $i$ is a $K\times 1$ vector denoted as ${\mathit{\alpha }}_i=({\mathrm{\alpha }}_{i1},{\mathrm{\alpha }}_{i2},\dots ,{\mathrm{\alpha }}_{\mathbf{iK}}{)}^{\mathbf{T}}$. The latent space expanded by the $K$ attributes thus contains $2^K$ latent proficiency classes, and the ultimate goal of CD is to assign examinees to the proficiency class to which they belong. The assignments are typically done by fitting the data with a CDM, which is also referred to as diagnostic classification models (DCMs).

A common component of CD modeling is the prespecified Q-matrix (K. K. Tatsuoka, 1985), which indicates the associations between the test items and the required attributes. Specifically, suppose a test consists of $J$ items. A $J\times K$ Q-matrix is specified along with the test such that the entry $(j,k)$, denoted as $q_{\mathrm{jk}}$, is 1 if Item $j$ requires attribute $k$ and 0 otherwise. Given the Q-matrix, the examinees’ class memberships can be estimated by fitting the data with a prespecified CDM. Various CDMs have been developed, and those used in the study are briefly reviewed in the following.

The DINA model

The DINA model (Junker & Sijtsma, 2001; MacReady & Dayton, 1977) is perhaps the most frequently researched CDM due to its simplicity. Let ${\mathit{Y}}_i=(Y_{i1},\dots ,Y_{\mathrm{iJ}})$ be the observed item response pattern for examinee $i$. The item response function of the DINA model is defined as

$$P(Y_{\mathrm{ij}}=1|{\mathit{\alpha }}_i)={(1-s_j)}^{{\mathrm{\eta }}_{\mathrm{ij}}}{g}_j^{(1-{\mathrm{\eta }}_{\mathrm{ij}})},$$ (1)

where $s_j$ and $g_j$ are the slipping and guessing parameters, respectively, for Item $j$, assuming $0<g_j<1-s_j<1$. In addition, ${\mathrm{\eta }}_{\mathrm{ij}}$, often referred to as the ideal item response, is defined as $\underset{k\in A_j}{\Pi }{\mathrm{\alpha }}_{\mathrm{ik}}$ where $A_j=\{k|q_{\mathrm{jk}}=1\}$ for Item $j$. ${\mathrm{\eta }}_{\mathrm{ij}}$ indicates whether examinee $i$ masters all the required attributes of Item $j$ and thus, makes the DINA model a typical conjunctive model. Equation 1 also shows that ${\mathrm{\eta }}_{\mathrm{ij}}$ is the most likely item response from examinee $i$ to Item $j$ when $s_j$ and $g_j$ are small.

The reduced reparameterized unified model (RUM)

The reduced RUM (Hartz, 2002; Hartz & Roussos, 2008) is another CDM that has received considerable attention among psychometric researchers. The item response function of the reduced RUM is defined as

$$P(Y_{\mathrm{ij}}=1|{\mathit{\alpha }}_i)={\mathrm{\pi }}_j^{*}\underset{k\in A_j}{\Pi }{r}_{\mathrm{jk}}^{*(1-{\mathrm{\alpha }}_{\mathrm{ik}})},$$ (2)

where $0<{\mathrm{\pi }}_j^{*}<1$ denotes the probability of a correct answer for an examinee who has mastered all the attributes required by Item $j$, and $0<r_{\mathrm{jk}}^{*}<1$ is a penalty parameter that reduces the probability of a correct response by a factor of $r_{\mathrm{jk}}^{*}$ for those who do not possess attribute $k$. The reduced RUM relaxes the restricted conjunction rule used in the DINA model and allows different probabilities of success for different proficiency classes based on the number of attributes possessed.

The general CD models

The DINA model and the reduced RUM are two examples of the restricted CDMs subsumed in general CDMs, such as the general diagnostic model (GDM; von Davier, 2005, 2008), the loglinear cognitive diagnosis model (LCDM; Henson, Templin, & Willse, 2009), and the generalized deterministic inputs, noisy “and” gate (G-DINA) model (de la Torre, 2011). Taking the G-DINA model as an example, its item response function can be expressed by relating the probability of a correct response and the model specifications through one of several possible link functions, such as the identity, logit, or log link. For succinctness, suppose Item $j$ requires $K_j^{*}\le K$ attributes that, without loss of generality, have been moved to the first $K_j^{*}$ positions of the q-vector $q_j$. Furthermore, let ${\mathrm{\alpha }}_{\mathrm{lj}}^{*}$ be the reduced attribute profile that consists of the first $K_j^{*}$ attributes, for $l=1,\dots ,2^{K_j^{*}}$. The saturated form of the G-DINA model is formulated using the identity link, and its item response function can be written as

$$P(Y_{\mathrm{ij}}=1|{\mathrm{\alpha }}_{\mathrm{lj}}^{*})={\mathrm{\beta }}_{j0}+\sum _{k=1}^{K_j^{*}}{\mathrm{\beta }}_{\mathrm{jk}}{\mathrm{\alpha }}_{\mathrm{lk}}+\sum _{k\prime >k}^{K_j^{*}}\sum _{k=1}^{K_j^{*}-1}{\mathrm{\beta }}_{\mathrm{jkk}\prime }{\mathrm{\alpha }}_{\mathrm{lk}}{\mathrm{\alpha }}_{\mathrm{lk}\prime }+\cdots +{\mathrm{\beta }}_{j(12\dots K_j^{*})}\underset{k=1}{\overset{K_j^{*}}{\Pi }}{\mathrm{\alpha }}_{\mathrm{lk}},$$ (3)

where ${\mathrm{\beta }}_{j0}$ denotes the intercept of Item $j$, ${\mathrm{\beta }}_{\mathrm{jk}}$ the main effect due to ${\mathrm{\alpha }}_{\mathrm{lk}}$, ${\mathrm{\beta }}_{\mathrm{jkk}\prime }$ the interaction effect of ${\mathrm{\alpha }}_{\mathrm{lk}}$ and ${\mathrm{\alpha }}_{\mathrm{lk}\prime }$, and ${\mathrm{\beta }}_{j(12\dots K_j^{*})}$ the interaction effects of ${\mathrm{\alpha }}_{l1},\dots ,{\mathrm{\alpha }}_{{\mathrm{lK}}_j^{*}}$. With careful parameterization, the restricted CDMs can be respecified in terms of the general CDMs.

The NPC Method

The NPC method (Chiu & Douglas, 2013) was used in the study to obtain the estimates of examinees’ attribute profiles in the CD-CAT process. The NPC method classifies examinees by evaluating the distance between the observed and ideal item responses. Specifically, for the $2^K$ possible attribute profiles, the corresponding ideal response patterns are denoted as ${\mathbf{\eta }}^{(1)},{\mathbf{\eta }}^{(2)},\dots ,{\mathbf{\eta }}^{{(2}^K)}$, where ${\mathbf{\eta }}^{(m)}=({\mathbf{\eta }}_1^{(m)},\dots ,{\mathbf{\eta }}_J^{(m)})$ for $m=1,\dots ,M=2^K$. The examinee’s attribute profile is then estimated by minimizing the distance between the observed and ideal item responses, $d({\mathit{Y}}_i,{\mathbf{\eta }}^{(m)})$, where $m=1,\dots ,M$. For binary data, a natural and frequently used distance measure is the Hamming distance, which simply counts the number of times that the entries in two vectors disagree, defined as

$$d_h({\mathit{Y}}_i,{\mathbf{\eta }}^{\left(m\right)})=\sum _{j=1}^J|Y_{\mathrm{ij}}-{\mathrm{\eta }}_j^{(m)}|.$$ (4)

Then the estimator of the proficiency class, ${\hat{\mathit{\alpha }}}_i$, is obtained by,

$${\hat{\mathit{\alpha }}}_i=\underset{{\mathit{\alpha }}_m\in \{{\mathit{\alpha }}_1,\dots ,{\mathit{\alpha }}_{2^{\mathit{K}}}\}}{\mathrm{argmin}}{d}_h({\mathit{Y}}_i,{\mathbf{\eta }}^{(m)}).$$ (5)

As an important theoretical result, S. Wang and Douglas (2015) proved that assuming $P(Y_j=1|{\mathrm{\eta }}_j=0)<0.5$, $P(Y_j=1|{\mathrm{\eta }}_j=1)>0.5$ for Item $j$, and the Q-matrix is complete (Chiu, Douglas, & Li, 2009; Köhn & Chiu, 2017), the estimator of the attribute profile obtained by using the NPC method is statistically consistent, provided a long test with some regulations on the number of examinees. Chiu and Douglas (2013) also pointed out some remarkable advantages of the NPC method. First, the method is computationally straightforward and inexpensive. Second, the resulting classification rates are almost as high as those obtained using parametric estimation methods. Third, the method can be used with data conforming to a variety of CDMs under the two assumptions provided above. Most importantly, there is no restriction on the sample size. The method is efficient and effective when samples are large; it can also be used even when the sample size is 1.

Parametric Item Selection Methods in CD-CAT

Item selection methods play a central role in CAT procedures. In traditional item response theory (IRT), the Fisher information is perhaps the most commonly used item selection method; however, it is no longer applicable to CDMs because the person parameter (i.e., the attribute profile $\mathit{\alpha }$) is discrete. Thus, other information functions and indexes have been developed, and those used in the study are briefly reviewed in the following.

Shannon entropy-based approach

The SHE (Shannon, 1948; C. Tatsuoka, 2002) algorithm for CD-CAT aims to minimize the variability of the posterior distribution of attribute profile $\mathit{\alpha }$. Suppose $t$ items have been given to examinee $i$. For Item $j$ in the item bank, its expected Shannon entropy for being item $(t+1)$ is expressed as

$$\mathrm{SH}{\mathrm{E}}_j=E\left[H\left({\mathrm{\pi }}_i^{(t+1)}\left(\mathit{\alpha }\right)\right)\right],$$ (6)

where the entropy $H$ is in general defined as $H(f)=-\sum _{x}f(x)\log f(x)$ and ${\mathrm{\pi }}_i^{(t+1)}(\mathit{\alpha })=P(\mathit{\alpha }|y_i^{(1)},\dots ,y_i^{(t)},Y_i^{(t+1)})$ is the posterior probability of ${\mathit{\alpha }}_i$ given the first $(t+1)$ item responses. Because the $(t+1)$th item has not been administered, it is denoted as $Y_i^{(t+1)}$ and treated as a random variable. The expectation of the entropy $H$, thus, is taken with respect to $Y_i^{(t+1)}$. The item associated with the least $\mathrm{SHE}$ is then chosen as the best item for examinee $i$.

The PWKL index

In addition to the SHE method, another frequently referenced method is the PWKL index (Cheng, 2009), which belongs to the family of the Kullback–Leibler (KL) information-based approaches. The PWKL index is a variation that improves the KL information by taking the KL distance and the posterior probability of ${\mathit{\alpha }}_m$ into account. In other words, the PWKL index of Item $j$ at cycle $t$ for examinee $i$ is expressed as

$${\mathrm{PWKL}}_j^{\left(t\right)}=\sum _{m=1}^{2^K}{D}_j(f(y|{\hat{\mathit{\alpha }}}_i)\parallel f(y|{\mathit{\alpha }}_m)){\mathrm{\pi }}_i^{\left(t\right)}\left({\mathit{\alpha }}_m\right),$$ (7)

where ${\mathrm{\pi }}_i^{(t)}({\mathit{\alpha }}_m)=P({\mathit{\alpha }}_m|y_i^{(1)},\dots ,y_i^{(t)})$ is the probability of examinee $i$ in proficiency class $m$, given the first $t$ observed item responses. When the PWKL index is used, the goal is to select an item as discriminating as possible so that the distributions of the response variable given examinees in different proficiency classes are far apart from each other or in technical terms, to select an item that maximizes the PWKL index.

As a side note, the mutual information (MI) item selection method (C. Wang, 2013) can be seen as a particular case of KL divergence (C. Wang, Chang, & Huebner, 2011). For readers who are interested in the method, please refer to C. Wang (2013) for more details.

The posterior-weighted cognitive diagnostic model discrimination index (PWCDI)

CDM discrimination index (CDI) was first proposed by Henson and Douglas (2005) to facilitate test construction for CD purposes. Later on, Zheng and Chang (2016) extended it to the context of CD-CAT and developed the PWCDI for select items. The index for the $j$th item is defined as

$$\mathrm{PWCD}{\mathrm{I}}_j=\frac{1}{\sum _{m\ne m\prime }{d}_h{({\mathit{\alpha }}_m,{\mathit{\alpha }}_{m\prime })}^{-1}}\sum _{m\ne m\prime }{d}_h{({\mathit{\alpha }}_m,{\mathit{\alpha }}_{m\prime })}^{-1}{\mathrm{\pi }}^{(t)}\left({\mathit{\alpha }}_m\right){\mathrm{\pi }}^{(t)}\left({\mathit{\alpha }}_{m\prime }\right)\mathrm{KL}({\mathit{\alpha }}_{m\prime }||{\mathit{\alpha }}_m),$$ (8)

where $d_h({\mathit{\alpha }}_m,{\mathit{\alpha }}_{m\prime })=\sum _{k=1}^K|{\mathit{\alpha }}_{\mathrm{mk}}-{\mathit{\alpha }}_{m\prime k}|$ is the Hamming distance between attribute profiles ${\mathit{\alpha }}_m$ and ${\mathit{\alpha }}_{m\prime }$. The best item is selected by maximizing the $\mathrm{PWCD}{\mathrm{I}}_j$.

Dynamic and stratified dynamic binary searching methods

The DBS method was first proposed by C. Tatsuoka and Ferguson (2003) as a heuristic named in their paper the halving algorithm, and later renamed by Zheng and Wang (2017) as the DBS method. The index for examinee $i$ to Item $j$ is defined as

$$B_{\mathrm{ij}}=|\sum _{{\mathrm{\eta }}_j^{\left(m\right)}=1}{\mathrm{\pi }}_i^{\left(t\right)}\left({\mathit{\alpha }}_m\right)-0.5|,$$

and the best item is selected by minimizing this index. It should be noted that the DBS method, as the NPS method, also benefits from the randomization mechanism (Georgiadou, Triantafillou, & Economides, 2007) for item exposure control.

Inspired by an IRT-based CAT technique (Chang & Ying, 1999), Zheng and Wang (2017) modified the method by stratifying the item bank according to item discrimination (Rupp, Templin, & Henson, 2010), and proposed the stratified dynamic binary searching (SDBS) method for fixed-length CD-CAT.

G-DINA model discrimination index (GDI)

The GDI was first proposed by de la Torre and Chiu (2016) as an index for validating a Q-matrix, and the GDI for Item $j$ is defined as

$${\mathrm{\varsigma }}_j^2=\sum _{l=1}^{2^{K_j^{*}}}{(P(Y_{\mathrm{ij}}=1|{\mathit{\alpha }}_{\mathrm{lj}}^{*})-{\overline{P}}_j)}^{2}P({\mathit{\alpha }}_{\mathrm{lj}}^{*}|\mathit{y}),$$ (9)

where $P({\mathit{\alpha }}_{\mathrm{lj}}^{*}|\mathit{y})$ is the posterior probability of ${\mathit{\alpha }}_{\mathrm{lj}}^{*}$ and ${\overline{P}}_j=\sum _{l=1}^{2^{K_j^{*}}}P({\mathit{\alpha }}_{\mathrm{lj}}^{*}|\mathit{y})P(Y_{\mathrm{ij}}=1|{\mathit{\alpha }}_{\mathrm{lj}}^{*})$ is the mean success probability. Because the GDI measures the ability of an item to discriminate the reduced proficiency classes, in the context of validating Q-matrix, the correct q-vector is chosen by maximizing the GDI.

Kaplan et al. (2015) later extended it to CD-CAT as an item selection index with a modification that $P({\mathit{\alpha }}_{\mathrm{lj}}^{*}|\mathit{y})$ is replaced by the posterior probability of ${\mathit{\alpha }}_{\mathrm{lj}}^{*}$ at cycle $t$ for examinee $i$, denoted as ${\mathrm{\pi }}_i^{(t)}({\mathit{\alpha }}_{\mathrm{lj}}^{*})=P({\mathit{\alpha }}_{\mathrm{lj}}^{*}|{\mathit{y}}_i^{(t)})$. Denote the modified GDI of Item $j$ for examinee $i$ at cycle $t$ as ${\mathrm{\varsigma }}_{\mathrm{ij}}^{2(t)}$; the best item is obtained by maximizing ${\mathrm{\varsigma }}_{\mathrm{ij}}^{2(t)}$.

NPS Method for CD-CAT

The proposed NPS method first uses the NPC method (Chiu & Douglas, 2013) to estimate examinees’ attribute profiles and then selects an item that can best differentiate an examinee’s estimated attribute profile from the other attribute profiles by evaluating the distance between the examinee’s observed and ideal response patterns. After the examinee responds to the selected item, the estimate of his or her attribute profile is then updated using the NPC method again, and another item that can better pinpoint the examinee’s attribute profile is further selected following the same item selection procedure just described. The algorithm continues until the stopping rule is met. In the rest of the section, two key concepts of the method will be discussed first, followed by a presentation of the algorithm of the NPS method.

Key Concepts

Two key concepts need to be elaborated first to support the legitimacy of the NPS method as an item selection method in a CD-CAT system. The first key concept regards the Q-optimal criterion used from Step 1 to Step 5 in the algorithm. To optimize the classification of examinees, the Q-optimal criterion proposed by Xu, Wang, and Shang (2016), rather than a random selection, was adopted to choose the first $K$ items. Suppose $\mathit{\alpha }$ is the true attribute profile of examinee $i$, and $\mathit{\alpha }\prime$ is a different attribute profile. A Q-matrix is “Q-optimal” in terms of classification accuracy for examinee $i$ if

$$P({\hat{\mathit{\alpha }}}_i=\mathit{\alpha }|\mathit{\alpha },{\mathbf{Q}}_i)>\underset{\mathit{\alpha }\prime \ne \mathit{\alpha }}{\max }P({\hat{\mathit{\alpha }}}_i=\mathit{\alpha }\prime |\mathit{\alpha },{\mathbf{Q}}_i).$$ (10)

It has been proven (Xu et al., 2016) that when data conform to the DINA model, a Q-matrix of the first $K$ items is Q-optimal for examinee $i$ if and only if the $K$ q-vectors have the following lower triangular matrix structure, with or without column swapping,

$${\mathit{Q}}_i={\left(\begin{array}{ccccc}1\hfill & 0\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{q_{21}}& 1\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& \ddots \hfill & \ddots \hfill & \ddots \hfill & ⋮\hfill \\ \multicolumn{1}{c}{q_{(K-1)1}}& \cdots \hfill & q_{(K-1)(K-2)}\hfill & 1\hfill & 0\hfill \\ \multicolumn{1}{c}{q_{K1}}& \cdots \hfill & q_{K(K-2)}\hfill & q_{K(K-1)}\hfill & 1\hfill \end{array}\right)}_{K\times K},$$ (11)

where the diagonals are 1s, and the entry $q_{k\prime k}$, for $k=1,\dots ,K-1$ and $k\prime =k+1,\dots ,K$, takes the value $0$ if the $K$th item is answered incorrectly by the examinee $i$ and can take either $0$ or $1$, otherwise.

However, the Q-optimal condition in Equation 11 holds only for the DINA model. For other complex CDMs, more assumptions need to be satisfied. Unfortunately, these assumptions could turn to a number of stringent and thus, unrealistic constraints if the Q-optimal condition in Equation 11 is to be applied to a more complex CDM. Another alternative is to use only the single-attribute items. This initial Q-matrix guarantees the Q-optimality for data conforming to any model but loses the flexibility Equation 11 provides. Considering the pros and cons, the Q-optimal criterion in Equation 11 without extra constraints is used in the simulation to select the initial $K$ items although multiple CDMs could underlie the initial $K$ items. As a consequence, the results should be interpreted with the caution that the Q-optimality may be achieved only partially if any of the first initial $K$ items conforms to a CDM different from the DINA model.

The second key concept concerns the search algorithm used in Steps 6 and 11 of the NPS algorithm for finding the best item for each examinee. Instead of using the complete search algorithm where an item is selected if it can best distinguish $\hat{\mathit{\alpha }}$ from all the other attribute profiles, the NPS method adopts a more economic algorithm by assessing only the separability of $\hat{\mathit{\alpha }}$ from the “second likely” attribute profile, denoted as $\tilde{\mathit{\alpha }}$ in the algorithm. The underlying rationale is that $\tilde{\mathit{\alpha }}$ is the attribute profile presumably most difficult to be separated from $\hat{\mathit{\alpha }}$, compared with the other $2^K-2$ attribute profiles; once $\tilde{\mathit{\alpha }}$ can be separated from $\hat{\mathit{\alpha }}$, the other less competitive attribute profiles will either be further “pushed” away or remain distant from $\hat{\mathit{\alpha }}$, and thus, there is no need to explore the separability of $\hat{\mathit{\alpha }}$ from the other attribute profiles.

The graphs in Figure 1 are used as examples to illustrate the idea under different conditions. The solid trajectory represents the distance corresponding to $\mathit{\alpha }$ and the dashed trajectories represent the others. Given that $\mathit{\alpha }$ is the correct estimate of the attribute profile, the left graph shows that the distance corresponding to $\mathit{\alpha }$ remains the minimum along the way and the other incorrect estimates become farther and farther away from $\mathit{\alpha }$. However, it could happen that the attribute profile is incorrectly estimated in the early stage of the estimation. The right graph of Figure 1 further illustrates that the NPS algorithm is able to correct the error and eventually obtain the correct estimate.

Figure 1.

Examples illustrating the trajectories of distances using the NPS Algorithm.

Note. NPS = nonparametric item selection.

Algorithm

In the NPS algorithm, the first $K$ items are selected based on the Q-optimal criterion (Xu et al., 2016) to yield the optimal classification results. After the first $K$ items are determined, the NPS method is applied and iterated to select the personalized items for each examinee until the stopping rule is met. Note that column swapping for the Q-optimal criterion is not applied in the following algorithm description, but is implemented in the actual algorithm.

• Step 0: Initialize the item pool $R^{(0)}=\{1,\dots ,J\}$.

• Step 1: For each examinee, randomly select an Item $j$ from $R^{(0)}$ such that the q-vector of Item $j$${\mathit{q}}_j={\mathit{e}}_1$, where ${\mathit{e}}_1$ is a unit vector with $e_1=1$ and all the other entries 0s. Update the set $R^{(0)}$ by deleting Item $j$; that is, $R^{(1)}=R^{(0)}\backslash \left\{j\right\}$.

• Step 2: Administer Item $j$ to the examinee and initialize the response vector as $y^{(1)}=(y^{(1)})$.

• Step 3: At cycle $k$ where $1<k\le K$, select an item, denoted as Item $j^{\prime }$, from $R^{(k-1)}$ such that ${\mathit{q}}_{j^{\prime }}={\mathit{e}}_k+\sum _{i=1}^{k-1}{B}_i{\mathit{e}}_{i}I(y^{(i)}=1)$, where $B_i$s are independent Bernoulli variables with success probability equal to 0.5. Update the set $R^{(k-1)}$ by deleting Item $j^{\prime }$; that is, $R^{(k)}=R^{(k-1)}\backslash \left\{j^{\prime }\right\}$.

• Step 4: Administer Item $j^{\prime }$ to the examinee and update the response vector by ${\mathit{y}}^{(k)}=({\mathit{y}}^{(k-1)},y^{(k)})$.

• Step 5: Let $k=k+1$ and repeat Steps 3 and 4 until $k=K$.

• Step 6: Estimate the examinee’s attribute profile using the NPC method with the observed responses ${\mathit{y}}^{(K)}$ and label the estimate as ${\hat{\mathit{\alpha }}}^{(K)}$. Arrange the distances between ${\mathit{y}}^{(K)}$ and the other $2^K-1$$K$-dimensional ideal response vectors in ascending order, denoted as $d_2^{(K)},\dots ,d_{2^K}^{(K)}$. Denote the attribute profile corresponding to $d_2^{(K)}$ as ${\tilde{\mathit{\alpha }}}^{(K)}$.

• Step 7: At cycle $t$, where $t>K$, choose an item from $R^{(t-1)}$ that can discriminate ${\hat{\mathit{\alpha }}}^{(t-1)}$ and ${\tilde{\mathit{\alpha }}}^{(t-1)}$ by establishing ${\mathrm{\eta }}_{\hat{\mathit{\alpha }}}\ne {\mathrm{\eta }}_{\tilde{\mathit{\alpha }}}$. Suppose the item is Item $j^{″}$. If Item $j^{″}$ can be found, go to Step 10; otherwise, continue with Step 8.

• Step 8: Replace ${\tilde{\mathit{\alpha }}}^{(t-1)}$ with the attribute profile corresponding to $d_m^{(t-1)}$, where $m\ge 3$. Choose an item from $R^{(t-1)}$ that can discriminate ${\hat{\mathit{\alpha }}}^{(t-1)}$ and ${\tilde{\mathit{\alpha }}}^{(t-1)}$.

• Step 9: Repeat Step 8 until Item $j^{″}$ is found and ${\hat{\mathit{\alpha }}}^{(t-1)}$ and ${\tilde{\mathit{\alpha }}}^{(t-1)}$ can be discriminated.

• Step 10: Administer Item $j^{″}$ to the examinee and update $R^{(t-1)}$ and ${\mathit{y}}^{(t-1)}$ by $R^{(t)}=R^{(t-1)}\backslash \{j^{″}\}$ and ${\mathit{y}}^{(t)}=({\mathit{y}}^{(t-1)},y^{(t)})$, respectively.

• Step 11: Estimate the attribute profile of the examinee using the NPC method with the responses in ${\mathit{y}}^{(t)}$ and denote the estimate as ${\hat{\mathit{\alpha }}}^{(t)}$. Arrange the distances between ${\mathit{y}}^{(t)}$ and the other $2^K-1$$t$-dimensional ideal response vectors as $d_2^{(t)},\dots ,d_{2^K}^{(t)}$. Denote the estimate corresponding to $d_2^{(t)}$ as ${\tilde{\mathit{\alpha }}}^{(t)}$.

• Step 12: Let $t=t+1$ and repeat Steps 7 through 11 until the stopping rule is met.

The fixed-length criterion is used in the study to stop the algorithm and other stopping criteria, if suitable, can be easily implemented in the algorithm. It should be emphasized that in Step 7, the discrimination power of an item is determined by whether the item can yield distinct $\mathrm{\eta }$s for ${\hat{\mathit{\alpha }}}_i$ and ${\tilde{\mathit{\alpha }}}_i$. Recall that the goal of the NPS method is to find an item that can result in distinct observed responses in the $(t+1)$th cycle for the two attribute profiles ${\hat{\mathit{\alpha }}}_i^{(t)}$ and ${\tilde{\mathit{\alpha }}}_i^{(t)}$ estimated in the $t$th cycle. Whereas the observed responses $y^{(t+1)}$s from ${\hat{\mathit{\alpha }}}_i^{(t)}$ and ${\tilde{\mathit{\alpha }}}_i^{(t)}$ are not yet obtainable at cycle $t$, ${\mathrm{\eta }}^{(t+1)}$s can already be computed using ${\hat{\mathit{\alpha }}}_i$ and ${\tilde{\mathit{\alpha }}}_i$ and the q-vectors of items in the item bank. Furthermore, it is known that examinees with ${\hat{\mathit{\alpha }}}_i^{(t)}$ and ${\tilde{\mathit{\alpha }}}_i^{(t)}$ are more likely to have distinct observed responses to an item at cycle $t+1$ if their ideal responses to the item are also distinct. Hence, it is justified to use ${\mathrm{\eta }}^{(t+1)}$s, rather than the unobtainable $y^{(t+1)}$s, to assess the discrimination power of the items.

Several remarkable advantages of the NPS method are worth noting. First, because no model fitting is needed for the NPC method, the NPS method does not require any calibration and thus, is ready to be used regardless of the availability of calibration samples. Second, it has been shown that the NPC method can be used when data conform to various CDMs under the assumptions described earlier in the “Technical Background” section (Chiu & Douglas, 2013; S. Wang & Douglas, 2015). Hence, the NPS method appears to be suitable for an item bank involving multiple CDMs. Third, because the NPS method only requires the q-vector, rather than the characteristics of a specific item, for item selection, items of the same q-vector have identical discrimination power and the best item is selected randomly from the candidate items. Hence, this built-in mechanism for controlling item exposure falls into the category of randomization method (Georgiadou et al., 2007) and nicely balances between estimation accuracy and test security. Fourth, the NPS method is computationally straightforward and requires only a fraction of CPU time, a feature that makes the NPS an accommodating method for an environment without powerful high-end computers.

A Variation: The Weighted NPS (WNPS) Algorithm

The algorithm can easily be modified if a variation of the NPC method is used to estimate the examinees’ attribute profiles. For instance, Chiu and Douglas (2013) proposed the weighted Hamming distance to accommodate item variability. Let ${\overline{p}}_j$ be the proportion of examinees responding correctly to Item $j$. Then the weighted Hamming distance is defined as

$$d_{\mathrm{wh}}({\mathit{Y}}_i,{\mathbf{\eta }}^{\left(m\right)})=\sum _{j=1}^J\frac{1}{{\overline{p}}_j(1-{\overline{p}}_j)}|Y_{\mathrm{ij}}-{\mathrm{\eta }}_j^{\left(m\right)}|,$$ (12)

and can be plugged into Equation 5 to estimate the examinees’ profiles. The weighted Hamming distance may be favored when the test is short, and thus, ties could happen with the ordinary Hamming distance. If the weighted Hamming distance is used, the only modification needed to be made is to replace $d$ with $d_{\mathrm{wh}}$ in Step 11. This version of the algorithm is referred to as the WNPS method and will also be examined in the simulation. However, it is important to emphasize that differing from the NPS algorithm, which is free of any calibration, the WNPS algorithm requires a sample to compute weight $1/({\overline{p}}_j(1-{\overline{p}}_j))$ for each item before the test is administered.

Simulation Studies

The goal of the simulations is to evaluate the performance of the two proposed NPS algorithms, NPS and WNPS, in comparison with six frequently researched parametric methods, SHE, PWKL, PWCDI, GDI, DBS, and SDBS with the random selection method serving as the baseline for all methods. Among these parametric methods, the first four do not have any mechanism to control item security and will be used in Simulation Study I. The last two, similar to the NPS and WNPS methods, have the built-in randomization mechanism for item exposure control and will be used in Simulation Study II. Specific focuses are on the effectiveness and efficiency of these compared methods in classifying examinees and their ability to control item exposure with small calibration samples. To provide information useful for and relevant to practice, multiple CDMs, rather than a single CDM, were used to generate data.

Design

To investigate the performance of the studied methods under the conditions of small calibration samples, three sample sizes, $N_0=30$, $50$, and $100$, were considered. The calibration samples were used for the WNPS algorithm to compute the weight of each item and for the parametric methods to calibrate the item parameters; however, the samples were not required by the NPS algorithm. The item bank consisted of J = 300 items; 150 of them conformed to the DINA model and the other 150 conformed to the reduced RUM. For both CDMs, the guessing, $P(Y_j=1|\mathit{\alpha }=0)$, and slipping, $P(Y_j=0|\mathit{\alpha }=1)$, parameters were randomly drawn from Unif(0.1, 0.2) and Unif(0.2, 0.3) for items with high and low discrimination (HD and LD), respectively. The other parameters in the reduced RUM were generated in such a way that each required attribute contributed equally to the probability of success. In addition, for the NPC method to perform adequately, the structure (i.e., conjunctive or disjunctive) of the items has to be known in advance (Chiu & Douglas, 2013). In the simulation studies, the conjunction structure was assumed across all conditions and the conjunctive ideal response $\mathrm{\eta }=\underset{k=1}{\overset{K}{\Pi }}{\mathit{\alpha }}_k^{q_{\mathrm{jk}}}$ was calculated for each item. The number of attributes ($K$) was set to be 5 and 8, and each entry of the Q-matrix was sampled from $\mathrm{Bern}(0.5)$. The fixed-length method with the length $L=4K$ was used to terminate the algorithms.

Examinees’ attribute profiles were generated either from a discrete uniform distribution with probability $1/2^K$ for each proficiency class or from a multivariate normal (MVN) threshold model (Chiu et al., 2009) with covariances being 0.5. Finally, because the items conformed to multiple CDMs, the item parameters were calibrated by fitting the calibration samples with the saturated G-DINA model to avoid model misfit. The R package G-DINA was used to calibrate the item parameters.

Measures

The performances of the methods were evaluated with classification accuracy and item exposure control. The classification accuracy was measured by the attribute profile recovery rates (RRs), computed by

$$\mathrm{RR}=\frac{\sum _{i=1}^{N}I[{\hat{\mathit{\alpha }}}_i={\mathit{\alpha }}_i]}{N}.$$

As implied, the methods were applied to a sample of $N$ examinees, in this study $N=500$, to produce smooth trajectories of the RRs along with the test length. To provide more detailed information, the RRs for all methods under each condition were computed at cycles $K,2K,3K$, and $4K$, and then presented in the same graph.

To measure the effectiveness of the item exposure control, four indices were used. The first two indices were the unused rate (UR) and the overexposed rate (OR), where an item is considered overexposed when its exposure rate is larger than 0.2 (Mao & Xin, 2013; Zheng & Wang, 2017). The third index is the average between-test overlap rate (TOR; Chen, Ankenmann, & Spray, 2003). In the study, an approximate TOR developed by Chen et al. (2003) was adopted to ease the computational burden of an exact TOR. Specifically, it is computed by

$$\mathrm{TOR}=\frac{J}{L}{S}_r^2+\frac{L}{J},$$

where $S_r^2$ is the sample variance of exposure rates. Note that a high TOR implies low test security. The fourth index is the chi-square statistic defined as

$${\chi }^2=\sum _{j=1}^J\frac{{(r_j-\overline{r})}^2}{\overline{r}},$$

where $r_j$ denotes the exposure rate of Item $j$ and $\overline{r}$ is the expected exposure rate for a random selection, defined as $\overline{r}=\frac{L}{J}=\frac{4K}{J}$ in the study. The formula indicated that the larger the ${\chi }^2$, the less capable the algorithm of selecting items randomly, which, in turn, implies low test security.

Result I: Comparisons With SHE, PWKL, PWCDI, and GDI Methods

Recall that a mechanism for controlling item exposure is automatically implemented in the NPS and WNPS algorithms, but not in any of these four parametric methods. Although it is possible for the parametric methods to use an external item exposure control method, none of the developed methods is equivalent to the mechanism used in the nonparametric algorithms. Therefore, to avoid unfaithful comparisons, no external method was used to control item exposure for the parametric methods in the simulation, and therefore, the results from the parametric methods represented the upper bounds of the RRs they could possibly produce in practice.

Figure 2 illustrates the RRs of the two nonparametric and four parametric item selection methods when the calibration sample size $N_0$ is 100. In this condition, the NPS and WNPS methods performed similarly, and both outperformed the parametric methods. Among the parametric methods, the GDI method outperformed the other three methods, and the SHE method performed slightly better than the PWKL and the PWCDI methods. For the nonparametric methods, there was no significant difference in the RRs between the attribute structures; however, parametric methods resulted in lower RRs when the attribute structure was uniform. In both instances, the NPS and WNPS methods resulted in RRs close to 0.85; however, the RRs for the GDI method were below 0.78, and those for the SHE, PWKL, and PWCDI methods went down to the 0.6s or worse.

Figure 2.

Pattern-wise RRs for the SHE, PWKL, PWCDI, and GDI methods with HD items when $K=5$ and $N_0=100$.

Note. PWKL = posterior-weighted Kullback–Leibler; PWCDI = posterior-weighted cognitive diagnostic model discrimination index; HD = high discrimination; NPS = nonparametric item selection; RR = recovery rate; GDI = G-DINA model discrimination index; G-DINA = generalized deterministic inputs, noisy “and” gate; WNPS = weighted NPS method; MVN = multivariate normal threshold model.

The results from $N_0=50$ were omitted because the effects of all the compared methods with this calibration sample size were, as expected, between those of $N_0=100$ and $N_0=30$.

When $N_0=30$, as shown in Figure 3, the NPS and WNPS methods considerably outperformed the parametric methods and their corresponding RRs stayed high because their performance was independent of the calibration samples. The discrepancy between the nonparametric and parametric methods increased compared with the results in Figure 2. The parametric methods suffered noticeably from the small calibration samples particularly when the attribute profiles were uniformly distributed. Across the two attribute structures, the RRs from the NPS and WNPS methods remained close to 0.85. The RRs from the GDI dropped to 0.68 and 0.45 with the MVN and uniform attribute structures, respectively, and those from the SHE, PWKL, and PWCDI methods further dropped to 0.5s and below 0.3s with the MVN and uniform attribute structures, respectively.

Figure 3.

Pattern-wise RRs for the SHE, PWKL, PWCDI, and GDI methods with HD items when $K=5$ and $N_0=30$.

Note. PWKL = posterior-weighted Kullback–Leibler; PWCDI = posterior-weighted cognitive diagnostic model discrimination index; GDI = G-DINA model discrimination index; G-DINA = generalized deterministic inputs, noisy “and” gate; HD = high discrimination; NPS = nonparametric item selection; RR = recovery rate.

To investigate the efficiency of the item selection methods when the number of attributes is large, a simulation for $K=8$ was also conducted. However, due to the excessive CPU time some of the compared parametric methods required to compute the criteria for all items in the item banks, not all the conditions were used to investigate the performance of the studied methods when $K=8$. Considering that the parametric methods performed the best under the condition of $N_0=100$ with the attributes following a multivariate normal threshold model when $K=5$, only this condition was used when $K=8$. The results presented here could serve as a benchmark for the parametric methods, and it is predictable that the results from the other conditions should only be worse.

As shown in the left graph of Figure 4, the RRs for the NPS and WNPS methods were about between 0.63 and 0.67, indicating that the classification was more difficult when $K$ increased. However, all parametric methods performed just slightly better than the random selection method with the RRs below 0.2, showing that when $K=8$, these parametric methods did not behave much differently from a random selection method when the size of the calibration sample was 100. The results further confirmed the significant importance of the nonparametric methods when CD-CAT is used in a small-scale setting.

Figure 4.

Pattern-wise RRs when $K=8$.

Note. PWKL = posterior-weighted Kullback–Leibler; PWCDI = posterior-weighted cognitive diagnostic model discrimination index; GDI = G-DINA model discrimination index; G-DINA = generalized deterministic inputs, noisy “and” gate; RR = recovery rate; NPS = nonparametric item selection; DBS = dynamic binary searching; SDBS = stratified dynamic binary searching.

Result II: Comparisons With DBS and SDBS Methods

Because the randomization mechanism for controlling item exposure is also implemented in the DBS and the SDBS methods, the four methods compared in this section were evaluated on the same bases. Figure 5 illustrates the RRs resulting from the studied methods when $N_0=100$ and the items had high discrimination.

Figure 5.

Pattern-wise RRs for the NPS, WNPS, DBS, and SDBS methods with HD items when $K=5$ and $N_0=100$.

Note. NPS = nonparametric item selection; DBS = dynamic binary searching; SDBS = stratified dynamic binary searching; HD = high discrimination; RR = recovery rate.

The graphs indicated that when the attribute structure was multivariate normal, the NPS and the WNPS methods outperformed the DBS and the SDBS methods. However, when the attribute structure was uniform, the four methods performed similarly well with the NPS and the WNPS methods only slightly better than the DBS and the SDBS methods. When items had low discrimination, the performance of the compared methods across different test lengths followed the similar patterns to those when the items were of high discrimination. The only noticeable difference was that the terminal RRs for the former condition were about 0.8 whereas those for the latter were about 0.6. To avoid redundancy, the graphs for the latter case are omitted.

Figure 6 illustrates the RRs resulting from the compared methods when $N_0=30$ and the items had high discrimination.

Figure 6.

Pattern-wise RRs for the NPS, WNPS, DBS, and SDBS methods with HD items when $K=5$ and $N_0=30$.

Note. NPS = nonparametric item selection; DBS = dynamic binary searching; SDBS = stratified dynamic binary searching; HD = high discrimination; RR = recovery rate.

The graphs showed that when $N_0=30$, the NPS and the WNPS methods substantially outperformed the DBS and the SDBS methods across the two attribute structures with the RRs from the NPS and WNPS methods at the 0.8s and those from the DBS and SDBS methods at the 0.5s. When items of low discrimination were used, the performances of the four compared methods, although not shown here, followed the similar patterns, but the RRs dropped to the 0.5s for the NPS and WNPS methods and the 0.3s for the DBS and the SDBS methods.

Result III: Item Exposure Control

The results of item exposure control are reported in Table 1. The table showed that when $K=5$, the NPS and WNPS methods performed similarly and had the lowest UR, OR, TOR, and ${\chi }^2$ among all compared methods across all conditions. The values resulted from the nonparametric methods were very close to the random selection method, which were omitted here. Specifically, the NPS and WNPS methods used all items in the item bank and did not have overexposed items except for one condition (i.e., MVN and $N_0=100$) with only several overused items. Regarding the parametric methods, the PWCDI method had the most severe item exposure control problem in terms of the TOR and the ${\chi }^2$ indices but did not seem to perform as poorly in terms of the OR index in some conditions. As to the DBS and SDBS methods, Table 1 also indicated that although the DBS and SDBS methods had a default exposure control mechanism, similar to the NPS and WNPS methods, they could not control item exposure as well as the nonparametric methods in terms of all the indices used in the study. In particular, the SDBS performed the worst aomg the four compared methods but both DBS and SDBS methods performed, not surprisingly, better than the other parametric methods.

Table 1.

The Exposure Rates and ${\chi }^2$ Index for the Nonparametric, SHE, PWKL, PWCDI, and GDI Methods.

		Study I						Study II
Condition	Index	NPS	WNPS	SHE	PWKL	PWCDI	GDI	NPS	WNPS	DBS	SDBS
$K=5$; $N_0=100$
MVN	UR	0	0	9.7	12.3	13.7	8.3	0	0	0.1	1.6
	OR	3.0	2.3	10.0	9.3	8.0	12.3	5.2	5.1	7.0	7.6
	TOR	11.0	11.5	31.0	72.4	83.8	35.2	11.8	11.9	15.6	20.5
	${\chi }^2$	12.8	14.5	72.6	196.7	230.7	85.3	15.3	15.8	26.8	41.4
UNIF	UR	0	0	6.3	13.7	15.3	9.7	0	0	0	1.8
	OR	0	0	12.3	13.3	11.3	11.3	3.2	3.3	8.4	6.2
	TOR	9.4	9.4	27.0	41.7	59.8	32.8	11.2	11.1	15.2	19.0
	${\chi }^2$	8.1	8.1	60.8	104.8	158.8	78.1	13.4	13.1	25.7	36.7
$K=5$; $N_0=30$
MVN	UR	0	0	8.0	15.7	16.0	8.3	0	0	0.3	2.3
	OR	0	0	10.0	9.7	7.3	11.0	0	0	5.3	5.7
	TOR	9.8	9.5	25.1	57.9	78.3	30.0	9.7	9.5	15.7	19.4
	${\chi }^2$	9.4	8.4	55.2	153.1	214.1	69.7	9.0	8.4	27.1	38.0
UNIF	UR	0	0	6.0	13.7	13.3	8.3	0	0	0	1.3
	OR	0	0	11.0	12.0	12.0	12.3	0.3	0	8.3	8.3
	TOR	8.4	8.5	25.0	38.0	45.8	29.0	9.2	9.0	15.5	16.9
	${\chi }^2$	5.1	5.6	54.7	93.6	117.1	66.9	7.5	7.0	26.5	30.6
$K=8$; $N_0=100$
MVN	UR	0.3	0.3	1.3	3.0	3.7	2.0	0	0	0.3	1.7
	OR	12.3	13.0	18.3	17.7	16.3	15.7	12.8	14.0	13.8	16.7
	TOR	22.4	22.6	23.0	31.3	37.1	24.9	28.7	29.0	43.2	37.4
	${\chi }^2$	35.0	35.7	37.0	61.7	78.9	42.6	54.0	55.0	97.1	79.9

Note. All the rates (i.e., OR, UR, and TOR) are expressed as percentages. PWKL = posterior-weighted Kullback–Leibler; PWCDI = posterior-weighted cognitive diagnostic model discrimination index; NPS = nonparametric item selection; DBS = dynamic binary searching; SDBS = stratified dynamic binary searching; UR = unused rate; OR = overexposed rate; TOR = between-test overlap rate; GDI = G-DINA model discrimination index; G-DINA = generalized deterministic inputs, noisy “and” gate; UNIF = uniform distribution.

The last panel of Table 1 shows that when $K=8$, the NPS and WNPS methods again outperformed the other methods in terms of item exposure control. However, although there were no unused items for the nonparametric methods, the OR, TOR, and ${\chi }^2$ values were elevated, indicating that it was more difficult for the nonparametric methods to control item exposure when $K$ was large.

Discussion

The NPS method was developed in this article to select items in the CD-CAT system by first using the NPC method (Chiu & Douglas, 2013) to classify examinees and then for each examinee, selecting an item that can best differentiate the examinee’s estimated attribute profile from the other attribute profiles based on the distance between the examinee’s observed and ideal responses. The examinee’s attribute profile is updated using the NPC method again after the new response is added to the response vector. The algorithm iterates until the stopping criterion is met.

The key concepts provided in this article confirmed and supported the legitimacy of using the NPS method in a CD-CAT system. The simulation study further demonstrated the effectiveness and efficiency of the NPS and WNPS algorithms and systematically verified their outperformance over the parametric methods when the calibration samples were small. It was also reported that the NPS and WNPS algorithms yielded UR, OR, TOR, and ${\chi }^2$ values substantially lower than those from the parametric methods and almost as low as those from the random selection method across all conditions, indicating effective control of item exposure.

Recall that the estimator obtained by using the NPC method is statistically consistent if $P(Y_j=1|\mathrm{\eta }=0)<0.5$ and $P(Y_j=1|\mathrm{\eta }=1)>0.5$. These assumptions hold by default for items generated from the DINA model; however, they may or may not hold for a complex CDM, such as the reduced RUM. The authors would like to point out that in the simulation studies, the items, on average, conforming to the reduced RUM were generated in such a way that these assumptions held. Specifically, when $K^{*}=2$ and 3, on average, the probabilities of success for the collapsed latent classes used to generate the data were $(P(00),P(10),P(01),P(11))=(0.15,0.36,0.36,0.85)$ and $(P(000),P(100),P(010),P(001),P(110),P(101),P(011),P(111))=(0.15,0.27,0.27,0.27,0.48,0.48,0.48,0.85)$, respectively (note that the single-attribute items are not considered here because the estimation results from different CDMs are identical). These probabilities indicated that $P(Y_j=1|\mathrm{\eta }=0)<0.5$ and $P(Y_j=1|\mathrm{\eta }=1)>0.5$. Although the reduced RUM items satisfied the conjunctive assumptions, they had low discrimination due to the large probabilities such as 0.36 and 0.48. Nevertheless, the NPC method performed sufficiently well that the NPS method outperformed the parametric methods used for comparison.

It is worth mentioning that in the simulation, it is required that all examinees in the calibration sample respond to all the items in the item bank for all the parametric methods and the WNPS algorithm, which is somewhat unrealistic. In this regard, the advantage of the NPS method that no calibration sample is needed cannot be overstated.

However, an issue remains to be addressed here. The Q-optimal criterion used in this article was developed for the parametric item selection methods that use the maximum likelihood estimation for $\mathit{\alpha }$. It is unknown whether the same criterion can also be applied to the NPS method. Further investigation is called for to either establish the legitimacy of using Equation 11 as a Q-optimal criterion for the NPS method or develop a Q-optimal criterion suitable for the NPS method.