Improved Wald Statistics for Item-Level Model Comparison in Diagnostic Classification Models

Abstract

Diagnostic classification models (DCMs) have been widely used in education, psychology, and many other disciplines. To select the most appropriate DCM for each item, the Wald test has been recommended. However, prior research has revealed that this test provides inflated Type I error rates. To address this problem, the authors propose to replace the asymptotic covariance matrix from the original version of the Wald statistic with a matrix obtained from improved computation methods. In this study, the Wald test based on the observed information matrix and the Wald test based on the sandwich-type matrix are proposed for item-level model comparisons and a simulation study is conducted to investigate their empirical behavior. Simulation results indicate that when the sample size is reasonably large ($N\ge 1,000$), the Type I error rates of the Wald test based on the sandwich-type matrix are accurate with adequate or excellent power under most of the simulation conditions.

Introduction

Diagnostic classification models (DCMs) characterize relationships between responses of an individual obtained from questionnaire/test items and his or her latent attributes. DCMs have great promise to provide multidimensional discrete diagnostic information on a set of fine-grained attributes (such as cognitive processes, skills, abilities or knowledge representations) for instructional interventions (Rupp, Templin, & Henson, 2010). To date, DCMs have been widely used in educational, psychological, and psychiatric measurement, and also in many other disciplines (e.g., Greeno, 1980; Leighton & Gierl, 2007; Sorrel et al., 2016; Templin & Henson, 2006).

As a class of statistical models, a variety of DCMs have been developed (Rupp et al., 2010) based on different assumptions about how the probability of a correct response conditional on the Q-matrix and an examinee’s latent attribute mastery pattern is defined. The different DCMs reflect specific assumptions about how latent attributes influence test performance in a specific domain, where the Q-matrix specifies the relationship between items and latent attributes (Tatsuoka, 1990). For example, the deterministic inputs, noisy, “and” gate model (DINA; Haertel, 1989; Junker & Sijtsma, 2001) represents a noncompensatory assumption in that all required attributes are expected to have been mastered to obtain a correct response to an item. Hence, the absence of any of the required attributes cannot be compensated by the presence of other attributes. The deterministic inputs, noisy, “or” gate model (DINO; Templin & Henson, 2006) and the additive cognitive diagnostic model (A-CDM; de la Torre, 2011) are compensatory models, in which the absence of some attributes can be compensated by the presence of other attributes. The DINA, DINO, and A-CDM are specific DCMs, in other words these models can be shown to be constrained special cases of the general DCMs, or saturated DCMs, such as the general diagnostic model (von Davier, 2005, 2008), the log-linear cognitive diagnosis model (Henson, Templin, & Willse, 2009) and the generalized DINA model (GDINA; de la Torre, 2011).

As many DCMs are available, one critical step is to select the most appropriate DCM for response data at the item level. Compared with the saturated DCMs, the selected specific DCMs are more suitable for practical applications (de la Torre & Lee, 2013; Ma, Iaconangelo, & de la Torre, 2016; Rojas, de la Torre, & Olea, 2012; Sorrel, Abad, Olea, de la Torre, & Barrada, 2017; Sorrel, de la Torre, Abad, & Olea, 2017). In DCMs, given that the Q-matrix is correctly specified and that the saturated model provides the best model-data fit (Liu, Tian, & Xin, 2016), there are at least five different statistics to use for item-level model selection: the Wald test, the $S-X^2$ test, the likelihood ratio (LR) test, the Lagrange multiplier test, and the two-step likelihood ratio (2LR) test (Sorrel, Abad, et al., 2017; Sorrel, de la Torre, et al., 2017). De la Torre (2011) originally proposed that the Wald ($W_d$) test can be used to select the most interpretable reduced model from the saturated DCM. De la Torre and Lee (2013) investigated the empirical Type I error rate control and power of $W_d$ by taking DINA, DINO, and A-CDM as examples. Furthermore, Ma et al. (2016) evaluated the performance of $W_d$ in selecting the most appropriate model at the item level for additive models under the log and logit link functions (Hartz, 2002; Maris, 1999). Sorrel, Abad, et al. (2017) examined the performance of $W_d$, $S-X^2$, LR, and the Lagrange multiplier test and found that, compared with other statistics, only the empirical Type I error of $S-X^2$ was close to the nominal level. However, $S-X^2$ had low power to reject false models. Sorrel, de la Torre, et al. (2017) compared the performance of the 2LR, LR, and $W_d$ in selecting the best reduced DCM. Although $W_d$ has some advantages to other methods, such as ease of computation and slightly higher power, all of the previous studies consistently found that the $W_d$ test that was computed using the item-wise information matrix proposed by de la Torre (2011) had inflated Type I error rates. This suggests that better computation methods of the covariance matrix or the item parameter standard errors (SEs) are needed (Ma et al., 2016) for the test to maintain the correct significance level.

Liu et al. (2016) have noted that since not only the item parameters but also the structural model parameters are estimated from the examinees’ response data (Rupp et al., 2010), both the item parameters and structural model parameters should be considered in the computation of the information matrix. Philipp, Strobl, de la Torre, and Zeileis (2018) have analytically shown that the incomplete information matrix (de la Torre, 2009) or the item-wise information matrix (de la Torre, 2011) lead to underestimated item SEs. Furthermore, in DCMs two new methods to directly estimate the item parameter covariance matrix, the inverse of the observed information matrix, and the sandwich-type estimator that take into account both the item and structural parameters have been proposed (Liu, Xin, Andersson, & Tian, 2018; Xin, Liu, & Tian, 2016). Liu et al. (2018) compared the performance of the two new methods, the empirical cross-product information matrix, and the two approaches proposed by de la Torre (2009, 2011) in providing consistent SEs of item parameter estimates and found that with correctly specified or slightly misspecified models the observed information matrix and the sandwich-type matrix had good performance. However, with substantial model misspecification only the SEs based on the sandwich-type covariance matrix were robust. Hence, in this study two new ways to calculate the Wald statistic for item-level model comparison will be introduced. That is, the Wald test based on the observed information matrix ($W_{\mathrm{Obs}}$) or the Wald test based on the sandwich-type matrix ($W_{\mathrm{SW}}$).

The main interest of this study is to investigate the empirical behavior of the two newly proposed methods together with the $W_d$ and LR statistics for item-level model comparisons under the GDINA framework via simulations. The remainder of this article is organized as follows: First, useful background on the saturated GDINA and its reduced models are briefly described, followed by an introduction of the statistical tests for item-level model comparisons that will be investigated in this study, and a discussion of several methods that can be used to obtain the asymptotic covariance matrix of the item parameter estimates. Then, results from a simulation study that systematically evaluates the performance of $W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR were reported. The article concludes with discussion, concluding remarks, and some directions for further research.

The Saturated GDINA and the Reduced DCMs

The GDINA is a general framework and many commonly used DCMs such the DINA, DINO, or A-CDM can be viewed as special cases of the GDINA model (de la Torre, 2011). Suppose that $K$ binary attributes are diagnosed in a DCM test with $J$ dichotomous items. Define $\mathbf{\alpha }=(\mathbf{\alpha }{\prime }_1,\dots ,\mathbf{\alpha }{\prime }_l,\dots ,\mathbf{\alpha }{\prime }_L)\prime$ as the discrete attribute mastery profile matrix with a total number of possible attribute mastery patterns equal to $L=2^K$. The structural parameter vector $\mathbf{\eta }=({\eta }_1,\dots ,{\eta }_l,\dots ,{\eta }_{L-1})\prime$ describes the probability $p({\mathbf{\alpha }}_l|\mathbf{\eta })$of a randomly selected examinee being a member of the $l$th attribute mastery profile. The number of freely estimated structural parameters is thus $L-1$ (Liu et al., 2016; Rupp et al., 2010). A $J\times K$ binary Q-matrix specifies which attributes are measured by each item, for example, ${\mathbf{q}\prime }_j=(q_{j1},\dots ,q_{\mathrm{jk}},\dots ,q_{\mathrm{jK}})$ is the $j$th row of the Q-matrix. In the Q-matrix, if attribute $k$ is required for solving item $j$, $q_{\mathrm{jk}}=1$, and $q_{\mathrm{jk}}=0$ otherwise. Let $\mathbf{x}=({{\mathbf{x}}^{\prime }}_1,\dots {{\mathbf{x}}^{\prime }}_i\dots ,{{\mathbf{x}}^{\prime }}_N)\prime$ be the observed response matrix of $N$ examinees, where ${\mathbf{x}}_i=(x_{i1},\dots ,x_{\mathrm{iJ}})\prime$ is the $i$th examinee’s response vector.

In the saturated GDINA, the probability of a correct response on item $j$ for examinee $i$ with attribute mastery pattern ${\mathbf{\alpha }}_l=({\alpha }_{l1},\dots ,{\alpha }_{\mathrm{lK}})\prime$ is given by de la Torre (2011).

$$P_j\left({\mathbf{\alpha }}_l\right)={\lambda }_{j,0}+\sum _{k=1}^K{\lambda }_{j,1,(k)}{\alpha }_{\mathrm{lk}}{q}_{\mathrm{jk}}+\sum _{k=1}^{K-1}\sum _{k\prime =k+1}^K{\lambda }_{j,2,(k,k\prime )}{\alpha }_{\mathrm{lk}}{\alpha }_{lk\prime }{q}_{\mathrm{jk}}{q}_{\mathrm{jk}\prime }+\cdots ,$$ (1)

where ${\lambda }_{j,0}$ is the intercept parameter, ${\lambda }_{j,1,(k)}$ is the main effect parameter associated with attribute $k$, and ${\lambda }_{j,2,(k,k\prime )}$ is the two-way interaction effect parameter associated with attribute $k$ and attribute $k\prime$. For example, if the first, the second, and the third attributes are required by item $j$, ${\mathbf{q}\prime }_j=(1,1,1)$ and the $i$th examinee has mastered all the required attributes by the item ${\mathbf{\alpha }\prime }_i=(1,1,1)$, then according to Equation 1 the saturated GDINA can be expressed as follows,

$$P_j\left({\mathbf{\alpha }}_i\right)={\lambda }_{j,0}+{\lambda }_{j,1,(1)}+{\lambda }_{j,1,(2)}+{\lambda }_{j,1,(3)}+{\lambda }_{j,2,(1,2)}+{\lambda }_{j,2,(1,3)}+{\lambda }_{j,2,(2,3)}+{\lambda }_{j,3,(1,2,3)}.$$ (2)

By setting some of the parameters of the saturated model to fixed values, the commonly used reduced DCMs can be obtained, see de la Torre (2011). In this example, the DINA model can be obtained from GDINA by setting the main and the lower order interaction effect terms to zero and thus the model becomes

$$P_j\left({\mathbf{\alpha }}_i\right)={\lambda }_{j,0}+{\lambda }_{j,3,(1,2,3)}.$$ (3)

To obtain the A-CDM, all the interaction effect parameters in the saturated GDINA are fixed to zero and the item response function of the A-CDM can be written as

$$P_j\left({\mathbf{\alpha }}_i\right)={\lambda }_{j,0}+{\lambda }_{j,1,(1)}+{\lambda }_{j,1,(2)}+{\lambda }_{j,1,(3)}.$$ (4)

The mathematical function for the DINO model is given by

$$P_j\left({\mathbf{\alpha }}_i\right)={\lambda }_{j,0}+{\lambda }_{j,1,(1)},$$ (5)

by setting ${\lambda }_{j,1,(1)}={\lambda }_{j,1,(2)}={\lambda }_{j,1,(3)}=-{\lambda }_{j,2,(1,2)}=-{\lambda }_{j,2,(1,3)}=-{\lambda }_{j,2,(2,3)}={\lambda }_{j,3,(1,2,3)}$.

Information Matrix Estimation Procedures for DCMs

In DCMs, the item parameters $\mathbf{\lambda }$ and the structural parameters $\mathbf{\eta }$ are simultaneously estimated from the observed data, typically by using the MMLE/EM (marginal maximum likelihood estimation via expectation–maximization) algorithm. Denote $\hat{\mathbf{\gamma }}=(\hat{\mathbf{\lambda }}\prime ,\hat{\mathbf{\eta }}\prime )\prime$ as the MLEs of the model parameters that maximizes the log likelihood function of the observed response data $\ell (\mathbf{\gamma }|\mathbf{x})$.

To directly estimate the asymptotic covariance matrix of the item parameter estimates in DCMs, there are at least five procedures available, namely, the incomplete information matrix in which only the item parameters are considered, the item-wise information matrix, the empirical cross-product information matrix, the observed information matrix, and the sandwich-type covariance matrix (Liu et al., 2018; Xin et al., 2016).

Philipp et al. (2018) have proved that the incomplete information matrix (de la Torre, 2009) and the item-wise information matrix (de la Torre, 2011) underestimates item SEs. They also studied the property of the empirical cross-product information matrix in providing consistent item parameter SEs, in which the item and structural parameters were simultaneously incorporated. The empirical cross-product information matrix is the cross-product of the derivative of the log likelihood of the observed data with respect to the model parameters,

$${\mathcal{I}}_{\mathrm{XPD}}=\frac{\partial \ell (\mathbf{\gamma }|\mathbf{x})}{\partial \mathbf{\gamma }}\frac{\partial \ell (\mathbf{\gamma }|\mathbf{x})}{\partial \mathbf{\gamma }\prime }.$$ (6)

They found that compared with the incomplete or the item-wise information matrix, the SEs based on the empirical cross-product information matrix showed smaller bias.

The observed information matrix and the sandwich-type estimator that included both the item and structural parameters for the estimation of the asymptotic covariance matrix were proposed by Liu et al. (2018). The observed information matrix is the negative of the second derivatives of the log likelihood of the observed data matrix with respect to the model parameters

$${\mathcal{I}}_{\mathrm{Obs}}=-\frac{{\partial }^2\ell (\mathbf{\gamma }|\mathbf{x})}{\partial \mathbf{\gamma }\partial \mathbf{\gamma }\prime },$$ (7)

while the sandwich-type covariance matrix can be expressed as

$${\mathcal{I}}_{\mathrm{SW}}={\mathcal{I}}_{\mathrm{Obs}}^{-1}{\mathcal{I}}_{\mathrm{XPD}}{\mathcal{I}}_{\mathrm{Obs}}^{-1}.$$ (8)

Their Monte Carlo results revealed that, under the assumption of a correctly specified DCM and a correctly specified Q-matrix, compared with the empirical cross-product, the incomplete, or the item-wise information matrix, the observed information matrix ${\mathcal{I}}_{\mathrm{Obs}}$ and the sandwich-type matrix ${\mathcal{I}}_{\mathrm{SW}}$ had good performance in providing consistent item parameter SEs. Meanwhile, with model misspecification only the SEs based on the sandwich-type covariance matrix ${\mathcal{I}}_{\mathrm{SW}}$ were robust.

The Wald and LR Statistics

The Wald Test

By setting particular item parameters of the saturated GDINA to fixed values, the reduced DCMs can be obtained. Under the condition that the saturated GDINA provides a good fit to the observed data, the Wald test can be used to evaluate the statistical significance that a set of parameters are equal to some value (de la Torre & Lee, 2013). In DCMs, the original version of $W_d$ is defined as (de la Torre, 2011),

$$W_d=(\mathbf{R}{\hat{\mathbf{\lambda }}}_j)\prime (\mathbf{R}{\hat{\mathbf{\sum }}}_j\mathbf{R}\prime {)}^{-1}(\mathbf{R}{\hat{\mathbf{\lambda }}}_j),$$ (9)

where ${\hat{\mathbf{\lambda }}}_j$ is the maximum likelihood estimate vector of item parameters for item $j$, $\mathbf{R}$ is a restriction matrix for the Wald test that can be used to impose a restriction on the item parameters of the saturated model and to obtain the corresponding item parameter covariance matrix ${\hat{\mathbf{\sum }}}_j$. Under the null hypothesis that the saturated DCM is equivalent to the specific DCM of interest, the Wald statistic asymptotically follows a chi-square distribution with degrees of freedom equal to the number of item parameters involved in the saturated DCM minus the number of item parameters in the reduced DCM.

Using the same example as above, here the authors also suppose that ${\mathbf{q}\prime }_j=(1,1,1)$ and ${\mathbf{\alpha }\prime }_i=(1,1,1)$. In the DINA model, the null hypothesis is that the main and the lower order interaction effect parameters in the saturated GDINA are simultaneously equal to zero. Then, the corresponding restriction matrix $\mathbf{R}$ is

$$\mathbf{R}=\left(\begin{array}{cccccccc}\hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill \end{array}\right).$$ (10)

According to Equation 10, $\mathbf{R}{\hat{\mathbf{\lambda }}}_j=({\hat{\lambda }}_{j,1,(1)},{\hat{\lambda }}_{j,1,(2)},{\hat{\lambda }}_{j,1,(3)},{\hat{\lambda }}_{j,2,(1,2)},{\hat{\lambda }}_{j,2,(1,3)},{\hat{\lambda }}_{j,2,(2,3)})\prime$ is used to set the main and the lower order interaction effect parameters to zero. The restriction matrix $\mathbf{R}$ for the A-CDM to set all the interaction effect parameters to zero and to compute the corresponding covariance matrix is

$$\mathbf{R}=\left(\begin{array}{cccccccc}\hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right).$$ (11)

Note that for the A-CDM, $\mathbf{R}{\hat{\mathbf{\lambda }}}_j=({\hat{\lambda }}_{j,2,(1,2)},{\hat{\lambda }}_{j,2,(1,3)},{\hat{\lambda }}_{j,2,(2,3)},{\hat{\lambda }}_{j,3,(1,2,3)})\prime$. To impose restriction on the saturated GDINA to obtain the DINO model and to test the hypothesis that ${\hat{\lambda }}_{j,1,(1)}={\hat{\lambda }}_{j,1,(2)}={\hat{\lambda }}_{j,1,(3)}=-{\hat{\lambda }}_{j,2,(1,2)}=-{\hat{\lambda }}_{j,2,(1,3)}=-{\hat{\lambda }}_{j,2,(2,3)}={\hat{\lambda }}_{j,3,(1,2,3)}$, the following restriction matrix is defined,

$$\mathbf{R}=\left(\begin{array}{cccccccc}\hfill 0\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 1\hfill & \hfill 1\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill -1\hfill & \hfill 1\hfill & \hfill 0\hfill \\ \multicolumn{1}{c}{0}& \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right).$$ (12)

Close inspection of Equation 9 reveals that the Wald statistic is dependent on the accuracy of the item parameter estimates ${\hat{\mathbf{\lambda }}}_j$ and the corresponding asymptotic covariance matrix ${\hat{\mathbf{\sum }}}_j$. However, in the original implementation of the Wald statistic $W_d$ for selecting the most suitable model, the item parameter covariance matrix was computed based on the item-wise information matrix, which leads to underestimated item parameter SEs (Philipp et al., 2018).

In this study, the authors propose that the Wald test for item-level model comparison is computed using ${\mathcal{I}}_{\mathrm{Obs}}$ or ${\mathcal{I}}_{\mathrm{SW}}$. As previous studies (Liu et al., 2018; Xin et al., 2016) have shown that the covariance matrices ${\hat{\mathbf{\sum }}}_j$ of the item parameter estimates based on the inverse of ${\mathcal{I}}_{\mathrm{Obs}}$, or ${\mathcal{I}}_{\mathrm{SW}}$ are more accurate than those based on the empirical cross-product, the incomplete, or the item-wise information matrix, in this study, the Wald statistic in which the computation of the item parameter covariance matrix ${\hat{\mathbf{\sum }}}_j$ is based on ${\mathcal{I}}_{\mathrm{Obs}}$ or ${\mathcal{I}}_{\mathrm{SW}}$, will be denoted by $W_{\mathrm{Obs}}$ or $W_{\mathrm{SW}}$. In theory, it is reasonable to expect that the performance of $W_{\mathrm{Obs}}$ or $W_{\mathrm{SW}}$ would be better than the original version $W_d$.

The LR Test

In DCMs, another way to address the item-level model comparison question is through use of the LR test. The LR statistic is computed as

$$\mathrm{LR}=-2\log \left(\frac{L_r}{L_s}\right)=2({\ell }_s-{\ell }_r),$$ (13)

where $L_r$ is the likelihood for the reduced DCM and $L_s$ is the likelihood for the saturated DCM. Suppose that the model parameter estimates $\hat{\mathbf{\gamma }}$ lie in the interior of the parameter space. Then, under the null hypothesis, this statistic is also asymptotically chi-square distributed with degrees of freedom equal to the number of item parameters that are constrained in the reduced DCM at the item level. By comparing the log likelihoods of the saturated and the reduced models ${\ell }_s-{\ell }_r$, the LR can be used to test whether the difference between these two models is statistically significant. The calculation of the LR is straightforward and does not require the estimation of the information matrix.

Sorrel, Abad, et al. (2017) found that the Type I error control of the LR test was more robust than $W_d$. Hence, it is of interest to empirically compare the behavior of the $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR under the same conditions.

Simulation Study

The primary interest of the current study was to examine the empirical performance of the $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, $W_d$, or LR via Monte Carlo simulations under conditions varying by sample size, attribute correlation and fitted DCM. The purpose of the first part of the simulation was to investigate the Type I error rate control of these statistics. The binary item response data matrix was generated from DINA, A-CDM, or DINO separately. The second part of the simulation was conducted to illustrate the empirical power of these statistics when the fitted model was misspecified. The saturated GDINA model was used to generate the response data.

Three factors were manipulated in a $5\times 3\times 3$ factorial design: (a) To systemically investigate the impact of sample size on the use of asymptotic sampling distributions, five sample sizes were considered, 500, 1,000, 2,000, 3,000, and 5,000. (b) To better reflect the reality of the correlations between attributes in applied research (Kunina-Habenicht, Rupp, & Wilhelm, 2012), the correlation coefficients between the attributes were $\rho =0$, $\rho =.5$, and $\rho =.8$. (c) The fitted DCMs were the DINA, A-CDM, and DINO models. This gave a total of 45 simulation conditions for each method. To further assess the effect of the manipulated between-subjects factors, the fitted DCM, sample size, and attribute correlation on the performance of the $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, $W_d$, or LR method with respect to the Type I error, the marginal means of the Type I error rates for each methods were analyzed using ANOVAs, and computing the effect size, omega-squared index (${\omega }^2$). Kirk (1996) offered the following guidelines for interpreting the magnitude of effect size ${\omega }^2$: small (0.010), medium (0.059), and large (0.138), consequently, only those factors whose ${\omega }^2$ index is greater than 0.138 were focused.

The true probabilities of a correct response for examinees who mastered none and all of the required attributes were fixed to .20 and .90. The main and/or interaction effect parameters were fixed to $(.9-.2)/(.9-.2)n_j{n}_j$, where $n_j$ is the number of the main and/or interaction effect parameters for item $j$. The Q-matrix with five attributes and 30 items is provided in the online appendix Table A1. For each simulation condition, 1,000 replications were performed to provide stable results. The 95% confidence interval $p_{\mathrm{epmical}}\pm 1.96\sqrt{p_{\mathrm{epmical}}(1-p_{\mathrm{epmical}})/R}$ of the empirical Type I error rate of these statistics which cover .05 is considered to be accurate (Bradley, 1978), $p_{\mathrm{epmical}}$ is the empirical Type I error rate, $R=1,000$ is the number of replications in each condition. As with de la Torre and Lee (2013) a power higher than or equal to .80 is considered adequate, at least .9 is considered excellent. The code was written in R (R Core Team, 2017) where the R functions for model parameter estimation, Wald test calculation and item parameter covariance matrix estimation were modified from the R packages CDM (Robitzsch, Kiefer, George, & Uenlue, 2017) and dcminfo (Liu & Xin, 2017). The R codes in the current study are available upon request from the corresponding author.

Results

Type I Error Rate Control

The empirical Type I error rate control results of the $W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR statistics across simulation conditions for DINA, A-CDM, and DINO at the 5% significance level are summarized in Tables 1 to 3, respectively. Results were averaged over items with the same number of required attributes ($K_j$). The 95% confidence interval of the empirical Type I error rate which do not cover .05 is marked in boldface.

Table 1.

Type I Error Rates for DINA ($\alpha =.5$).

		$W_d$		$W_{\mathrm{Obs}}$		$W_{\mathrm{SW}}$		LR
$\rho$	N	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$
0	500	.066	.110	.060	.096	.044	.070	.055	.063
	1,000	.065	.072	.061	.068	.053	.058	.054	.053
	2,000	.055	.065	.053	.062	.050	.058	.053	.053
	3,000	.054	.055	.052	.054	.050	.051	.050	.049
	5,000	.048	.051	.047	.050	.046	.049	.053	.049
.5	500	.082	.139	.071	.111	.046	.069	.061	.069
	1,000	.064	.089	.060	.081	.051	.065	.057	.057
	2,000	.054	.069	.052	.066	.047	.059	.053	.055
	3,000	.056	.061	.053	.059	.051	.055	.047	.055
	5,000	.054	.061	.053	.060	.051	.057	.050	.048
.8	500	.112	.131	.102	.121	.037	.050	.072	.097
	1,000	.076	.135	.068	.111	.048	.074	.058	.070
	2,000	.063	.090	.060	.084	.052	.069	.054	.055
	3,000	.056	.072	.052	.067	.047	.060	.052	.057
	5,000	.053	.064	.051	.061	.049	.057	.054	.051

Note. $W_d$ is the Wald test based on the item-wise information matrix; $W_{\mathrm{Obs}}$ is Wald test based on the observed information matrix; $W_{\mathrm{SW}}$ is Wald test based on the sandwich-type covariance matrix; LR is the likelihood ratio test; $K_j$ is the number of required attributes by the $j$th item. The 95% confidence interval of the value which do not cover .05 is marked in boldface. DINA = deterministic inputs, noisy, “and” gate model; $\rho$ = the attribute correlation; N = the sample size.

Table 3.

Type I Error Rates for DINO ($\alpha =.5$).

		$W_d$		$W_{\mathrm{Obs}}$		$W_{\mathrm{SW}}$		LR
$\rho$	N	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$
0	500	.079	.108	.071	.087	.044	.059	.061	.075
	1,000	.062	.100	.058	.094	.050	.080	.058	.057
	2,000	.057	.072	.055	.070	.051	.066	.052	.051
	3,000	.054	.064	.052	.063	.049	.059	.053	.056
	5,000	.053	.058	.052	.057	.051	.055	.048	.048
.5	500	.091	.071	.077	.063	.039	.035	.081	.083
	1,000	.065	.128	.061	.115	.052	.094	.061	.057
	2,000	.062	.090	.060	.084	.054	.076	.055	.054
	3,000	.055	.072	.053	.071	.050	.066	.053	.051
	5,000	.055	.064	.054	.063	.052	.060	.052	.057
.8	500	.091	.035	.089	.084	.029	.040	.227	.209
	1,000	.093	.136	.081	.095	.055	.067	.195	.179
	2,000	.072	.138	.069	.128	.060	.101	.146	.142
	3,000	.065	.109	.062	.105	.057	.093	.119	.108
	5,000	.057	.083	.056	.081	.053	.076	.080	.093

Note. $W_d$ is the Wald test based on the item-wise information matrix; $W_{\mathrm{Obs}}$ is Wald test based on the observed information matrix; $W_{\mathrm{SW}}$ is Wald test based on the sandwich-type covariance matrix; LR is the likelihood ratio test; $K_j$ is the number of required attributes by the $j$th item. The 95% confidence interval of the value which do not cover .05 is marked in boldface. DINO = deterministic inputs, noisy, “or” gate model; $\rho$ = the attribute correlation; N = the sample size.

As shown in Table 1, for the DINA model among the four statistics under investigation, the performance of the $W_d$ or $W_{\mathrm{Obs}}$ to select the most appropriate DCM tended to inflate the Type I error rates, and although the $W_{\mathrm{Obs}}$ was slightly better than that of $W_d$, the $W_{\mathrm{SW}}$ or LR test had much better Type I error rates under the same conditions. The empirical Type I error rates presented in Table 2 tell a different story. For the A-CDM, the performance of the $W_{\mathrm{Obs}}$ and $W_{\mathrm{SW}}$ was better than $W_d$ and LR in almost all cases. The Type I error rates of the LR were inflated under most conditions. The Type I error rates for the DINO model are reported in Table 3. It can be observed that only the $W_{\mathrm{SW}}$ demonstrated reasonably robust Type I error behavior across all conditions. Under the $\rho =.8$ condition the LR rejected too many well-fitting items (as high as .227).

Table 2.

Type I Error Rates for A-CDM ($\alpha =.5$).

		$W_d$		$W_{\mathrm{Obs}}$		$W_{\mathrm{SW}}$		LR
$\rho$	N	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$
0	500	.068	.098	.057	.075	.034	.035	.094	.094
	1,000	.056	.067	.051	.059	.040	.042	.083	.069
	2,000	.053	.062	.050	.058	.047	.050	.075	.065
	3,000	.052	.059	.050	.056	.047	.050	.074	.064
	5,000	.052	.055	.051	.053	.048	.050	.067	.062
.5	500	.073	.116	.063	.090	.030	.036	.078	.085
	1,000	.058	.077	.054	.068	.045	.047	.071	.072
	2,000	.055	.060	.052	.056	.048	.047	.068	.064
	3,000	.056	.060	.053	.056	.049	.051	.066	.061
	5,000	.049	.057	.046	.055	.044	.051	.062	.059
.8	500	.087	.176	.083	.162	.018	.036	.084	.111
	1,000	.065	.098	.056	.078	.038	.045	.066	.072
	2,000	.062	.072	.056	.064	.050	.049	.062	.065
	3,000	.059	.063	.054	.059	.050	.051	.058	.060
	5,000	.056	.053	.051	.049	.049	.043	.059	.058

Note. $W_d$ is the Wald test based on the item-wise information matrix; $W_{\mathrm{Obs}}$ is Wald test based on the observed information matrix; $W_{\mathrm{SW}}$ is Wald test based on the sandwich-type covariance matrix; LR is the likelihood ratio test; $K_j$ is the number of required attributes by the $j$th item. The 95% confidence interval of the value which do not cover .05 is marked in boldface. A-CDM = additive cognitive diagnostic model; $\rho$ = the attribute correlation; N = the sample size.

ANOVAs were conducted to further assess the effect of the manipulated between-subjects factors on the performance of the $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, $W_d$, or LR methods. For each method, the mean empirical Type I error was used as the ANOVA dependent variable. The ${\hat{\omega }}^2$ values for$W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR presented in Table 4 can be summarized as follows: (a) The Type I error of $W_d$ was largely affected by sample size; (b) The main effects of sample size and attribute correlation for $W_{\mathrm{Obs}}$ were large; (c) The performance of $W_{\mathrm{SW}}$ was substantially affected by the fitted DCM, and sample size; (d) The main effects of the fitted DCM and attribute correlation factors, and their interaction for LR were large. In considering the ${\hat{\omega }}^2$ values for $W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR in Table 4 and the empirical Type I error rates in Tables 1 to 3, the authors can note that (a) as the sample size increased, the performance of all statistics were considerably better; (b) the attribute correlation strongly affected the performance of $W_{\mathrm{Obs}}$ and increasing the attribute correlation leads to inflated Type I error rates; (c) the Type I error rates of the $W_{\mathrm{SW}}$ were well controlled for the DINA and A-CDM under most of the simulation conditions, but in general slightly inflated for the DINO model; and (d) for A-CDM, the Type I error of LR was closer to the nominal level as the attribute correlation increased, however, for the DINO model, the Type I error was inflated when the attribute correlation was high ($\rho =.8$).

Table 4.

${\hat{\omega }}^2$ Values for $W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR.

	$\mathrm{DCM}$	N	$\rho$	$\mathrm{DCM}\times N$	$\mathrm{DCM}\times \rho$	$N\times \rho$
$W_d$	0.015	0.467	0.125	0.092	−0.031	−0.041
$W_{\mathrm{Obs}}$	0.066	0.429	0.179	0.058	−0.022	0.010
$W_{\mathrm{SW}}$	0.343	0.260	−0.002	0.117	0.004	0.075
LR	0.142	0.127	0.198	0.006	0.334	0.027

Note. $W_d$ is the Wald test based on the item-wise information matrix; $W_{\mathrm{Obs}}$ is Wald test based on the observed information matrix; $W_{\mathrm{SW}}$ is Wald test based on the sandwich-type covariance matrix; LR is the likelihood ratio test; DCM is the fitted diagnostic classification model; N = the sample size; $\rho$ = the attribute correlation; The ${\hat{\omega }}^2$ value that is greater than 0.138 is marked in boldface.

Close inspection of the simulation results in Tables 1 to 4 reveals that, as expected, the Wald test $W_d$ was improved by using the observed information matrix or the sandwich-type covariance matrix, and among the four tests, the newly proposed $W_{\mathrm{SW}}$ had the best performance in providing reliable Type I error control across most of the simulation conditions.

Power

For the DINA and DINO models, the correct detection rates of the $W_d$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR tests were all above .831 under all of the simulation conditions, with one exception: The empirical power of $W_{\mathrm{SW}}$ was .524 ($K_j=2$) and .729 ($K_j=3$) for DINA model, when $\rho =.8$ and $N=500$. Thus, only the correct detection rates for A-CDM are provided in Table 5. The correct detection rates of $W_{\mathrm{d}}$, $W_{\mathrm{Obs}}$, $W_{\mathrm{SW}}$, and LR in Table 5 are shown in strikeout font for conditions with inflated or deflated Type I errors. As shown in Table 5, for A-CDM, when $N>500$, among the four tests under investigation in this study, only the $W_{\mathrm{SW}}$ had accurate Type I error rate control and adequate or excellent power (higher than .863) under all conditions with one exception ($\rho =.8$ and $N=1,000$). Sample size and attribute correlation influenced the power of $W_{\mathrm{SW}}$, the A-CDM tended to have less power to correctly reject the null hypothesis when the sample size was small ($N=500$), and attribute correlation was high ($\rho =.8$). This is not surprising, given that the previous study showed that the omission of interaction effect terms had little impact on classification accuracy (Kunina-Habenicht et al., 2012). Although the correct detection rates of the $W_d$, $W_{\mathrm{Obs}}$, and LR tests are similar to the patterns of the power results of the $W_{\mathrm{SW}}$, it should be pointed out that due to the inaccurate Type I error rates in the simulation, these results are not necessarily indicative of good performance in real applications.

Table 5.

The Correct Detection Rates for A-CDM ($\alpha =.5$).

		$W_d$		$W_{\mathrm{Obs}}$		$W_{\mathrm{SW}}$		LR
$\rho$	N	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$	$K_j=2$	$K_j=3$
0	500	.699	.728	.673	.683	.582	.560	.714	.733
	1,000	.928	.950	.928	.950	.916	.936	.942	.961
	2,000	1.000	1.000	1.000	1.000	1.000	1.000	.999	1.000
	3,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
	5,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
.5	500	.624	.833	.592	.800	.456	.697	.634	.823
	1,000	.884	.985	.877	.983	.863	.978	.893	.989
	2,000	.994	1.000	.994	1.000	.994	1.000	.995	1.000
	3,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
	5,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
.8	500	.495	.760	.453	.682	.179	.440	.469	.721
	1,000	.754	.955	.731	.946	.667	.922	.747	.953
	2,000	.958	.999	.955	.999	.951	.999	.965	1.000
	3,000	.994	1.000	.994	1.000	.994	1.000	.995	1.000
	5,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000

Note. $W_d$ is the Wald test based on the item-wise information matrix; $W_{\mathrm{Obs}}$ is Wald test based on the observed information matrix; $W_{\mathrm{SW}}$ is Wald test based on the sandwich-type covariance matrix; LR is the likelihood ratio test; $K_j$ is the number of required attributes by the $j$th item. The value that is smaller than .80 in each condition is in boldface. The condition with inflated or deflated Type I error is in strikeout font. A-CDM = additive cognitive diagnostic model; $\rho$ = the attribute correlation; N = the sample size.

Discussion

Many DCMs with different assumptions or theories about how an examinee’s latent attributes influence test performance are now available in education, psychology, and many other disciplines. Selecting the most appropriate DCM for each item is crucial in theory and actual applications, since no particular DCM suits all the diagnostic test items in many, if not all, diagnostic assessments. For item-level model selection, a Wald test based on a simplified statistic $W_d$ has been commonly used. However, according to several studies the $W_d$ statistic yielded inflated Type I error rates (de la Torre & Lee, 2013; Ma et al., 2016; Sorrel, Abad, et al., 2017; Sorrel, de la Torre, et al., 2017).

As an alternative, the item parameter asymptotic covariance matrix ${\hat{\mathbf{\sum }}}_j$ from the original version of the Wald statistic $W_d$ was replaced with a matrix obtained from the observed information ${\mathcal{I}}_{\mathrm{Obs}}$ or the sandwich-type matrix ${\mathcal{I}}_{\mathrm{SW}}$ to address the inflation of Type I error in this study, building on work by Liu et al. (2016), Xin et al. (2016), Philip et al. (2018), and Liu et al. (2018). Evidence from the Monte Carlo simulation suggests that when the sample size was reasonably large ($N\ge 1,000$) for all the three fitted models, the DINA, DINO, and A-CDM, the Type I error rate control of $W_{\mathrm{SW}}$ was accurate or robust. When $N\ge 1,000$, for the DINA, DINO, or A-CDM, the power of $W_{\mathrm{SW}}$ was adequate or excellent under all the simulation conditions with only one exception. Although the Type I error rate control of $W_{\mathrm{Obs}}$ was better than that of $W_d$, the $W_{\mathrm{SW}}$ had the best performance under all simulation conditions. In this study, the LR statistic demonstrated good Type I error behavior under some conditions, however, its good behavior is not guaranteed. Similar to previous studies (Sorrel, Abad, et al., 2017; Sorrel, de la Torre, et al., 2017), the authors also found that the computation of the LR was time consuming. For example, with the DINO model the LR and Wald took approximately 1 min and less than 1 s to compute in one replication, respectively, when $N=5,000$ and $\rho =.8$. According to the simulation results in this study, $W_{\mathrm{SW}}$ is the recommended tool for item-level model comparisons in empirical applications.

Although, as expected, $W_{\mathrm{Obs}}$ or $W_{\mathrm{SW}}$ had Type I error rates close to the nominal level and had good power under most of simulation conditions, and the simulation results indicate that the $W_{\mathrm{SW}}$ may serve as a practical tool for empirically selecting the most appropriate model at item level in DCMs, further research is needed. Many other factors that could potentially affect the behavior of these statistics were not manipulated in this study. For example, previous studies (Ma et al., 2016; Sorrel, Abad, et al., 2017; Sorrel, de la Torre, et al., 2017) have found that item quality, attribute distribution, the fitted model and test length affected the performance of $W_d$, and the authors believe it would be instructive to investigate how these factors affect the statistical properties of $W_{\mathrm{Obs}}$ or $W_{\mathrm{SW}}$.