A Sequential Higher Order Latent Structural Model for Hierarchical Attributes in Cognitive Diagnostic Assessments

Abstract

The higher-order structure and attribute hierarchical structure are two popular approaches to defining the latent attribute space in cognitive diagnosis models. However, to our knowledge, it is still impossible to integrate them to accommodate the higher-order latent trait and hierarchical attributes simultaneously. To address this issue, this article proposed a sequential higher-order latent structural model (LSM) by incorporating various hierarchical structures into a higher-order latent structure. The feasibility of the proposed higher-order LSM was examined using simulated data. Results indicated that, in conjunction with the deterministic-inputs, noisy “and” gate model, the sequential higher-order LSM produced considerable improvement in person classification accuracy compared with the conventional higher-order LSM, when a certain attribute hierarchy existed. An empirical example was presented as well to illustrate the application of the proposed LSM.

Introduction

Cognitive diagnosis models (CDMs) or diagnostic classification models (see, for example, Rupp, Templin, & Henson, 2010) are typically composed of two components: the measurement model and the latent structural model (LSM). The former defines the probability of an observed response given a specific attribute pattern of a respondent, whereas the latter models the probability that a respondent has each attribute pattern, which is the mixing proportion of attribute pattern in the population.

For CDMs, the most straightforward and frequently used approach to modeling the latent structure is to estimate the mixing proportion (π_c) values directly for all attribute patterns, which is referred to as the unstructured or saturated LSM. For K binary attributes, the number of attribute patterns is C = 2^K, where only 2^K– 1 mixing proportion parameters need to be estimated because $\sum _{c=1}^C{\pi }_c=1$. The number of latent structural parameters in the unstructured LSM increases exponentially with K. Thus, it is vital to impose additional constraints to create parsimonious and structured LSMs (for reviews, see chapter 8 in Rupp et al., 2010).

Among the structured LSMs, the ease of the interpretability of the conventional higher-order LSM (also known as the higher-order latent trait model; de la Torre & Douglas, 2004) makes it the most popular one. There are some advantages of using the conventional higher-order LSM. First, when the number of attributes is greater than 2, the use of the higher-order LSM reduces the number of latent structural parameters from 2^K– 1 to 2K (i.e., K attribute slopes and K attribute difficulties); Second, in practice, the attributes in a test are often conceptually related and statistically correlated; the use of the higher-order LSM explicitly accounts for the correlations among the attributes. Third, employing the higher-order LSM produces an estimate for the general ability for each respondent in addition to a profile of estimated attributes. Fourth, the higher-order LSM may be in better alignment with test designs that contain the second-order target latent variables.

Both the unstructured LSM and the conventional higher-order LSM assume that attributes are structurally independent, in that mastery of one attribute is not a prerequisite to the mastery of another attribute. However, cognitive research suggests that cognitive skills should not be investigated in isolation (Kuhn, 2001). The attribute hierarchy specifies a network of interrelated attribute mastery process (Leighton, Gierl, & Hunka, 2004). In other words, if attributes are in line with some existing cognitive theories, some attribute patterns are unlikely to be contained in the latent attribute space. Thus, the attribute hierarchy not only makes the test blueprint consistent with the cognitive theory (Gierl, Leighton, & Hunka, 2007) but also helps reduce the latent attribute space (Rupp et al., 2010).

Both higher-order LSM and the attribute hierarchy have their own advantages. Simultaneously accounting for such two concepts not only extends the application range of the conventional higher-order LSM but also allows hierarchical attributes to follow a higher-order latent structure. However, the conventional higher-order LSM cannot take into account the attribute hierarchy. Such an issue is also documented in the CDM (version 6.6-5; George, Robitzsch, Kiefer, Gross, & Uenlue, 2016) and the GDINA (generalized deterministic-inputs, noisy “and” gate; version 2.2.0; Ma & de la Torre, 2018) packages in R software (R core team, 2016), and the flexMIRT® (version 3.5; Cai, 2017).

To address this issue, this article proposes a sequential higher-order LSM for hierarchical attributes based on the sequential tree/process. The sequential tree is not a new concept as it has been used as a building block for many psychometric models for observed item responses (e.g., De Boeck & Partchev, 2012; Ma, 2019; Ma & de la Torre, 2016). However, the distinguishing feature of the sequential higher-order LSM is that the sequential tree is used to accommodate the hierarchical structure of latent attributes rather than observed item responses.

The proposed sequential higher-order LSM can be used with various measurement models for diagnostic purposes. For demonstration purpose, in this study, the deterministic-inputs, noisy “and” gate (DINA) model (Junker & Sijtsma, 2001) is used as an example. The rest of the article starts with an overview of the DINA model, the conventional higher-order LSM, and the attribute hierarchy. Then the proposed sequential higher-order LSM is presented, followed by three simulation studies evaluating the psychometric properties of the proposed model. An empirical example is also analyzed to illustrate the application of the proposed approach. Finally, the authors summarize the findings and discuss the directions for future research.

Background

DINA Model

Let y_ni be the observed response of person n (n = 1, . . ., N) to item i (i = 1, . . ., I). The DINA model can be expressed as follows:

$$p_{\mathrm{ni}}=P(y_{\mathrm{ni}}=1|{\mathbf{\alpha }}_{\mathbf{n}},{\mathbf{\Omega }}_i)=g_i+(1-s_i-g_i)\underset{k=1}{\overset{K}{\Pi }}{\alpha }_{\mathrm{nk}}^{q_{\mathrm{ik}}},$$ (1)

where p_ni is the probability of a correct response to item i for person n; α_nk is the binary variable for person n on attribute k (k = 1, . . ., K), where α_nk = 1 indicates that person n shows mastery of attribute k and α_nk = 0 indicates non-mastery, and let α_n = (α_n1, . . ., α_nK)′ be the n-th person’s attribute pattern; and Ω_i = (g_i, s_i)′ is a vector of item parameters, where s_i and g_i are the slip and guessing parameters, respectively, of item i. The Q-matrix (Tatsuoka, 1983) is an I-by-K confirmatory matrix with element q_ik indicating whether attribute k is required to answer item i correctly: q_ik = 1 if the attribute is required, and 0 otherwise.

Conventional Higher Order LSM

The conventional higher-order LSM can be expressed as follows:

$$m_{\mathrm{nk}}=P({\alpha }_{\mathrm{nk}}=1|{\theta }_n)=\frac{\exp ({\lambda }_k{\theta }_n-{\beta }_k)}{1+\exp ({\lambda }_k{\theta }_n-{\beta }_k)},$$ (2)

where m_nk is the probability of mastering attribute k for respondent n given θ_n; θ_n is the higher-order latent trait of respondent n, and assumed to follow the standard normal distribution for model identification; and λ_k and β_k are the slope and difficulty parameters of attribute k, respectively; to reduce the computational burden, the attribute slope parameter λ_k can be further constrained as λ_k = λ or λ_k = 1 for all attributes (de la Torre, Hong, & Deng, 2010).

According to the assumption of conditional independence, the m_nks are conditionally independent given θ_n. The conventional higher-order LSM allows for all C = 2^K possible attribute patterns, and the probability of respondent n having attribute pattern c (c = 1, . . ., C) is

$${\pi }_{\mathrm{nc}}=\underset{k=1}{\overset{K}{\Pi }}{m}_{\mathrm{nk}}^{{\alpha }_{\mathrm{ck}}}{(1-m_{\mathrm{nk}})}^{1-{\alpha }_{\mathrm{ck}}},$$ (3)

where $\sum _{c=1}^C{\pi }_{\mathrm{nc}}=1.$

Attribute Hierarchy

The attribute hierarchy represents theoretical assumptions about which attribute patterns are impossible to be observed. Figure 1 presents four prototypical attribute hierarchies (Leighton et al., 2004). To distinguish from the unstructured LSM, the original unstructured hierarchy is referred to as the outspread hierarchy throughout this article. Of the four hierarchies, take the linear hierarchy as an example. Six attributes are involved in the linear hierarchy, where A1 is a prerequisite for A2, A2 a prerequisite for A3, and so on. If attribute k is a prerequisite for attribute k′, attribute k is called a parent of attribute k′ and attribute k′ is a child attribute of attribute k. Under the linear hierarchy, whenever an attribute is mastered, its parent attribute must be mastered too. Thus, there are only C* = 7 permissible attribute patterns, namely, (000000), (100000), (110000), (111000), (111100), (111110), and (111111). Likewise, there are 8¹, 16, and 33 permissible patterns for convergent, divergent, and outspread hierarchies, respectively (see Online Appendix B). It should be noted that, for the attainable attribute patterns, the mixing proportions of them still sum to one.

Figure 1.

Four attribute hierarchies (Leighton, Gierl, & Hunka, 2004).

As aforementioned, currently, the attribute hierarchy and the higher-order latent structure cannot be simultaneously considered, which implies that, when a certain attribute hierarchy exists, the number of permissible attribute patterns is less than 2^K, which does not match the assumption of conditional independence of the conventional higher-order LSM. In addition, in the conventional higher-order LSM, the estimated latent structural parameters are attribute slopes and difficulties rather than the mixing proportions; therefore, it is impossible to assign zero priors/proportions to the unattainable attribute patterns. Note that, as de la Torre et al. (2010) mentioned, the conventional higher-order LSM might have some adjusting ability to assign unattainable attribute patterns with low proportions by regulating the attribute difficulty parameters. For example, specifying high attribute difficulty parameters to child attributes yields low mastery probabilities, whereas specifying low attribute difficulty parameters to parent attributes produces high mastery probabilities. Take two linear hierarchical attributes as an example, see Table 1 and Figure 2. Let us assume m_n1 and m_n2 are the probabilities of mastering A1 and A2 given a higher-order latent trait of respondent n, θ_n, respectively. As A1 is a parent of A2, we assume m_n1 = 0.8 and m_n2 = 0.4. Then, in conventional higher-order LSM, the mixing proportion of the impossible pattern, (01), is (1 –m_n1)m_n2 = 0.08, and the mixing proportions of all other permissible attribute patterns cannot sum to 1. Thus, it still does not guarantee a hierarchical attribute structure, where the proportion of unattainable patterns must be zero.

Table 1.

Mixing Proportion of Attribute Patterns Based on Two Linear Hierarchical Attributes.

Attribute pattern		Mixing proportion		Example
		Conventional	Sequential	Conventional	Sequential
√	(00)	(1 –mn1)(1 –mn2)	1 –mn1	0.12	0.20
√	(10)	mn 1(1 –mn2)	mn 1(1 –mn2)	0.48	0.48
×	(01)	(1 –mn1)mn2	0	0.08	0.00
√	(11)	mn 1 mn 2	mn 1 mn 2	0.32	0.32

Note. Conventional = conventional higher-order latent structural model; sequential = sequential higher-order latent structural model; √ = permissible; × = impermissible; m_n1 = 0.8; m_n2 = 0.4.

Figure 2.

A sequential linear mastery process of two attributes.

Sequential Higher Order LSM

Sequential Mastery Tree

Under the aforementioned attribute hierarchies, attributes are mastered in a (partially) sequential manner, which is referred to as a sequential mastery tree with a series of latent mastery process. Specifically, the attributes with parents are activated after their parents have been mastered, and the activation yields two possible statuses—either mastery or non-mastery. Figures 2, 3, and 4 depict three basic modules representing the relation among the permissible attribute patterns and latent mastery processes. They can be seen as simple sequential mastery trees and can serve as building blocks for more complicated sequential mastery trees.

Figure 3.

A sequential divergent mastery process of three attributes.

Figure 2 is a linear module with two attributes. Specifically, the circle denotes an attribute, and the dotted line denotes a latent mastery process of an attribute with two exclusive outcomes—mastery (abbreviated to “M”) or non-mastery (abbreviated to “N”), which are represented by rectangles. The probabilities of having such two outcomes for an individual with a higher-order ability θ are referred to as sequential probabilities and presented on the corresponding branches. Specifically, m_nk (Equation 2) is used to denote the probability of mastering attribute k for an individual with higher-order ability who has already mastered all parents of attribute k. The arrow lines indicate the sequential structure of attributes, in that A1 is a prerequisite for A2. The first latent mastery process yields the mastery of A1 with a probability of m_n1 and the non-mastery with a probability of 1 –m_n1. When A1 is mastered, A2 is activated. The probability of mastering A2 after it is activated is m_n2. Finally, this sequential mastery tree produces three permissible patterns, (00), (10), and (11), and π_n(00), π_n(10), and π_n(11) are used to denote their proportions. It is important to emphasize that, based on the logic of sequential tree, non-master A1 is equivalent to non-master both A1 and A2 (and all possible child attributes, if exist). Thus, π_n(00) = 1 –m_n1 rather than (1 –m_n1)(1 –m_n2). Likewise, π_n(10) = m_n1(1 –m_n2) and π_n(11) = m_n1m_n2, also see Table 1. Obviously, π_n(00)+π_n(10)+π_n(11) = 1.

Figures 3 and 4 give the divergent and convergent modules with three attributes. In the divergent module, one parent attribute is a prerequisite for more than one child attributes (i.e., A1 is a prerequisite of both A2 and A3), whereas in the convergent module, more than one parent attributes share a common child attribute (i.e., A1 and A2 are both prerequisites for A3). When K* attributes have the same parent or child attribute, they are called parallel attributes and are viewed as a single super-attribute with 2^K* possible outcomes. For example, the first latent mastery process in the divergent module in Figure 3 yields the outcome non-mastery with a probability of 1 –m_n1. Complementary, after mastering the first attribute with a probability of m_n1, individuals have four permissible mastery profiles associated with the super-attribute that contains both A2 and A3, namely, N_A2 and N_A3 with probability (1 –m_n2)(1 –m_n3), M_A2 and N_A3 with probability m_n2(1 –m_n3), N_A2 and M_A3 with probability (1 –m_n2)m_n3, and M_A2 and M_A3 with probability m_n2m_n3. Finally, five permissible patterns exist, with π_n(000) = 1 –m_n1, π_n(100) = m_n1(1 –m_n2)(1 –m_n3), π_n(110) = m_n1m_n2(1 –m_n3), π_n(101) = m_n1(1 –m_n2)m_n3, and π_n(111) = m_n1m_n2m_n3. Likewise, for the convergent module, five permissible patterns exist, with π_n(000) = (1 –m_n1)(1 –m_n2), π_n(100) = m_n1(1 –m_n2), π_n(010) = (1 –m_n1)m_n2, π_n(110) = m_n1m_n2(1 –m_n3), and π_n(111) = m_n1m_n2m_n3.

Figure 4.

A sequential convergent mastery process of three attributes.

A more general equation can be formulated with the help of a prespecified C*-by-K sequential Z-matrix (see Equation 4), which is the association between permissible attribute patterns and outcomes of latent mastery processes. Specifically, z_c*k = 1 means that attribute pattern c* requires the mastery of attribute k, and z_c*k = 0 indicates non-mastery of the attribute k in attribute pattern c*. Note that z_c*k = NA (or any number) means attribute pattern c* does not activate attribute k. In addition, another prespecified C*-by-K assistant D-matrix (see Equation 5) is also needed, where d_c*k = 0 when Z_c*k is NA, and d_c*k = 1 otherwise.

$${\mathbf{Z}}_{\mathrm{linear}}=\left[\begin{array}{cc}0\hfill & \mathrm{NA}\hfill \\ \multicolumn{1}{c}{1}& 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill \end{array}\right],{\mathbf{Z}}_{\mathrm{divergent}}=\left[\begin{array}{ccc}0\hfill & \mathrm{NA}\hfill & \mathrm{NA}\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \end{array}\right],{\mathbf{Z}}_{\mathrm{convergent}}=\left[\begin{array}{ccc}0\hfill & 0\hfill & \mathrm{NA}\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & \mathrm{NA}\hfill \\ \multicolumn{1}{c}{0}& 1\hfill & \mathrm{NA}\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \end{array}\right],$$ (4)

$${\mathbf{D}}_{\mathrm{linear}}=\left[\begin{array}{cc}1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill \end{array}\right],{\mathbf{D}}_{\mathrm{divergent}}=\left[\begin{array}{ccc}1\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \end{array}\right],{\mathbf{D}}_{\mathrm{convergent}}=\left[\begin{array}{ccc}1\hfill & 1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 1\hfill \end{array}\right],$$ (5)

Then, the probability of respondent n having attribute pattern c* can be written as

$${\pi }_{\mathrm{nc}*}=\underset{k=1}{\overset{K}{\Pi }}{\left[m_{\mathrm{nk}}^{z_{c*k}}{(1-m_{\mathrm{nk}})}^{1-z_{c*k}}\right]}^{d_{c*k}},$$ (6)

where z_c*k is the element of Z-matrix, d_c*k is the element of D-matrix, and m_nk is the sequential probability of respondent n mastering attribute k, see Equation 2, and obviously for $\sum _{c*=1}^{C*}{\pi }_{\mathrm{nc}*}=1$ all C* patterns. The resulting LSM is referred to as the sequential higher-order LSM.

Comparing Equations 3 and 6, when no attribute hierarchy exists, one has C* = C = 2^K, which means the Z-matrix contains all 2^K attribute patterns. The D-matrix has all elements of 1s, and the sequential higher-order LSM is equivalent to the conventional higher-order LSM. In other words, the conventional higher-order LSM can be seen as a special case of the sequential higher-order LSM when attributes are structurally independent.

Like the conventional higher-order LSM, the sequential higher-order LSM can be combined with many measurement models. As aforementioned, the DINA model is used in this study and referred to as the sequential higher-order DINA (denoted as the SHO-DINA) model. The conventional higher-order DINA model is denoted as the HO-DINA model.

Bayesian Parameter Estimation

Similar to the HO-DINA model, the parameters of the SHO-DINA model can be estimated using the Bayesian approach with the Markov chain Monte Carlo (MCMC) method. In this study, the JAGS (Just Another Gibbs Sampler) was used to estimate the parameters. By default, the JAGS uses the Gibbs sampler (Gelfand & Smith, 1990), and the sample code for SHO-DINA model is provided in Online Appendix A. More details about how to use the JAGS for Bayesian CDM estimation can be found in a tutorial by Zhan, Jiao, Man, & Wang (2019).

To begin with, item responses are assumed to be independently distributed following a Bernoulli distribution: y_ni ~ Bernoulli(p_ni). Imposing the monotonicity restriction that g_i < 1 –s_i for all items, the priors of item parameters are specified as follows $s_i~\mathrm{\beta }(a_s,b_s)$ and $g_i~\mathrm{\beta }(a_g,b_g)\mathrm{I}(0,1-s_i)$, and the scale parameters are assigned as a_s = b_s = a_g = b_g = 1. The priors of latent structural parameters are specified as ${\lambda }_k~\mathrm{Normal}(0,{\sigma }_{\lambda }^2)\mathrm{I}({\lambda }_k>0)$ and ${\beta }_k~\mathrm{Normal}(0,{\sigma }_{\beta }^2)$. Here ${\sigma }_{\lambda }^2={\sigma }_{\beta }^2=2$. Then m_nk and ${\pi }_{\mathrm{nc}*}$ is computed from Equations 2 and 6, respectively. Furthermore, $x_n*~\mathrm{Categorical}({\mathbf{\pi }}_n)$, where ${\mathbf{\pi }}_n=({\pi }_{n1},\dots ,{\pi }_{\mathrm{nc}*},\dots ,{\pi }_{\mathrm{nC}*})\prime$ is a C* dimensional probabilistic vector of respondent n belongs to every permissible attribute patterns; x_n* indicating respondent n′s attribute pattern is assumed to follow a categorical distribution, with the probability of membership in x*th profile, then ${\mathbf{\alpha }}_n={\mathbf{\alpha }}_{x_n*}$ (i.e., ${\mathrm{\alpha }}_{\mathrm{nk}}={\mathrm{\alpha }}_{x_n*k}$). Finally, the posterior mean is treated as the estimated value for continuous parameters (e.g., s_i, g_i, and ${\pi }_{\mathrm{nc}*}$) and the posterior mode is treated as the estimated value for categorical parameters (e.g., α_nk), respectively.

Simulation Studies

Three simulation studies were conducted to evaluate the performance of the SHO-DINA model under various conditions. The purpose of Simulation Study 1 is twofold: (a) to examine whether parameters of the SHO-DINA can be recovered accurately and (b) to evaluate whether the sequential higher-order LSM can provide more accurate classifications than the conventional higher-order LSM when certain attribute hierarchies exist.

It is not unexpected that the SHO-DINA model outperforms the HO-DINA model when the data were generated from the SHO-DINA model. To investigate whether the improvement is due to the attribute hierarchy, in Simulation Study 2, the DINA model with attribute hierarchy (denoted as the DINA-H model) (e.g., Akbay, 2016; Su, Choi, Lee, & McAninch, 2013; de la Torre et al., 2010) was used as the true model to generate data. In the DINA-H model, the prior probabilities or mixing proportions of impossible attribute patterns were fixed at zero. In such cases, if the SHO-DINA model still outperforms the HO-DINA model, one can say that it is caused by the fact that the attribute hierarchy is accommodated by the SHO-DINA model. In addition, the consequences of additionally imposing a higher-order latent structure in which the data simulated the DINA-H model also can be evaluated.

In addition, a basic assumption of the SHO-DINA model is that the attribute hierarchy is correctly specified. However, expert-defined attribute hierarchies may not be always correct (Liu, 2018). Thus, Simulation Study 3 aims to assess the impact of misspecified attribute hierarchy on parameter recovery of the proposed model. The results of the Simulation Study 3 were omitted for brevity but can be found in the online appendix (see section S1).

Simulation Study 1

Design and data generation

In Simulation Study 1, four factors were manipulated. First, sample size N = 500 and 1,000, and test length I = 15 and 30. Second, four attribute hierarchies, including the linear, convergent, divergent, and outspread structures as shown in Figure 1, were considered. The corresponding sequential mastery trees can be found in Online Appendix B. The model that used to analyze the data includes the SHO-DINA model and the HO-DINA model. Six attributes were measured; s_i and g_i were generated from a uniform distribution, U(.01, .3). The higher-order latent traits were generated from a standard normal distribution. For latent structural parameters, λ_ks were all set as 1.5 and β_k = (–1.0, – .75, – .5, .5, .75, 1.0)′ for six attributes. The true pattern of each person was generated from a categorical distribution with parameter ${\mathbf{\pi }}_n=({\pi }_{n1},\dots ,{\pi }_{\mathrm{nc}*},\dots ,{\pi }_{\mathrm{nC}*})\prime$, which was computed based on Equation 6.

Note that only the structured Q-matrix (Akbay, 2016; Tu, Wang, Cai, Douglas, & Chang, 2018) was used, which means items are not allowed if they require only the child attribute but not include the parent attributes (von Davier & Haberman, 2014). The structured Q-matrices under different attribute hierarchies are given in Figure 5. Besides the necessary reachability matrices (first six items for I = 15 and first 12 items for I = 30, respectively) for completeness (Ding, Yang, & Wang, 2010), required attribute patterns for the rest of items were chosen from the corresponding reduced Q-matrix (Tatsuoka, 1983).

Figure 5.

Simulated K-by-I structured Q′-matrices in Simulation Study 1.

Note.“*” denotes items used in the I = 15 conditions; “gray” means “1” and “blank” means “0.”

Analysis

Thirty replications were implemented in each condition. In each replication, two Markov chains with random starting points were used, and each chain ran 15,000 iterations, with the first 10,000 iterations in each chain as burn-in. Finally, the remaining 10,000 iterations for the model parameter inferences. The potential scale reduction factor (PSRF; Brooks & Gelman, 1998) was computed to assess the convergence of every parameter. Values of PSRF less than 1.1 or 1.2 indicate convergence (Brooks & Gelman, 1998; de la Torre & Douglas, 2004). Our studies indicated that PSRF was generally less than 1.01, suggesting good convergence for the settings specified. To evaluate parameter recovery, the bias and root mean square error (RMSE) of the item parameter estimates and posterior mixing proportion were computed:$\mathrm{bias}\left(\widehat{\omega }\right)={\sum }_{r=1}^R\raisebox{1ex}{${\widehat{\omega }}_r-\omega $}\!\left/ \!\raisebox{-1ex}{$R$}\right.$, $\mathrm{RMSE}\left(\widehat{\omega }\right)=\sqrt{{\sum }_{r=1}^R\raisebox{1ex}{${({\widehat{\omega }}_r-\omega )}^2$}\!\left/ \!\raisebox{-1ex}{$R$}\right.}$, where $\hat{\omega }$ and $\omega$ is the estimated value and the true value of model parameters, respectively; R is the total number of replications. For attribute recovery, the attribute and pattern correct classification rate (i.e., ACCR and PCCR) were computed to evaluate the classification accuracy of individual attributes and profiles: $\mathrm{ACCR}={\sum }_{r=1}^R{\sum }_{n=1}^N\raisebox{1ex}{$\mathrm{I}({\widehat{\mathrm{\alpha }}}_{nkr}={\mathrm{\alpha }}_{nk})$}\!\left/ \!\raisebox{-1ex}{$N\times R$}\right.$ and $\mathrm{PCCR}={\sum }_{r=1}^R{\sum }_{n=1}^N\raisebox{1ex}{$\mathrm{I}({\widehat{\mathbf{\alpha }}}_{nr}={\mathbf{\alpha }}_n)$}\!\left/ \!\raisebox{-1ex}{$N\times R$}\right.$, where I(·) is an indicator function.

Results

Figures S1 and S2 in the online appendix display the bias and the RMSE of item parameter estimates, respectively. First, the patterns in the recovery of item parameters for the SHO-DINA model were almost the same as those for the HO-DINA model. This means that, for a certain attribute hierarchy, the recovery of item parameters was almost unaffected by ignoring the sequential tree or attribute hierarchy in the HO-DINA model. Second, guessing parameters were generally estimated more accurately than slip parameters, probably because the number of individuals who mastered all required attributes is typically less than the number of individuals who do not master all required attributes. Third, increasing sample size and test length yields better recovery of the item parameters. Fourth, it seems that different attribute hierarchies have no significant effect on the recovery of item parameters.

Figure 6 summarizes the recovery of individual attribute patterns. Across all conditions, the PCCRs of the SHO-DINA model were higher than those of the HO-DINA model. The differences in PCCRs between the SHO-DINA and HO-DINA models appeared substantial for all structures, but more marked under the linear structure. A possible explanation is that if a structure is more restricted with fewer permissible attribute patterns, the CDMs that cannot take attribute hierarchy into consideration are prone to classify individuals into incorrect latent classes. More specifically, there are 7, 8, 16, and 33 permissible patterns in the linear, convergent, divergent, and outspread hierarchies, respectively. This makes the linear and convergent structures particularly challenging for the conventional HO-DINA model. In addition, for a given model, the PCCR was higher when the sample size was larger, and the test length was longer.

Figure 6.

The PCCR in Simulation Study 1.

Note. The PCCR values were estimated from 30 replications. SHO-DINA = the sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate; PCCR = pattern correct classification rate.

Figure 7 displays the RMSE of mixing proportions in conditions with N = 1,000 and I = 30 (for the bias and the RMSE in other conditions, see Figures S3 and S4 in the online appendix, respectively). Using the SHO-DINA model, the bias of mixing proportions was close to zero and the RMSEs were all less than .1 across all conditions. In contrast, under the HO-DINA model, the bias and RMSE can be quite large in their absolute value, especially for attribute pattern (000000). It seems that the HO-DINA model tends to overestimate the ability of students who do not master any attributes.

Figure 7.

The RMSE of mixing proportions in conditions with N = 1,000 and I = 30 in Simulation Study 1.

Note. There are 7, 8, 16, and 33 permissible attribute patterns for linear, convergent, divergent, and outspread hierarchy, respectively. SHO-DINA = the sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate; RMSE = root mean square error.

Regarding the higher-order latent trait, the recovery of the SHO-DINA model was slightly better than that of the HO-DINA model, especially for the linear structure (details can be found in Figure S5 in the online appendix). In addition, for the SHO-DINA model, the correlation between the true and the estimated values ranged from .655 to .735 (M = .690, SD = .024); for the HO-DINA model, the correlation ranged from .635 to .738 (M = .689, SD = .028).

Simulation Study 2

Data generation and analysis

In Simulation Study 2, two factors were manipulated. First, four attribute hierarchies in Figure 1 were still considered. Second, the models that used to analyze the data include the DINA-H model, the HO-DINA model, and the SHO-DINA model. As aforementioned, the DINA-H model was used as the true model to generate data. The structured Q-matrices in Figure 5 were still used. The test length and the sample size were fixed at 30 and 1,000, respectively. The true attribute profile of each person was randomly chosen from all possible patterns (C* = 7, 8, 16, and 33 for linear, convergent, divergent, and outspread hierarchies, respectively) with equal probability. The s_i and g_i were still generated from a uniform distribution, U(.01, .3). Finally, the observed responses were generated from Bernoulli(p_ni), where p_ni was given in Equation 1. Thirty data sets were generated in each condition. Analysis processes were identical to those in Simulation Study 1.

Results

Tables S1 and S2 in the online appendix present the recovery of item parameters. First, consistent with the results in Simulation Study 1, the patterns in the recovery of item parameters for the SHO-DINA model were almost the same as those for the HO-DINA model. Second, no matter the bias or the RMSE, the SHO-DINA model and the HO-DINA model were quite similar with the baseline model, that is, the DINA-H model, indicated that imposing the sequential tree surely has negligible impact on the measurement of item parameters.

Figure 8 displays the recovery of latent attributes (more details can be found in Table S3 in the online appendix), and several findings can be observed. First, across all conditions, the ACCRs and PCCRs of the SHO-DINA model were almost identical to those of the DINA-H model and were all higher than those of the HO-DINA model. Second, consistent with the results in Simulation Study 1, the performance of the HO-DINA model worsened more markedly under the more compact hierarchy.

Figure 8.

Summary of the recovery of attributes in Simulation Study 2.

Note. SHO-DINA = the sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate; DINA-H = deterministic-inputs, noisy “and” gate model with attribute hierarchy.

Overall, the performance of the SHO-DINA model was almost identical to that of the DINA-H model but the former can additionally provide the estimates of the higher-order latent trait and latent structural parameters (see Figure S6 and Table S4 in the online appendix, respectively). The higher-order latent trait reflects the overall performance of each respondent. The estimates of latent structural parameters can be used to reflect the attribute hierarchy. For example, in the linear hierarchy, the attribute difficulty increased monotonically with the order of the attributes. More importantly, the results of Simulation Study 2 indicate that the advantages of the SHO-DINA over the HO-DINA are caused by the capability of accommodating the attribute hierarchy.

An Empirical Example

Data Description and Analysis

To further demonstrate the application of the sequential higher-order LSM, the examination for the certificate of proficiency in English (ECPE) data (Templin & Hoffman, 2013) were used. In the ECPE data, 2,922 persons answered 28 multiple-choice items with three attributes, namely, (A1) morphosyntactic rules, (A2) cohesive rules, and (A3) lexical rules. Note that an unstructured Q-matrix was originally used in the ECPE data. Actually, it is unreasonable to construct an unstructured Q-matrix when a certain attribute hierarchy is established prior to item development by using a theory-driven approach via content experts (Gierl et al., 2007; von Davier & Haberman, 2014). Even when the attribute hierarchy is explored after data analysis (e.g., Templin & Bradshaw, 2014), the originally unstructured Q-matrix should be converted to a structured Q-matrix in subsequent studies. According to the study by Templin and Bradshaw (2014), a linear hierarchy exists among such three attributes: A3 → A2 → A1. Thus, it is expected that there are four rather than eight permissible attribute patterns, that is, (000), (001), (011), and (111). The original unstructured Q-matrix and the corresponding structured Q-matrix² were used and given in Figure 9. The SHO-DINA model and the HO-DINA model were employed to fit the data. The deviance information criterion (DIC; Spiegelhalter, Best, Carlin, & van der Linde, 2002) was obtained for each model to evaluate the model-data fit.

Figure 9.

The unstructured and structured K×I Q′-matrices for the ECPE data.

Note. ECPE = examination for the certificate of proficiency in English.

Results

Table 2 presents the DIC of the SHO-DINA model and the HO-DINA model with the unstructured and structured Q-matrices. The SHO-DINA model fits the data better than the HO-DINA model, regardless of the Q-matrix. In addition, the DIC was smaller when using the structured Q-matrix, indicating the structured Q-matrix is better than the original unstructured Q-matrix. However, comparing with the HO-DINA model, the improvement in model-data fit by using the structured Q-matrix was less apparent for the SHO-DINA model, because the linear hierarchy has been imposed into the latent attribute space of the SHO-DINA model.

Table 2.

The DIC for the ECPE Data.

Analysis model	Q-matrix	DIC
SHO-DINA	Unstructured	85,096.695
	Structured	85,020.798
HO-DINA	Unstructured	87,019.968
	Structured	85,157.668

Note. DIC = deviance information criterion; ECPE = examination for the certificate of proficiency in English. SHO-DINA = sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate.

Similar to previous studies (e.g., Akbay, 2016; Templin & Bradshaw, 2014; Templin & Hoffman, 2013), the estimated guessing parameters of many items were unusually high. For the SHO-DINA model, the item parameter estimates (see Table S5 in the online appendix) using the structured and unstructured Q-matrices were virtually identical. In addition, with the structured Q-matrix, the SHO-DINA and the HO-DINA models had almost identical parameter estimates too, which is consistent with the finding from the simulation studies.

Table 3 presents the correlation coefficients between the estimated higher-order latent traits using the SHO-DINA and HO-DINA models. The corresponding scatter plots were given in Figure S12 in the online appendix. Comparing the results of the SHO-DINA model and those of the HO-DINA model, they have quite high correlations, which may indicate the same latent trait was estimated by them.

Table 3.

The Correlation Coefficients Among the Estimated Higher Order Latent Traits for the ECPE Data.

		Structured		Unstructured
		SHO-DINA	HO-DINA	SHO-DINA	HO-DINA
Structured	SHO-DINA	1
	HO-DINA	.995	1
Unstructured	SHO-DINA	.992	.992	1
	HO-DINA	.991	.985	.983	1

Note. ECPE = examination for the certificate of proficiency in English; structured = structured Q-matrix; unstructured = unstructured Q-matrix; SHO-DINA = sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate.

Table 4 presents the posterior mixing proportion. The sum of the posterior mixing proportion was 1 for the SHO-DINA model regardless of the Q-matrix; however, the sum was .946 for the HO-DINA model with unstructured Q-matrix, suggesting that a few respondents (about 5.4% of the sample) were estimated to have other improbable patterns. In addition, Table 5 presents the tetrachoric correlations among three estimated attributes. For the SHO-DINA model with unstructured and structured Q-matrices, the tetrachoric correlations were all close to 1, which can be treated as the evidence of the linear hierarchy (Rupp et al., 2010).³ Note that, for the HO-DINA model with structured Q-matrix, the posterior mixing proportion and the tetrachoric correlation were quite different from the other three conditions. Such results may be related to the findings in Akbay (2016) that when the model does not consider the attribute hierarchy, using structured Q-matrix could decrease the classification accuracy of respondents. Essentially, when the model does not take into account the attribute hierarchy, at least one identity matrix (i.e., the reachability matrix for structurally independent attributes) is needed for Q-matrix for completeness (Köhn & Chiu, 2017). Thus, such results may be caused by the use of the structured Q-matrix, which is incomplete for the HO-DINA model.

Table 4.

Posterior Mixing Proportion for the ECPE Data.

Permissible patterns	Unstructured		Structured
	SHO-DINA	HO-DINA	SHO-DINA	HO-DINA
(000)	.352	.319	.353	.000
(001)	.062	.063	.062	.012
(011)	.112	.098	.112	.104
(111)	.473	.466	.473	.518
Sum	1.000	.946	1.000	.633

Note. ECPE = examination for the certificate of proficiency in English; unstructured = unstructured Q-matrix; structured = structured Q-matrix; SHO-DINA = sequential higher-order–deterministic-inputs, noisy “and” gate; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate.

Table 5.

Tetrachoric Correlations Among Three Estimated Attributes for the ECPE Data.

	Unstructured			Structured
	A1	A2	A3	A1	A2	A3
A1	1.00	.92	.92	1.00	.21	.43
A2	.99	1.00	.95	.99	1.00	−.38
A3	.98	.99	1.00	.98	.99	1.00

Note. Upper and lower triangular matrix gives the results of the HO-DINA model and the SHO-DINA model, respectively; ECPE = examination for the certificate of proficiency in English; unstructured = unstructured Q-matrix; structured = structured Q-matrix; A1 = morphosyntactic; A2 = cohesive; A3 = lexical; HO-DINA = higher-order–deterministic-inputs, noisy “and” gate; SHO-DINA = sequential higher-order–deterministic-inputs, noisy “and” gate.

Summary and Discussion

In this article, the authors developed a new higher-order LSM for hierarchical attributes—the sequential higher-order LSM. Unlike the conventional higher-order LSM, by taking the sequential tree into account, the sequential higher-order LSM is able to accommodate various attribute hierarchies and simultaneously retains the advantages of the higher-order latent structure. Three simulation studies showed that (a) the proposed Bayesian MCMC estimation algorithm can produce accurate model parameter recovery; (b) the sequential higher-order LSM can provide more accurate person classifications and mixing proportion estimation than the conventional higher-order LSM, when a certain attribute hierarchy exists; (c) the item parameter estimates were almost unaffected by imposing the sequential tree, especially with structured Q-matrix; (d) the performance of the SHO-DINA model was almost identical to that of the DINA-H model when the latter was the true model for data generation, but the SHO-DINA model can additionally provide higher-order latent trait estimates and latent structured parameter estimates; and (e) for the SHO-DINA model, misspecified attribute hierarchy tends to worsen the person classification accuracy, but has little impact on the estimation of item parameters. An empirical example was used to illustrate the applications and advantages of the proposed model. First, the SHO-DINA model fitted the ECPE data better than the HO-DINA model regardless of the Q-matrix. Second, for the SHO-DINA model, the parameter estimates using a structured or unstructured Q-matrix are almost identical. In addition, when an attribute hierarchy and its corresponding structured Q-matrix have been established, the models (e.g., the HO-DINA model) that do not consider the attribute hierarchy may not be appropriate. However, how to determine attribute hierarchy is still an open question (e.g., Akbay, 2016; Yu, Ding, Qin, & Liu, 2011).

As an extension of the conventional higher-order LSM, the sequential higher-order LSM extends the application of higher-order latent structure from structurally independent attributes situations to hierarchical attributes situations. Some existing methods that employed the conventional higher-order LSM can be further extended to hierarchical attributes by using the proposed LSM. For instance, the cognitive diagnosis computerized adaptive testing using the HO-DINA model (Hsu & Wang, 2015); the joint cognitive diagnosis modeling framework for responses and response times (Zhan, Jiao, & Liao, 2018); the hierarchical diagnostic item response model (Hansen, Cai, Monroe, & Li, 2016); CDMs incorporating covariates in the higher-order LSM (Ayers, Rabe-Hesketh, & Nugent, 2013); and the longitudinal cognitive diagnosis modeling using multidimensional higher-order LSM (Zhan, Jiao, Liao, & Li, 2019).

This study assumed that the attribute hierarchy is correctly specified, which, nevertheless, may not be the case in practice. The impact of misspecified attribute hierarchy has been briefly studied in Simulation Study 3, but further investigations may be carried out. In addition, in practice, different students may follow different learning paths. It is possible that these learning paths cannot be accommodated by a single attribute hierarchy; in such a case, one may consider adopting more complex models to simultaneously handle multiple attribute hierarchies or models without attribute hierarchy.

It is worth noting that the number of parameters in the proposed LSM is 2K. Thus, for some attribute hierarchies, the number of latent structural parameters in the DINA-H model may be less than that in the SHO-DINA model. For instance, for the linear hierarchy, there are K+ 1 latent structural parameters in the DINA-H model, which is less than 2K when K > 1. Fortunately, however, the number of parameters in the proposed LSM can be further reduced by constraining the attribute slope parameter, for example, constraining λ_k = λ and λ_k = 1 will reduce the number of latent structural parameters to K+ 1 and K, respectively.

Despite promising results, further studies are still needed. First, this study assumed that the sequential mastery probability was only affected by a higher-order latent trait (see Equation 2). In practice, some covariates may also affect the sequential mastery probability, such as gender, educational interventions, and the attributes the student has already mastered. Incorporating these covariates into the model is an interesting topic. Second, the Bayesian MCMC approach is employed in this study. Other estimation methods for the proposed LSM can also be developed in the future and incorporated into existing software. Third, only a DINA-based model was employed as the measurement model, though other measurement models such as the G-DINA model (de la Torre, 2011) and its special cases can be used. The performance of the proposed LSM together with other measurement models requires further investigation. Fourth, Templin and Bradshaw (2014) proposed a hierarchical CDM to accommodate attribute hierarchy by imposing constraints to item parameters in the measurement model, which is different from the idea that considers the attribute hierarchy in LSM in this study. Further comparisons of such two ideas are needed.