Abstract
E-learning systems are capable of providing more adaptive and efficient learning experiences for learners than traditional classroom settings. A key component of such systems is the learning policy. The learning policy is an algorithm that designs the learning paths or rather it selects learning materials for learners based on information such as the learners’ current progresses and skills, learning material contents. In this article, the authors address the problem of finding the optimal learning policy. To this end, a model for learners’ hierarchical skills in the E-learning system is first developed. Based on the hierarchical skill model and the classical cognitive diagnosis model, a framework to model various mastery levels related to hierarchical skills is further developed. The optimal learning path in consideration of the hierarchical structure of skills is found by applying a model-free reinforcement learning method, which does not require any assumption about learners’ learning transition processes. The effectiveness of the proposed framework is demonstrated via simulation studies.
Keywords: personalized learning, reinforcement learning, hidden Markov model, Markov decision process, cognitive diagnostic model, attribute hierarchy model
Introduction
Designing the optimal learning path, which is accomplished by selecting the most appropriate learning materials to each individual learner based on their historical information, has emerged as a promising and important topic in recent years, along with the widespread shift from traditional classroom teaching to E-learning systems (Means et al., 2009). Thanks to online learning technologies, information such as learners’ test results and response times can be monitored which allows for E-learning systems to first estimate learners’ latent skills using psychometric models and then design the optimal learning path for each individual based on the estimated results. This can be done in such a way as to minimize the total learning time, which is the total time required for individual learners to master skills. For example, learners can be routed to the finest learning materials with appropriate contents and difficulty levels based on their learning history and estimated skill status, instead of following a standardized learning path that does not differentiate between individual learners. This notion is referred to as personalized learning, adaptive learning, or smart learning (S. Zhang & Chang, 2016).
Several studies have provided novel methods to E-learning systems. One such approach is to track learners’ skill acquisition and model changes in their skills, creating what are referred to as learning paths. For example, cognitive diagnosis models (CDMs), known as the foundation of assessing learners’ mastery of skills, are extended to model learners’ learning paths (Y. Chen, Culpepper, Wang, & Douglas, 2018; Wang et al., 2018). The knowledge tracing method (Corbett & Anderson, 1994) functions similarly in modeling learning, but focuses on one skill each time. These individualized, model-based learning trajectories can be used as priors to provide learning advice for learners in the future. Another direction toward personalized learning is finding the optimal learning path that recommends learning materials for E-learning systems (Y. Chen, Li, et al., 2018; Tang et al., 2019) or intelligent tutoring systems (Brusilovsky & Peylo, 2003). These systems can then select personalized learning materials for learners and each individual’s total learning time to master skills is shortened.
Despite their strengths, there are two main challenges in the existing approaches. First, with the exception of Tang et al. (2019), aforementioned works typically characterize the learning path as a Markov decision process (MDP) assuming its transition probabilities are known. However, the transition probabilities are rarely known in practice. In fact, the transition paths of learners’ skills are unobservable and may vary across different learning materials. Second, skills are typically assumed to be without a hierarchical structure and mastery levels in previous learning tracking research when in reality a hierarchical structure is likely present. That is, mastery of higher level skills will depend on previous learning and mastery of other lower level skills. Ignoring skill hierarchy and mastery levels may contaminate classification results (Tu et al., 2019).
Tang et al. (2019) showed how to solve several example MDP problems using the model-free Q-learning method with the linear function approximation without assuming that the transition probabilities were known. However, their examples are relatively trivial without considering the intrinsic hierarchical structures among skills. Subject experts can find optimal learning policies directly based on learning materials and the skill hierarchy structure. The skill hierarchy structure can be found with empirical considerations or through curriculum development. The optimal learning policies in these examples can be found by subject experts directly based on contents of learning materials and the skill hierarchy structure that can be derived from empirical considerations or curriculum development (Leighton et al., 2004). Besides, it is well known in reinforcement learning (RL) communities that the Q-learning method with linear function approximation may fail to find an optimal policy in certain situations (Thrun & Schwartz, 1993).
In this article, the authors address the two aforementioned challenges by proposing an integrated E-learning system equipped with the optimal learning policy, an algorithm that designs the optimal learning paths for learners, without assuming that the transition probabilities are known and using a model-free method that accounts for the skill hierarchy. After finding the optimal learning policy and selecting the most appropriate learning materials for new learners, the system continuously trains the algorithm using the new learners’ information (data). In the rest of the article, the authors refer to a set of skills as attribute profile and a skill as an attribute which are conventional terms in CDMs. Specifically, the authors first develop a unified hierarchical learning model to explicitly characterize attribute hierarchy and mastery levels of attributes (known as ordered polytomous attributes in the CDM literature; J. Chen and de la Torre, 2013) which, while important, have not been addressed yet in existing E-learning system research. Attribute hierarchy widely exists in practice because many curricula are designed to follow a hierarchical structure. For example, mathematics contains many attributes which are related and often built on one another (Sternberg & Ben-Zeev, 1996). In addition, the optimal learning path cannot be easily designed by subject experts due to unobservable transition models of mastery levels across hierarchical attributes. Thus, it is crucial to consider hierarchical attributes and their mastery levels when building an E-learning system. The authors model the mastery levels related to hierarchical attributes following the same form of CDMs for binary attributes, instead of using polytomous attributes which can be accommodated by only a few CDMs (J. Chen & de la Torre, 2013). By doing this, the latent attributes and their mastery levels can be first pre-specified as binary labels by subject experts and later estimated using conventional CDMs. The optimal learning path is next formulated as an MDP, in which the state is the (discrete) attribute profile of a learner and the action is the (discrete) learning material selected for the learner. The proposed hierarchical learning model transforms the polytomous attributes to take a uniform form which is flexible and easy to implement in E-learning systems and can accommodate various types of attribute hierarchies (Leighton et al., 2004), the likes of which will be discussed in a later section. In addition, the number of latent states to be estimated, with respect to the restricted state space defined in the model, is largely reduced.
After developing the unified hierarchical learning model, the authors apply a model-free RL method to find the optimal learning policy. RL is a type of machine learning technique that takes suitable actions to achieve an objective (e.g., minimize total learning time) by interacting with the environment through trial-and-error search and collecting feedback. Using RL techniques, the proposed E-learning system is fully data-driven and does not require any prior information to solve the MDP. After each stage of learning, a set of items are distributed to the learners, whose responses to these items are collected by the E-learning system. Learners’ latent attributes are estimated and their attribute profile status is updated based on the responses using CDMs. The authors compare the model-free RL method with a heuristic method that makes a selection from available learning materials in accordance with model assumptions via simulation studies constructed under the proposed hierarchical learning model. The results indicate that the model-free RL method can quickly find a learning policy that outperforms the heuristic one.
The rest of the article is organized as follows. The CDMs and conventional hidden Markov models (HMM) used for modeling learning paths are introduced in the “Preliminaries” section. The hierarchical learning model and the model-free RL algorithm for finding the optimal learning policy are presented in the “Models and Algorithms” section. Results from simulation studies are presented in the “Simulation Studies” section. Some concluding remarks and potential future directions are discussed in the “Concluding Remarks and Future Directions” section.
Preliminaries
In this section, the authors provide some background on CDMs and the Markov model which will be used to characterize the learning paths.
CDMs
CDMs are psychometric models that examine learners’ mastery of specific attributes at a fine-grained level. These attributes are assumed to be latent and discrete. CDMs are ideal frameworks that aid in identifying optimal learning materials distributed to each learner, because of their capacity to keep track of learners’ different attributes while considering their multidimensional features. This is done by summarizing information in the form of attribute profiles. Each element of an attribute profile represents the learner’s mastery level related to an attribute. If only the mastery or non-mastery of an attribute is modeled, the elements take binary values. These latent attribute profiles can be reflected by the responses that examinees give to items, each of which measures one or more attributes.
Most CDMs require the construction of a Q-matrix for implementation. To be specific, suppose that the E-learning system considers attributes and contains items. The Q-matrix is a matrix whose element , , , on the row and column taking binary values, indicates whether the item is associated with the attribute. The Q-matrix specifies the cognitive specification for each test item explicitly.
An example is provided to illustrate the construction of a Q-matrix. Consider the mixed attributes in the system including addition and multiplication. For the item “5+4,” as only knowledge of addition is required to answer it correctly, this item only measures the addition attribute. Differently, the item “” measures both the addition and multiplication attributes. Thus, the corresponding row of the Q-matrix for the first item is and that for the second is . The Q-matrix provides a method to formulate the conditional independence between item responses and attribute profiles. That is, conditioning on measured attributes, item responses are independent of irrelevant attributes. The Q-matrix is generally specified before a test and further improved based on learners’ responses during the test (Y. Chen, Culpepper, Chen, & Douglas, 2018; Liu et al., 2012).
One widely used CDM is the deterministic inputs, noisy “and” gate (DINA) model (Junker & Sijtsma, 2001) which is both tractable and interpretable. In the DINA model, the probability of correctly answering an item is defined based on the Q-matrix. The authors refer the interested readers to the supplementary material for more details.
CDMs are classified as non-compensatory and compensatory models (DiBello et al., 2007). The DINA model is a non-compensatory model because it assumes the learner who lacks any of the required attributes will fail to answer the item correctly unless guessing. Unlike non-compensatory models, compensatory models allow a high ability attribute to compensate for a low ability attribute on another dimension. A more detailed review on these models can be found in our supplementary material. Both non-compensatory and compensatory models, along with the software programs that implement them, have been extensively researched with respect to modeling diagnostic attributes and the estimation of CDMs (e.g., expectation-maximization and Markov Chain Monte Carlo [MCMC] algorithms) (George et al., 2016; von Davier, 2006).
Learning Paths With the HMM
Learning paths can be modeled by an HMM because the attribute profile is latent (Wang et al., 2018). The Markov model specifies that a learner’s next state, after being provided with a certain learning material, will only depend on his or her current state and the material. Figure 1 illustrates how to model the learning path with an HMM. Define the attribute profile as the state and denote it by for a learner with attributes at time step . The state transition is as follows:
Figure 1.

An illustration of learning path with the Hidden Markov Model.
Note. denotes the latent attribute profile, denotes the learning material, and denotes the learner’s responses at time step.
| (1) |
where denotes the learning material distributed at time , and , which is the set of all learning materials. The transition process from the current state to the next is thus formulated as an MDP. In addition, represents the set of observable item responses of a learner at time step , where a value of 1 in indicates a correct answer while indicates an incorrect one.
The learning paths with latent attribute profiles can either be considered as a partially observable MDP (Kaelbling et al., 1998) or as two separate components—a psychometric model and an MDP. In both cases, the authors assume no retrogression exists—once learners master an attribute, they will not lose it, for example, for
| (2) |
| (3) |
In this study, the CDM and an HMM are used to estimate the attribute profiles. Specifically, given time-invariant item parameters and a proper psychometric model such as a CDM, the attribute profile of a learner at time step can be estimated from item responses at time step , as shown in Figure 1.
Models and Algorithms
In this section, a uniform and flexible hierarchical learning model that considers attribute hierarchy and mastery levels of attributes is constructed in the framework of CDMs. The problem of finding the optimal learning policy is next formulated as an MDP and solved by an RL algorithm, an efficient and stable algorithm for solving MDPs with unknown models.
Hierarchical Learning Model
In an E-learning system, learners are first given a test to assess what mastery levels they have reached on different attributes and next provided with learning materials based on their responses to improve mastery levels in as few learning steps as possible under the optimal learning policy. In the first step, a psychometric model such as a CDM is applied to estimate the latent attribute profiles. In the second step, a material is selected for learners by the optimal learning policy based on their estimated attribute profiles. To characterize the transition process of learners’ hierarchical attributes while considering mastery levels, the authors propose a unified and flexible hierarchical learning model that incorporates information about both the hierarchical structure and the mastery levels of attributes by representing attributes in an attribute profile as binary values. Thus, attribute profiles can be easily estimated using conventional CDMs. Furthermore, so that the proposed model conforms to a more realistic learning process, several assumptions are made with respect to the transitions between mastery levels of the hierarchical attributes. The following paragraphs discuss these assumptions along with the principles of attribute hierarchy upon which they are built.
The attribute hierarchy method (AHM) was first proposed to deal with situations where cognitive attributes are hierarchically related and thus correlated with each other (Leighton et al., 2004). In particular, the AHM investigates precedence ordering of cognitive competencies required to solve test problems. It has four different structures including linear, convergent, divergent, and unstructured. An intuitive example of the hierarchical structure is how learners learn addition “+” and multiplication “×.” Addition is considered as a prerequisite for multiplication. Learners are able to learn multiplication only after they are equipped with basic knowledge of it.
All structures investigated by AHM can be split into dependent relationships between two attributes. For example, Figure 2 exhibits the divergent structure among five attributes, denoted as , . The hierarchical structure among the five can be split into the four dependent links shown as dotted arrow lines in Figure 2. That is, is a prerequisite of and , while is a prerequisite of and . Therefore, the hierarchical learning model is proposed to capture the relationship between two dependent attributes and the complete hierarchical attribute structure can be completely defined using the relationship between paired attributes.
Figure 2.

A divergent hierarchical structure among five cognitive attributes.
First, assumptions on the link between two dependent attributes are made. Assume attribute is a prerequisite for attribute . There are different mastery levels for each attribute. Denote the lack of attribute as , , and different mastery levels as . A binary value is used to indicate whether or not a certain level of an attribute has been attained. Throughout this article, the following assumptions are made:
A1. Learners can only possess a high mastery level after they have mastered the lower level of the same attribute. That is,
| (4) |
A2. A certain level of attribute can only be learned after the same level of attribute is achieved. That is,
| (5) |
A3. The probability that a learner reaches a certain mastery level of the attribute , conditional on mastering a high level of attribute , is no smaller than mastering a lower level of attribute , . That is, for and ,
| (6) |
and for and ,
| (7) |
The authors next model different mastery levels of hierarchical attributes to be elements of attribute profiles as in CDMs. Assume that an attribute has different mastery levels, denoted as . If a learner reaches level for on this attribute, then , while for and for . The attribute profile with two attributes and , each with and mastery levels, respectively, is thus represented by . Note that is not related to , that is, they can be equal or different. An example of a Q-matrix for two hierarchical attributes with two mastery levels is provided in Table 1. The addition attribute (+) here is presumed to be a prerequisite of the multiplication attribute (×). One-digit calculation is assumed to be the low mastery level while two-digit calculation is assumed to be the high mastery level for both operations.
Table 1.
A Q-Matrix of Addition (+) and Multiplication (×) Attributes With Two Levels.
| Item | ||||
|---|---|---|---|---|
| 1 | 0 | 0 | 0 | |
| 1 | 1 | 1 | 0 | |
| 1 | 1 | 1 | 1 |
A deeper intuition for what these three assumptions mean can be gained using the same example. Under Assumption A1, a high level of any attribute, for example, , cannot be possessed until its lower level, , is mastered. Furthermore, under Assumption A2, if the learner has not reached a certain level on the prerequisite attribute, for example, , the higher level of × cannot be reached such that . Equation 6 in Assumption A3 indicates that the ability of multiplication is likely easier to learn when the ability of addition reaches a higher level, for example, , while Equation 7 in Assumption A3 indicates that the ability of addition is likely easier to learn when the ability of multiplication reaches a higher level, for example, .
To incorporate the attribute hierarchy, the state space is constructed following the hierarchical learning model assumptions. For standard CDMs, possible states are included in the HMM because this is the number of possible combinations of the four attributes. With the hierarchical learning model, the state space is reduced to six states, shown as rows in Table 2. As a result, the attribute profile of a learner at time step , , can only be one of the rows in Table 2.
Table 2.
State Space for Addition (+) and Multiplication (×) Attributes With Two Levels.
| State | ||||
|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 0 |
| 2 | 1 | 0 | 0 | 0 |
| 3 | 1 | 1 | 0 | 0 |
| 4 | 1 | 0 | 1 | 0 |
| 5 | 1 | 1 | 1 | 0 |
| 6 | 1 | 1 | 1 | 1 |
All attribute hierarchies can be generalized using the hierarchical learning model. In practice, more strict assumptions can be added if necessary. For example, an attribute cannot be learned before its prerequisite is fully mastered. In this case, the state space for the previous example would be further restricted by removing the fourth row of Table 2.
The design of the hierarchical learning model makes it possible to incorporate not only attribute hierarchy but also different mastery levels of attributes in CDMs. The model follows the common form of CDMs so that the restricted Q-matrix is easy to construct, and both the parameters and the attributes in CDMs can be estimated easily (Tu et al., 2019). In addition, the hierarchical design largely reduces the number of attributes to be estimated in CDMs.
MDP and RL
The authors first formulate the problem of finding the optimal learning policy as an MDP. The state space is the set of all attribute profiles while the attribute profiles are constructed under the proposed hierarchical learning model. The action space is defined to be the set of all learning materials . Denote the Markovian transition model as . As discussed earlier, the transition model satisfies the Markov property. Let and be two random variables that represent the state and action at time step , respectively. Then, the transition model (where represents the probability operator) is the probability of transitioning from state to state after taking action at time step . Finally, let be a reward function such that for and , is the reward obtained after transitioning from state to state after taking action . The goal of RL is to maximize the expected cumulative reward where a reward is a scalar feedback signal indicating how good the taken action is (Sutton & Barto, 2018). In this article, the optimal learning policy is a policy that selects, from among all available learning materials, the learning material for each individual learner that will minimize the number of total learning steps required to attain all attributes.
The overall framework of the algorithm is illustrated in Figure 3. The agent is the E-learning system that selects an action (i.e., learning materials) and sends the action to the environment (i.e., learners). The environment then sends the state (i.e., attribute profiles) along with a reward signal back to the agent. As both the state space and the action space are discrete, a classical model-free RL algorithm—the Q-learning algorithm—can be applied to learn the optimal learning policy (C. J. Watkins & Dayan, 1992). RL is widely used in solving problems by interacting with the environment, without requiring an explicitly expressed MDP model (Sutton & Barto, 2018). The RL method has several advantages when it comes to finding the optimal learning policy in the E-learning system. First, as it does not require an explicit model to estimate the utility of taking actions in the environment (Sutton & Barto, 2018), the RL method can be ideal for finding the best solution for an E-learning system, a system where it is unknown how learners’ attribute profiles change after learners are given learning materials. Second, the learning path with attribute hierarchy modeled by an HMM can be well solved using the RL method. Finally, the RL method searches for the long-term optimal solution which takes future rewards into consideration instead of simply choosing the best option at the immediate step (Sutton & Barto, 2018).
Figure 3.
RL system in the optimal learning policy problem.
Note. RL = reinforcement learning.
The Q-learning algorithm estimates an action-value function—the so-called Q-function—that is the expected cumulative discounted reward of a state-action pair. Let denote a deterministic learning policy for the MDP defined, that is, , , and . The action-value function under policy is defined as
| (8) |
where is the expectation operator, is a discount rate parameter, and is a random variable (whose value is denoted as ) which represents the reward obtained after taking action at state . Note that discounting is an additional concept in RL algorithms. The discount rate determines the present value of future rewards such that a reward received time steps in the future is worth only times what it would be worth if it were received immediately (see, for example, Sutton & Barto, 2018, for more details). Therefore, the agent (E-learning system) selects actions that will maximize the sum of the discounted rewards it receives over the future.
Denote the optimal action-value function as . Then, the greedy policy regarding is an optimal policy. Define , , and . The MDP is solved if is found, or equivalently, the maximum action-value function is found. Note that satisfies the Bellman optimality equation (see, for example, Sutton & Barto, 2018), which is defined as
| (9) |
This equation is the key to solving the MDP. As both the state and action sets are discrete, the action-value function can be represented in a tabular form covering all possible state-action pairs .
Algorithm 1: Q-learning Algorithm for Hierarchical Learning Model
Input: attribute profile (state) set , action set , learning rate , discount factor , decay rate for learning , decay rate for exploration , initial exploration probability , weights for the reward function , , ,
Output: Q function
Receive initial state and randomly initialize the value of
for do
Select with probability of and otherwise randomly
Select with probability of
Receive a new state
Calculate as the number of mastered attributes at time step for the learner
Compute reward according to
Compute
Update learning rate and exploration rate
end
The Q-learning algorithm approximates by iteratively updating the action-value function , where is a learning rate that decays over iterations with a decay rate of (see, for example, C. J. C. H. Watkins, 1989, for more details). The detailed algorithm for the optimal learning policy problem is presented in Algorithm 1. The -greedy exploration policy is adopted where the probability of exploration at the beginning of the algorithm is and this probability decays over time with a decay rate of , allowing for more exploitation to occur. In addition, the reward in Algorithm 1 is
| (10) |
where is the number of mastered attributes in the attribute profile at time step , and , , , are positive real numbers that reflect the relative importance of two objectives–increasing the number of mastered attributes and decreasing the length of the learning episode. As our objective is to minimize the episode length required for each learner to master all attributes, the reward decreases while the length of the learning episode (the entire learning process of a learner from beginning to mastery) increases, and the reward increases while the number of mastered attributes increases. Intuitively, the design of the reward function is expected to incentivize the discovery of learning policies that can help learners to master more attributes as fast as possible. It is important to note that can be smaller than . This is because is not available and has to be estimated using CDMs which results in estimation errors for and . The Q-learning algorithm has been proven to converge, that is, the values converge to with probability 1 if the learning rate is properly chosen and the state-action space is sufficiently explored (C. J. Watkins & Dayan, 1992).
Simulation Studies
In this section, the authors apply the Q-learning algorithm to find the optimal learning policy with attribute profiles modeled by the hierarchical learning model under two separate conditions, with or without the presence of measurement errors. The authors also investigate the impacts of various initial states.
Overview
The purpose of the simulation studies is to explore how the RL method performs in finding the optimal learning policy with the transition process of hierarchical attributes built into the framework of the hierarchical learning model. The optimal learning policy found by the RL method is compared with a heuristic method. As measurement errors always exist in the estimated attribute profiles using CDMs, it is important to evaluate the impact of estimation errors on the optimal learning policy found by the RL method. Representations of the estimated attribute profiles are created by adding different magnitudes of measurement errors to the true attribute profiles. In addition, because of the fact that learners should be given different optimal learning paths according to their initial attribute profiles, whether the system can find the optimal learning policy for learners with different initial attribute profiles is examined.
As explained earlier, an attribute profile hierarchical structure can be represented by its paired attributes. In this simulation study, the authors consider two attributes with a linear hierarchical structure. Denote the two attributes as and . Assume each attribute has three mastery levels denoted as , , , , , and . or is used when the corresponding attribute is not mastered. Assume is a prerequisite attribute for , and that these two attributes satisfy all of the assumptions outlined earlier. An intuitive way to understand the hierarchical structure here is to assume that is the addition attribute and that is the multiplication attribute. The three mastery levels can be translated as beginner, intermediate, and advanced, while and indicate a learner has no knowledge of and , respectively.
Assume that six learning materials are available. Three of these learning materials are for the beginner, intermediate, and advanced levels of attribute , denoted as , , , while the other three attributes are for the three different mastery levels of attribute , denoted as , , . The authors thus construct the MDP shown as a directed graph in Figure 4. Each circle represents a state. A filled arrow shows a transition of attribute while a dotted arrow shows a transition of attribute . Only one attribute can be improved in each learning step. If the learner acquires the attribute profile of , no more learning material will be provided and the learning process ends. The process satisfies the three assumptions in the hierarchy learning model. Note that the transition from a state to itself is neglected in the directed graph and can be easily calculated by Markov properties. The transition matrix, which is unknown to the environment and only applied to predict learners’ next states, is constructed accordingly. The corresponding state space is shown in Table 3.
Figure 4.

Markov process for the attribute profile with attribute and .
Table 3.
State Space for Two Attributes With Three Levels.
| State | ||||||
|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | 1 | 0 | 0 | 0 | 0 | 0 |
| 3 | 1 | 1 | 0 | 0 | 0 | 0 |
| 4 | 1 | 1 | 1 | 0 | 0 | 0 |
| 5 | 1 | 0 | 0 | 1 | 0 | 0 |
| 6 | 1 | 1 | 0 | 1 | 0 | 0 |
| 7 | 1 | 1 | 1 | 1 | 0 | 0 |
| 8 | 1 | 1 | 0 | 1 | 1 | 0 |
| 9 | 1 | 1 | 1 | 1 | 1 | 0 |
| 10 | 1 | 1 | 1 | 1 | 1 | 1 |
Figure 4 reveals the difference between the policy that only considers immediate reward and the policy given by the RL method that takes future rewards into account. For instance, suppose a learner reaches the beginner level of the first attribute and has no knowledge of the second attribute , that is, the learner is in state . At this step, the beginner level material for attribute , that is denoted as , gives the shorter expected learning time, which is defined as which is .1 However, although the intermediate level material for attribute , denoted as , brings a relatively longer learning time, which in turn leads to a smaller reward at the current step, the overall expected learning time of the path that goes through to , which is , is less than that of the path that goes through to , which is . Thus, it can be seen that although learning the beginner level of attribute first is quicker at the current step, it is not the long-term optimal learning path.
As the estimated attribute profile is used in place of the true state in practice, an estimation error—the probability that the estimated attribute profile is incorrect—of was added to the state, that is, the estimated attribute profile may be any other attribute profile except the true one with a probability of . In CDM research, the average pattern correct classification rate (PCCR) is usually larger than . Therefore, an estimation error of is large enough to show the reliability of the optimal learning policy. In addition, simulation results for cases with an estimation error ranging from to are included to show that the Q-learning algorithm is reliable and stable and can still find the optimal learning policy even with the presence of an estimation error. In practice, the attribute profiles are estimated and updated based on learners’ responses to test items.
The rest of parameters in the algorithm are set as follows. The initial learning rate is and the discount factor is , both of which are values widely used in machine learning algorithms. The initial value of the exploration probability is , that is, the algorithm will choose an action randomly in the beginning. To stabilize the learning algorithm, the learning rate and the exploration probability need to be decreased gradually. Therefore, a decay rate of is applied for the learning rate and a decay rate of is used for the exploration probability . After episodes (where a single episode represents the entire learning process for an individual learner), the learning rate decays to a value of and the exploration probability decays to . Weights for the reward function are , , . These weights indicate that it is more important to increase the number of attributes that have been mastered than to decrease the learning episode length.
The Q-learning algorithm is trained for episodes to acquire a stable policy. After that, the trained model is applied for another episodes (e.g., new learners) and compared with a heuristic learning policy, which is the policy that selects the next learning materials in a manner that respects the hierarchical learning model assumptions while also improving the learner’s mastery levels of both attributes. For instance, if the learner’s attribute profile is estimated to be , the next learning material will be selected from the beginner level learning materials for attribute and the intermediate level learning materials for attribute .
Recall that the objective of the optimal learning policy is to minimize the total number of learning steps and the reward in the Q-learning algorithm is designed to increase when the total number of learning steps decreases. The evaluation criterion used to compare the two learning methods is the episode reward. In other words, the method that finds the learning policy with a higher episode reward is considered to be the superior method. The standard deviation (SD) of rewards is also calculated and compared between the two methods as a smaller SD indicates a more stable optimal learning policy. The number of episodes that the RL method takes to find the optimal learning policy is also examined. A short episode length indicates that the method can find the optimal learning policy quickly. Two simulation studies are conducted in this article. In the first simulation study, the initial states for all learners are , which means none of the learners have any knowledge of the two attributes. In the second simulation study, learners can start at any mastery level except for . The second simulation study shows that as long as a learner has not fully mastered attributes specified in the E-learning system, then no matter which level the learner begins with, the system can find the optimal learning policy for the learner.
Results
Learning Policy Comparison
Figures 5 and 6 present the rewards of the first episodes when the RL method is trained and includes the immediate reward along with a smoothed reward (obtained with a smoothing window of 20). Figure 5 shows that the reward becomes stable after episodes under the RL method without estimation error, which means the method finds the optimal learning policy after self-training on learners. This result indicates that the RL method finds the optimal learning policy quickly. Figure 6 shows that even after adding a estimation error to the system, the RL method still finds the optimal learning policy after around episodes.
Figure 5.

Rewards under optimal learning policy without estimation error.
Note. RL = reinforcement learning.
Figure 6.

Rewards under optimal learning policy with estimation error.
Note. RL = reinforcement learning.
Table 4 shows the mean and SD of rewards and episode lengths for the two methods applied in new episodes. The RL method has a much higher mean and a lower SD than the heuristic method. It also has shorter episode lengths and a smaller SD of episode lengths. It is worth noting that although the average episode length with a estimation error is slightly higher than the average episode length without an estimation error, the difference is trivial.
Table 4.
Mean and Standard Deviation (SD) of Rewards and Episode Lengths (EL).
| Estimation error | ||||
|---|---|---|---|---|
| Methods | RL | Heuristic | RL | Heuristic |
| Reward mean | 6.43 | 3.99 | 6.41 | 3.98 |
| Reward SD | 3.61 | 5.34 | 3.60 | 5.37 |
| EL mean | 7.34 | 8.57 | 7.73 | 9.01 |
| EL SD | 1.90 | 2.62 | 2.07 | 2.74 |
Note. RL = reinforcement learning.
Figure 7 gives a comparison between the RL method and the heuristic method across episodes under different estimation errors and no estimation error using a box plot. The figure shows that the average reward under the RL method is much higher than the average reward under the heuristic method across the conditions. In addition, the RL method also produces a smaller SD of rewards than the heuristic method. Although the SD of the RL method tends to increase when the estimation error increases, it is still smaller than that of the heuristic method.
Figure 7.

Comparison of rewards under optimal learning policy learned via RL and heuristic learning policy with estimation errors.
Note. RL = reinforcement learning.
Simulation results indicate that the RL method finds a better learning policy than the heuristic method. More importantly, the estimation error has negligible impact on the performance of the RL method in searching for the optimal learning policy.
Concluding Remarks and Future Directions
In this article, the authors proposed a hierarchical learning model that incorporates attribute hierarchy and mastery levels of attributes together in an E-learning system. The model follows the same form of discrete attributes and Q-matrix that are required by CDMs so that parameters and hidden states can be easily recovered and estimated. In addition, the transition process for the learning path, given a learning material, is formulated as an MDP. Finally, a model-free RL method is applied to find the optimal learning policy while incorporating the hierarchical framework.
Simulation results suggest that the optimal design with the RL method substantially outperforms the heuristic learning policy with and without an estimation error. The mean and the SD of the learning episode length from the RL method are significantly smaller than those obtained by the heuristic method. Moreover, the RL method can find the optimal learning policy quickly for all learners even when learners have different initial attribute profiles. Thus, learners with various levels of attributes will be assigned the most appropriate materials at each step. In practice, a set of items needs to be given to learners after they finish each learning material and then their attributes are estimated and updated based on their responses to the given items.
Several directions are possible for future research. First, dimensionality assessment methods can be applied to classify learners at the first stage (J. Zhang, 2013; J. Zhang & Stout, 1999), in addition to using estimation methods to get learners’ initial states. It is important to have an accurate estimate of learners’ initial states so that the most appropriate optimal learning policy can be assigned to each individual as early as possible. Second, different algorithms can be proposed to select the personalized learning materials that can maximize learners’ immediate or future rewards (Manickam et al., 2017). Finally, CDMs with a restricted hierarchical state space and Q-matrix can be further explored (Tu et al., 2019). The identifiability conditions for the restricted latent structure model should also be further studied (Xu, 2017).
Supplemental Material
Supplemental material, supplement for Optimal Hierarchical Learning Path Design With Reinforcement Learning by Xiao Li, Hanchen Xu, Jinming Zhang and Hua-hua Chang in Applied Psychological Measurement
Expected value for a geometric distribution.
Footnotes
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
ORCID iD: Xiao Li
https://orcid.org/0000-0003-0214-0130
Supplemental Material: Supplementary material is available for this article online.
References
- Brusilovsky P., Peylo C. (2003). Adaptive and intelligent web-based educational systems. International Journal of Artificial Intelligence in Education, 13, 159–172. [Google Scholar]
- Chen J., de la Torre J. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37(6), 419–437. [Google Scholar]
- Chen Y., Culpepper S. A., Chen Y., Douglas J. (2018). Bayesian estimation of the dina q matrix. Psychometrika, 83(1), 89–108. [DOI] [PubMed] [Google Scholar]
- Chen Y., Culpepper S. A., Wang S., Douglas J. (2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chen Y., Li X., Liu J., Ying Z. (2018). Recommendation system for adaptive learning. Applied Psychological Measurement, 42(1), 24–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Corbett A. T., Anderson J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. [Google Scholar]
- DiBello L., Roussos L., Stout W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao C. R., Sinharay S. (Eds.), Handbook of statistics, vol. 26: Psychometrics (pp. 970–1030). North-Holland Publications. [Google Scholar]
- George A. C., Robitzsch A., Kiefer T., Groß J., Ünlü A. (2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1–24. [Google Scholar]
- Junker B. W., Sijtsma K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. [Google Scholar]
- Kaelbling L. P., Littman M. L., Cassandra A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2), 99–134. [Google Scholar]
- Leighton J. P., Gierl M. J., Hunka S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205–237. [Google Scholar]
- Liu J., Xu G., Ying Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Manickam I., Lan A. S., Baraniuk R. G. (2017, June). Contextual multi-armed bandit algorithms for personalized learning action selection. In Proc. IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing, New Orleans, LA, pp. 6344–6348. [Google Scholar]
- Means B., Toyama Y., Murphy R., Bakia M., Jones K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. https://eric.ed.gov/?id=ED505824
- Sternberg R. J., Ben-Zeev T. (1996). The nature of mathematical thinking. Routledge. [Google Scholar]
- Sutton R. S., Barto A. G. (2018). Reinforcement learning: An introduction. MIT Press. [Google Scholar]
- Tang X., Chen Y., Li X., Liu J., Ying Z. (2019). A reinforcement learning approach to personalized learning recommendation systems. British Journal of Mathematical and Statistical Psychology, 72(1), 108–135. [DOI] [PubMed] [Google Scholar]
- Thrun S., Schwartz A. (1993). Issues in using function approximation for reinforcement learning. In Proc. Connectionist Models Summer, Hillsdale, NJ, USA, pp. 1–9. [Google Scholar]
- Tu D., Wang S., Cai Y., Douglas J., Chang H.-H. (2019). Cognitive diagnostic models with attribute hierarchies: Model estimation with a restricted Q-matrix design. Applied Psychological Measurement, 43, 255–271. [DOI] [PMC free article] [PubMed] [Google Scholar]
- von Davier M. (2006). Multidimensional Latent Trait Modelling (MDLTM) [Software program]. Educational Testing Service. [Google Scholar]
- Wang S., Yang Y., Culpepper S. A., Douglas J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87. [Google Scholar]
- Watkins C. J., Dayan P. (1992). Q-learning. Machine Learning, 8(3–4), 279–292. [Google Scholar]
- Watkins C. J. C. H. (1989). Learning from delayed rewards (Unpublished doctoral dissertation). King’s College. [Google Scholar]
- Xu G. (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2), 675–707. [Google Scholar]
- Zhang J. (2013). A procedure for dimensionality analyses of response data from various test designs. Psychometrika, 78(1), 37–58. [DOI] [PubMed] [Google Scholar]
- Zhang S., Chang H.-H. (2016). From smart testing to smart learning: How testing technology can assist the new generation of education. International Journal of Smart Technology and Learning, 1(1), 67–92. [Google Scholar]
- Zhang J., Stout W. (1999). The theoretical detect index of dimensionality and its application to approximate simple structure. Psychometrika, 64(2), 213–249. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental material, supplement for Optimal Hierarchical Learning Path Design With Reinforcement Learning by Xiao Li, Hanchen Xu, Jinming Zhang and Hua-hua Chang in Applied Psychological Measurement

