Model Learning and Knowledge Sharing for Cooperative Multiagent Systems in Stochastic Environment

Abstract

An imposing task for a reinforcement learning agent in an uncertain environment is to expeditiously learn a policy or a sequence of actions, with which it can achieve the desired goal. In this article, we present an incremental model learning scheme to reconstruct the model of a stochastic environment. In the proposed learning scheme, we introduce a clustering algorithm to assimilate the model information and estimate the probability for each state transition. In addition, utilizing the reconstructed model, we present an experience replay strategy to create virtual interactive experiences by incorporating a balance between exploration and exploitation, which greatly accelerates learning and enables planning. Furthermore, we extend the proposed learning scheme for a multiagent framework to decrease the effort required for exploration and to reduce the learning time in a large environment. In this multiagent framework, we introduce a knowledge-sharing algorithm to share the reconstructed model information among the different agents, as needed, and develop a computationally efficient knowledge fusing mechanism to fuse the knowledge acquired using the agents’ own experience with the knowledge received from its teammates. Finally, the simulation results with comparative analysis are provided to demonstrate the efficacy of the proposed methods in the complex learning tasks.

I. Introduction

Reinforcement learning (RL) is a popular machine-learning tool employed to tackle complex decision-making problems in uncertain environments. Inspired by naturalistic decision-making processes in biological systems, the RL methodology enables an agent to progressively learn a sequence of actions to achieve a desired objective [1]. The environment, wherein the agent performs an action, provides a scalar feedback, called the reinforcement/reward signal, and as a consequence of the action, the agent transits from one state (current state) to a new state. During this process of interaction with the environment, the agent explores the environment and accumulates experiences or knowledge regarding the environment. Using the collected experiences, over time, the agent learns the best sequence of action from any situation/state [2], [3] to achieve a desired objective.

Typically, these experiences are collected when the agent is exploring the environment, that is, the agent actually takes actions in an uncertain environment hoping that these actions do not jeopardize the agent and receives reinforcement signals and collects experiences. Therefore, the process of collecting experiences in an uncertain environment is expensive and time consuming, and it is imperative that the collected experiences are efficiently utilized [4]–[7]. For example, a rover exploring an uncertain terrain may crash onto an obstacle or plunge into a crevasse. In this context, sample efficiency in RL is a challenging and important issue, that is, extracting the model/policy information efficiently from the collected experience becomes an important objective.

The RL approaches are broadly classified as model-free and model-based learning strategies. In the model-free RL, the collected experiences are used to learn a policy based on the expected value of applying an action at each step. These experiences are discarded once the learning is complete. On the other hand, the model-based RL approach is introduced to achieve sample efficiency [8]–[10]. In the model-based RL approach, the agent strives to reconstruct a virtual model of the environment which not only includes the reward function but also the transition function, the function that encodes how an action from the current state is mapped to the next state.

The Dyna architecture is a well-known model-based RL approach [11], [12] in which an agent reconstructs a model of the environment from actual experience (through direct learning), and after the learning process, when the model is fed with the current state and an action, it will predict the next state and the expected reward. This allows for the agent to create virtual experiences that can be used to update the policy and incorporate planning efficiently through indirect learning. Indirect learning also enables the agent to plan and achieves a better balance between exploration and exploitation [13]. Several algorithms that improve/accelerate the learning process in the model-based RL framework, have been proposed in [14]–[16].

An alternate approach to achieve sample efficiency in the learning process is to employ experience replay strategy [17]–[20]. In both the Dyna architecture and the experience replay strategy, all the collected experiences are stored. However, in some applications, the memory space to store the experiences may be limited or costly. Therefore, in the experience replay strategy, a mechanism to omit experiences during the learning process is introduced. Different algorithms incorporating SARSA, Q-learning, etc., are studied along with the experience replay strategies to improve the learning efficiency [21]. The characteristic difference of the traditional Q-learning or SARSA learning with the experience replay-based strategies is the additional mechanism which discards experiences when the memory is full. This mechanism works either based on time (discarding oldest experience first) or based on the reward (discarding experiences with least reward) [20].

It should be noted that the RL algorithms, for example, Q-learning, propagate the Q-values from the terminal state to the initial state during the learning process. As a result, experiences collected initially, that are near the initial states, may be useful, and when some experiences are omitted based on time or reward, these useful experiences may be discarded. This renders the learning process inefficient. Similarly, in the Dyna architecture, the agent, during indirect learning, always chooses an action from a state that is visited previously to predict the next state and the corresponding reward. In some cases, the environment (state space) is too large for an agent to cover, slowing down this learning process.

Therefore, when a single agent deals with a large environment, the learning time will increase. In such a case, a suitable learning scheme, incorporating multiple learning agents that cooperate and share their experiences collected independently with the other agents [22]–[25], will accelerate the learning process. The multiagent learning framework can accommodate a variety of decision-making tasks [26], for instance, tasks where multiple agents cooperate or compete to maximize a coupled objective (e.g., Half-field offence [27]). However, in this article, we will adopt the multiagent learning framework to accelerate the learning process by using a team of RL agents to learn a large environment. Here, each agent will take actions that are independent of the other agents and share their knowledge with the other agents when there is a request for information. Such cooperative knowledge-sharing methods were reported in [22], [23], and [28]–[32], and the references therein. For instance, the agents explored the environment independently to update their Q-functions and then shared the Q-values with their teammates as needed in the results presented in [28]–[30]. In these approaches, each agent’s policy is shared with the others [28]–[32]. Alternatively, in a model sharing approach, each agent builds its own model of the environment during the learning period and exchanges its model information with the other agents [33]–[37].

In the cooperative knowledge-sharing approach, each agents’ policy/model is shared with the others, and an important task for an agent is to effectively combine the newly obtained knowledge from the other agents with the knowledge gained using its own experience [28]–[32]. This depends on how the information is stored in each agent. In [32], the heterogeneous tree structures were proposed, for each agent, for state aggregation, where the Q-values were stored in different tree nodes. A model-learning scheme using heterogeneous tree structures along with a request-based knowledge-sharing framework was proposed by Hwang et al. [37]. In contrast to the algorithm in [38], where the entire tree structure was shared with the other agents, in [37], agents only shared their leaf nodes to the others to reduce the merging cost. However, due to the heterogeneous tree structure, a resampling method was introduced to create virtual experiences for tractable fusing of the knowledge shared between the cooperative agents.

In summary, sample efficiency and reduction in the learning time in a large, uncertain environment could potentially be resolved if an algorithm can: 1) efficiently reconstruct a virtual model that can store more information, especially, in a stochastic state space; 2) generate useful experiences from the virtual model for indirect learning, without discarding potentially valuable experiences; and 3) effectively share knowledge in the multiagent setting and combine the shared knowledge without increasing the computational burden on each agent.

Therefore, in this article, we propose a multiagent learning scheme, wherein each agent collects experiences for policy and model learning in an uncertain, stochastic environment. To learn the environmental model information from the experience collected by each agent, we introduce an incremental model learning algorithm, and to effectively capture the state transition, we propose a clustering scheme. In the proposed clustering-based model learning approach, clusters are formed to store the information regarding the probability for each state transition. As opposed to the traditional “look-up” table-based storage of the environmental models or the tree structure, the clusters offer a concise way to store the model information.

Furthermore, we present an experience replay strategy along with the Q-learning scheme to achieve sample efficiency, by incorporating a balance between exploration and exploitation. The experience replay strategy is used to create virtual experiences corresponding to states which may or may not be experienced by the agents, for updating the Q-function. Furthermore, when an agent in a cooperative team lacks sufficient knowledge of a specific state, a request will prompt its teammates to share their model information. After obtaining the knowledge, an efficient fusing mechanism (using the clusters and T-statistics) is proposed to help each agent fuse its own knowledge and the knowledge shared by its teammates. The efficiency of the proposed algorithms is validated with three numerical simulations, and the comparison with related state-of-the-art methods demonstrates the advantages of the proposed scheme.

The contributions of this article include developing: 1) an incremental model learning algorithm to establish a virtual model of a stochastic environment; 2) a clustering scheme to assimilate the experiences in a virtual model to evaluate the probability of the state transitions; 3) an experience replay strategy with incremental model learning schemes to achieve sampling efficiency via multistep Q-learning; and 4) a knowledge fusing scheme for the multiagent framework to fuse the model knowledge shared by the teammates.

The remainder of this article is organized as follows. In Section II, a brief background on RL is presented and the problem considered in this article is detailed. The proposed incremental model learning method along with the clustering scheme are presented in Section III. Section IV introduces the knowledge-sharing algorithm which is employed in tandem with the incremental model learning method in a multiagent setting. In Section V, the simulation results for the proposed method are presented. Finally, in Section VI, we present our conclusions.

II. Background

In this section, we first provide a brief background on the RL problem and then state the design objectives considered in this article. The RL problems can be formulated as Markov decision processes (MDPs) which consist of a state space S_p, an action space A_p, a state-transition function T, and a reward function R [1]. During the learning process, an RL agent perceives the environmental information, denoted as a state s_t ∈ S_p and chooses an action a_t ∈ A_p so that the agent transits from the current state to the next state s_t+1 based on T. The action a_t is, in general, based on a policy, and in this process, the agent receives an external scalar reward/reinforcement signal r_t+1 ∈ R from the environment. The tuple (s_t, a_t, s_t+1, r_t+1) will be collected as an experience by the RL agent and the aggregated experiences will be used to update the policy, which will influence the agent’s future actions. Specifically, the RL agent uses the experiences to update the long-term expected reward that is stored as the value function or the Q-function.

In the RL tasks, the model of the environment can be deterministic or stochastic [1], [39]. In the deterministic environment, the transition function T : S_p×A_p → S_p maps a state and an action to a new next state in the state space, while in the stochastic environment, the transition function maps the state and the action to the next state with a probability for each state transition defined by a distribution. Therefore, in a stochastic environment, the RL agent must encode the probability distribution using the collected experiences in the learnt model. In this article, we consider an MDP with a continuous state space $X_p\subseteq {ℝ}^n$, a discrete action space A_p, a stochastic state-transition function T_p, and a reward function R_p. In addition, we define a map s_t ← ϕ(x_t), that maps the continuous states x_t to a discrete state $s_t\in S_p\subseteq {ℝ}^n$.

Furthermore, learning with multiple RL agents enables solving complex tasks in a large state space [26] and can include a wide variety of RL problems (e.g., Nash games). In this article, we focus on developing an efficient model learning and knowledge-sharing mechanism in a large stochastic environment by using multiple agents. Specifically, we let the agents learn a model of the environment, and a policy to reach a goal, while also helping their teammates by sharing their model and policy when requested.

The objectives considered in this article are: 1) to develop a model learning scheme with efficient storage structure to facilitate indirect learning; 2) to design an algorithm for incorporating indirect learning/planning for the RL agent to efficiently utilize the virtual model; 3) to design a multiagent knowledge-sharing framework; and 4) to develop a knowledge fusing algorithm without imparting computational burden (by avoiding a resampling scheme) on each agent.

III. Model Learning for Experience Replay With Q-Learning

In this section, an incremental model learning scheme for experience replay is presented. Since the major focus of this article is not Q-learning, for completeness, the details of the policy learning scheme are briefly introduced first.

A. Policy Learning via Direct Learning

At time t, an agent perceives a state x_t = [x₁, x₂, …, x_n], where n is the dimension of the state space. The perceived continuous state is encoded to a discretized state s_t = [s₁, s₂, …, s_n] ∈ S_p using the map ϕ. An -greedy method is used by the agent to choose an action a_t ∈ A_p in the current state s_t [1]. Following the state transition, the agent will perceive the next continuous state x_t+1, which is encoded to the next discretized state s_t+1, and a reward r_t+1 is returned by the environment. In the model-free Q-learning approach, the agent utilizes the experience (s_t, a_t, s_t+1, r_t+1) to update the Q-values denoted as Q(s_t, a_t). The Q-function [40] is given by

$$\begin{gathered} Q{\left(s_t,a_t\right)}_{\text{new}}=(1-\beta )Q{\left(s_t,a_t\right)}_{\text{old}} \\ +\beta \left(r_{t+1}+\gamma \underset{a^{\prime }\in A_p}{\max }Q\left(s_{t+1},a^{\prime }\right)\right) \end{gathered}$$ (1)

where 0 < β < 1 is the learning rate and 0 < γ < 1 is the discount factor. If the agent acquires a desirable reward, the Q-values will be increased for the corresponding state–action pair, while the Q-values will be decreased, otherwise. With the Q functions updated iteratively, the Q-values for all state–action pairs will converge to the optimal Q* values asymptotically with probability one [41], [42]. The update error is defined as

$$\mathrm{\Delta }Q\left(s_t,a_t\right)=Q{\left(s_t,a_t\right)}_{\text{new}}-Q{\left(s_t,a_t\right)}_{\text{old}}.$$ (2)

In our proposed algorithm, the agent learns the optimal policy via both the direct learning and indirect learning, where Q(s, a) stores the Q-values for all the state–action pairs, ΔQ(s, a) stores all the updating error, and M(s, a) stores the state-transition and reward functions. Next, we present our incremental model learning scheme for encoding the model information in M(s, a) using the collected experiences.

B. Incremental Model Learning via Direct Learning

In addition to learning the Q-function, the experiences (x_t, a_t, x_t+1, r_t+1) are utilized for learning the model of the environment. First, the model learning scheme is presented for the deterministic environment and, then, for the stochastic environment.

1). Deterministic Environment

For a deterministic environment, if the agent takes an action a₁ from the continuous state x₁, it will transit to the next continuous state x₂ with probability 1.

In this article, we propose a model that is composed of two functions, a variation transition function $T_v:S_p\times A_p\to \mathrm{\Delta }{X}_p\subseteq {ℝ}^n$, which maps a state and an action to a variation of state (or the first difference), and a reward function $R_v:S_p\times A_p\times S_p\to ℝ$, which maps a state and an action to a scalar reward. For instance, the Nth state transition due to an action a results in the variation of the continuous state $\mathrm{\Delta }{x}_N^a$ and the reward $r_N^a$ that is computed as

$$\mathrm{\Delta }{x}_N^a=x_{t+1}-x_t,r_N^a=r_{t+1}.$$ (3)

The variation of the continuous state and the reward is stored in a model, in cells, wherein each cell stores the variation of the continuous state $\mathrm{\Delta }{x}_1^a$, the square of the variation of the continuous state ${\left(\mathrm{\Delta }{x}_1^a\right)}^{\prime }$, the reward $r_1^a$, and the square of the reward ${\left(r_1^a\right)}^{\prime }$. If the agent visits a state more than once, say (N − 1) times, the variations and rewards are averaged and stored. These quantities are calculated as

$$\begin{gathered} \mathrm{\Delta }{\overline{x}}_N^a=\frac{\mathrm{\Delta }{x}_1^a+\mathrm{\Delta }{x}_2^a+\cdots +\mathrm{\Delta }{x}_{N-1}^a}{N-1} \\ {\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^{\prime }=\frac{{\left(\mathrm{\Delta }{x}_1^a\right)}^2+{\left(\mathrm{\Delta }{x}_2^a\right)}^2+\cdots +{\left(\mathrm{\Delta }{x}_{N-1}^a\right)}^2}{N-1} \\ {\overline{r}}_N^a=\frac{r_1^a+r_2^a+\cdots +r_{N-1}^a}{N-1} \\ {\left({\overline{r}}_N^a\right)}^{\prime }=\frac{{\left(r_1^a\right)}^2+{\left(r_2^a\right)}^2+\cdots +{\left(r_{N-1}^a\right)}^2}{N-1} \end{gathered}$$ (4)

where $\mathrm{\Delta }{\overline{x}}_N^a$ is the averaged variation of the continuous state, ${\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^{\prime }$ is the average of squared variation of the continuous state, ${\overline{r}}_N^a$ is the average of the reward, and ${\left({\overline{r}}_N^a\right)}^{\prime }$ is the average of squared reward.

Hence, at the Nth visit to this state, the new $\mathrm{\Delta }{\overline{x}}_{N+1}^a$ and ${\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a\right)}^{\prime }$ _{are updated according to}

$$\begin{gathered} \mathrm{\Delta }{\overline{x}}_{N+1}^a=\frac{1}{N}\sum _{i=1}^N\mathrm{\Delta }{x}_i^a=\frac{1}{N}\left(\mathrm{\Delta }{x}_N^a+\sum _{i=1}^{N-1}\mathrm{\Delta }{x}_i^a\right) \\ =\frac{1}{N}\left(\mathrm{\Delta }{x}_N^a+(N-1)\frac{1}{(N-1)}\sum _{i=1}^{N-1}\mathrm{\Delta }{x}_i^a\right) \\ =\mathrm{\Delta }{\overline{x}}_N^a+\frac{1}{N}\left(\mathrm{\Delta }{x}_N^a-\mathrm{\Delta }{\overline{x}}_N^a\right) \end{gathered}$$ (5)

$${\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a\right)}^{\prime }={\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^2+\frac{1}{N}\left({\left(\mathrm{\Delta }{x}_N^a\right)}^2-{\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^2\right).$$ (6)

Since it is infeasible to know a priori if a state will be visited by the agent more than once, (4) is not implementable and the incremental update in (6) can be used.

Analogously, ${\overline{r}}_{N+1}^a$ and the new ${\left({\overline{r}}_{N+1}^a\right)}^{\prime }$ are updated by

$${\overline{r}}_{N+1}^a={\overline{r}}_N^a+\frac{1}{N}\left(r_N^a-{\overline{r}}_N^a\right)$$ (7)

$${\left({\overline{r}}_{N+1}^a\right)}^{\prime }={\left({\overline{r}}_N^a\right)}^2+\frac{1}{N}\left({\left(r_N^a\right)}^2-{\left({\overline{r}}_N^a\right)}^2\right).$$ (8)

Algorithm 1.

Incremental Model Learning for Deterministic Environment (ModelLearning)

(1)	Calculate the variation of the continuous state and the reward using equation (3)
(2)	s ← ϕ(xt)
(3)	Retrieve $\mathrm{\Delta }{\overline{x}}_N^a$, ${\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^{\prime }$, ${\overline{r}}_N^a$, and ${\left({\overline{r}}_N^a\right)}^{\prime }$, from the model using state- action pair, (s, a)
(4)	Use (5)–(8) to calculate the new $\mathrm{\Delta }{\overline{x}}_{N+1}^a$, ${\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a\right)}^{\prime }$, ${\overline{r}}_{N+1}^a$ and ${\left({\overline{r}}_{N+1}^a\right)}^{\prime }$
(5)	Update $T_v\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a|s,a\right)$ and Rv(s′, s, a) using (9)
(6)	Store these two functions in the model, $$M(s,a)\leftarrow \left(T_v\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a|s,a\right),R_v\left(s^{\prime },s,a\right)\right)$$
(7)	Return M(s, a).

Therefore, the variation transition function and the reward function are handled by the proposed model that is defined as

$$\begin{gathered} T_v\left(\mathrm{\Delta }{\overline{x}}_{N+1}^a|s,a\right)=\left\{\mathrm{\Delta }{\overline{x}}_{N+1}^a|s\leftarrow \varphi \left(x_t\right),a\leftarrow a_t\right\} \\ {R}_v\left(s^{\prime },s,a\right)=\left\{{\overline{r}}_{n+1}^a|\left(s^{\prime },s\right)\leftarrow \varphi \left(x_{t+1},x_t\right),a\leftarrow a_t\right\} \end{gathered}$$ (9)

where T_v maps the discretized state s and the action a_t to an average variation of the continuous states $\mathrm{\Delta }{\overline{x}}_{N+1}^a\in \mathrm{\Delta }{X}_p$, and the reward function R_v maps s′ and s, and the action a to an average reward ${\overline{r}}_{N+1}^a\in ℝ$. The algorithm of the incremental model learning method for a deterministic environment is shown in Algorithm 1.

2). Stochastic Environment

For the case of stochastic environment, the probability distribution for the state transitions needs to be taken into consideration. In the example shown in Fig. 1, when an agent in the continuous state x₁ chooses an action a₁, it would transit to the next continuous state x₂ or x₃ with probabilities 0.6 and 0.4, respectively. If the agent in the continuous state x₂ takes the action a₁, it will have different transition probabilities for the next continuous states. Since the probability distribution of the state transitions is unknown, in the model-based RL, the probability distribution must be encoded in the environmental model that is being learned by the agent.

Fig. 1.

Diagram representing a three-state Markov process, with the states labeled x₁, x₂, and x₃ and actions a₀ and a₁. Each number represents the probability of the Markov process changing from one state to another state, with the direction indicated by the arrow.

Therefore, for the stochastic environment, the variation transition function becomes $T_p:S_p\times A_p\to \mathbb{P}\left(\mathrm{\Delta }{X}_p\right)$, where $\mathrm{\Delta }{X}_p\subseteq {ℝ}^n$ is the variation of the state space and $\mathbb{P}$ is the probability operator and, similarly, the reward function is modified as ${\mathbf{R}}_p:{\mathbf{S}}_p\times {\mathbf{A}}_p\times {\mathbf{S}}_p\to \mathbb{P}\left({\mathbf{R}}_p\right)$, where ${\mathbf{R}}_p\subseteq ℝ$ is the reward value. To evaluate and encode the probability distribution in the model, we introduce a clustering method in addition to the incremental model learning scheme proposed for the deterministic case. The clustering algorithm evaluates the variation vector of the state, and the reward values for each state transition and assigns them to different clusters.

Specifically, when an agent obtains an experience (x_t, a_t, x_t+1, r_t+1), (3) is used to calculate the normalized variation of the state to obtain the normalized variation vector $v=\left(\mathrm{\Delta }{x}_G^a,r_G^a\right)$. The normalization is carried out as follows. Let d_i,m and d_i,M denote the lower and upper bounds of the ith dimension of the state space, and let R_m and R_M denote the smallest and largest reward. Then, the normalized state variation $\left(\mathrm{\Delta }{x}_G^a\right)$ and normalized reward $\left(r_G^a\right)$ are calculated as

$$\mathrm{\Delta }{x}_{G,i}^a=\frac{\mathrm{\Delta }{x}_{N,i}^a-d_{i,m}}{d_{i,M}-d_{i,m}},r_G^a=\frac{r_N^a-R_m}{R_M-R_m}$$ (10)

where $i=1,2,\dots ,\left|\mathrm{\Delta }{x}_{N,i}^a\right|=n$, $\mathrm{\Delta }{x}_{N,i}^a\in \mathrm{\Delta }{x}_N^a$ and $\mathrm{\Delta }{x}_{G,i}^a\in \mathrm{\Delta }{x}_G^a$.

When the first normalized variation vector is received, a new cluster will be established and it will be set as the center of the cluster c₁ = v, and the information of this cluster is updated using (5)–(8). When the next normalized variation vector v′ arrives, the Euclidean distance is used to calculate the similarity, D(v′, c_l), between the new normalized variation vector v′ and the centers of the existing clusters c_l, l = 1, 2, …, |C|, where |C| is the cardinality of the cluster set, C = {c₁, c₂, …, c_l}.

Next, the most similar cluster $\underset{\_}{c_l}$ will be chosen and the similarity $d_{\underset{\_}{c_l}}$ is compared with a threshold $D_{\text{th}}^a$. If the similarity is smaller than the threshold, the cluster $\underset{\_}{c_l}$ is activated and the information of this cluster is updated by (5)–(8). The number of times a cluster is activated is represented by $\underset{\_}{c_l}$. Therefore, in each discretized cell of the model, there may exist multiple clusters, and the probabilities of each cluster in a cell are evaluated using the activation number. The probabilities are used in the experience replay strategy to create a collection of virtual experiences. Therefore, in addition to the M(s, a), in the stochastic environment, the cluster centers and the activation number of each cluster are stored in the model.

In Fig. 2, an example cell and the model structure M(s, a) are shown for an environment with 1-D state and action spaces. The x, y-axes represent the actions and continuous state, discretized into cell blocks. Each discrete cell may contain several clusters, for instance, c₁, c₂, and c₃ are the center of the three clusters in a cell. In each of these clusters, the activation time $N_{c_1}$ of cluster 1 is 8, $N_{c_2}$ is 8, and $N_{c_3}$ is 6. The algorithm of the incremental model learning method for stochastic environment is shown in Algorithm 2.

Fig. 2.

Depiction of information stored in the model learned by an agent. The model structure M(s, a) corresponds to an environment with 1-D state and action spaces. The x, y-axes represent the actions and continuous state, discretized into cell blocks (black square boxes). Each discrete cell may contain several clusters, for instance (the red pop-out) shows the contents of a cell. This cell contains three clusters wherein c₁, c₂, and c₃ are their centers, respectively. In this example, the activation time $N_{c_1}$ for cluster 1 is 8, $N_{c_2}$ is 8, and $N_{c_3}$ is 6.

Note that the proposed model learning algorithms incorporate a discretization step and, then, store state variation, rewards, and their respective squared values (10). However, in contrast to the deterministic model, the stochastic model must encode the transition probabilities associated with each state transition. In both the cases, a finer discretization can provide a better approximation of the learned model, it increases the computations and, thus, introduces a tradeoff between model accuracy and computational complexity. On the other hand, the number of clusters in the stochastic environment is not predetermined, and agents automatically vary the number of clusters based on the complexity of the environment.

Next, an indirect learning algorithm incorporating experience replay is proposed to improve the sample-efficiency of the RL scheme.

C. Experience Replay for Planning (Indirect Learning)

In planning, virtual experiences, created using the model reconstructed from actual experiences via direct learning, are employed to update the Q-function (and as a consequence, the policy). To increase the efficiency of planning, we propose a multistep planning algorithm as illustrated in Fig. 3.

Fig. 3.

Example of multistep planning.

To generate virtual experiences, first, a continuous state x_t is discretized as s_t and an action a_t must be chosen from the action space. Different from the Dyna architecture, in this article, the action is chosen from the action space such that

$${\underset{\_}{a}}_t=\arg \underset{a^{\prime }\in A_p}{\max }\left((1-\lambda )e^{\mathrm{\Delta }Q\left({\underset{\_}{s}}_t,a^{\prime }\right)}+(\lambda )\kappa \right)$$ (11)

where λ ∈ [0,1] is a constant and κ is a random number selected in the interval of [0,1]. The action selection criterion evaluates the actions from the given state based on their updating error and picks the action with a large error. Alternatively, when all the actions have minor and comparable updating errors, an action is randomly picked. The first part of this selection rule enables exploitation, while the second part is used to incorporate exploration. If λ is selected close to zero, then the agent tends to choose an action based on the current Q values (with less preference to explore) while if λ is selected close to 1, the agent tends to randomly choose an action (promoting exploration) (see Fig. 10 in the supplementary material).

Algorithm 2.

Incremental Model Learning for Stochastic Environment (ModelLearning)

(1)	Calculate the variation of the continuous state and the reward using equation (3)
(2)	A normalized procedure is employed to calculated a normalized variation vector, $v\leftarrow \left(\mathrm{\Delta }{x}_N^a,r_N^a\right)$
(3)	s ← ϕ(xt)
(4)	Retrieve all clusters, C = {c1, c2, …, cl}, from the cell of the model, M(s, a) using state-action pair, (s, a), |C| is the total number of cluster in this cell.
(5)	if |C| == 0 then
(6)	The first variation vector is set as the center of the first cluster, i.e., c1 = v, c1 ∈ C
(7)	else if |C| > 0 then
(8)	$d_{\underset{\_}{c_l}}\leftarrow arg{\min }_{c_l\in C}D\left(c_l,v\right)$
(9)	if $d_{\underset{\_}{c_l}}>D_{th}^a$ then
(10)	Create a new cluster, cl+1 = v, cl+1 ∈ C
(11)	else
(12)	Activate the cluster, $\underset{\_}{c_l}$, and retrieve the information from this cluster, $\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}}}^a$, ${\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}}}^a\right)}^{\prime }$, $\left({\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)$ and ${\left({\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)}^{\prime }$
(13)	Calculate the new $\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a$, the new ${\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a\right)}^{\prime }$, the new $\left({\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a\right)$ and the new ${\left({\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a\right)}^{\prime }$ using the equations (5)–(8).
(14)	$N_{\underset{\_}{c_l}}\leftarrow N_{\underset{\_}{c_l}+1}$
(15)	end if
(16)	end if
(17)	Update the variation transition function, $T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right)$, and the reward function, Rp(s′, s, a) $$\begin{gathered} T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right)\leftarrow \mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a \\ {R}_p\left(s^{\prime },s,a\right)\leftarrow {\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a \end{gathered}$$
(18)	Store these two functions in the model, M(s, a). $$M(s,a)\leftarrow \left(T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right),R_p\left(s^{\prime },s,a\right)\right)$$
(19)	return M(s, a)

After choosing an action, the discretized state s_t and an action a_t are the input to the model. This state–action pair (s_t, a_t) will activate a cell of the model, which may contain several clusters. Using the activation number of clusters in the cell, the probability for each cluster is defined as

$$P_{c_l}\left({\underset{\_}{s}}_t,{\underset{\_}{a}}_t\right)=\frac{N_{c_l}}{{\sum }_{l=1}^{\left|C\right|}{N}_{c_l}}.$$ (12)

The cluster with higher probability, $\underset{\_}{c_l}$, is selected to retrieve the variation vector and expected reward to form the virtual experience. For example, in Fig. 2, the probability of clusters 1–3 approximates 36.36%, 36.36%, and 27.27%, respectively.

Algorithm 3.

Experience Replay for Stochastic Environment (ExperienceReplay)

(1)	g ← 0
(2)	Choose a simulated continuous state, ${\underset{\_}{\mathit{x}}}_t$, from the state-space
(3)	while g < K do
(4)	${\underset{\_}{s}}_t\leftarrow \varphi \left({\underset{\_}{x}}_t\right)$
(5)	${\underset{\_}{a}}_t\leftarrow arg{\max }_{a^{\prime }\in A_p}\left((1-\lambda )e^{\mathrm{\Delta }Q\left({\underset{\_}{s}}_t,a^{\prime }\right)}+(\lambda )\kappa \right)$
(6)	$\left(\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{{}_{c_l}}}^a,{\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)\leftarrow M\left({\underset{\_}{s}}_t,{\underset{\_}{a}}_t\right)$
(7)	${\underset{\_}{x}}_{t+1}\leftarrow {\underset{\_}{x}}_t+\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{c_l}}^a$
(8)	${\underset{\_}{\mathit{s}}}_{t+1}\leftarrow \varphi ({\underset{\_}{\mathit{x}}}_{t+1})$
(9)	Use (1) and (2) to update the Q-value and ΔQ-value
(10)	${\underset{\_}{x}}_t\leftarrow {\underset{\_}{x}}_{t+1}$
(11)	g ← g + 1
(12)	end while

From the activated cluster $\underset{\_}{c_l}$, the average variation of the continuous state $\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{c_l}}^a$ and the average reward ${\overline{r}}_{N\underset{\_}{c_l}}^a$ are retrieved, and the next continuous state is calculated as

$${\underset{\_}{x}}_{t+1}\leftarrow {\underset{\_}{x}}_t+\mathrm{\Delta }{\overline{x}}_{N\underset{\_}{c_l}}^a$$ (13)

The virtual experience, $\left({\underset{\_}{x}}_t,{\underset{\_}{a}}_t,{\underset{\_}{x}}_{t+1},{\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)$, is discretized and is used to update the policy. The algorithm of experience replay is listed in Algorithm 3, where K is the number of the steps in the multistep planning algorithm. The Q-learning has been proven to converge after the agents have had many interactions with the environment [41]. As the number of interactions between the agent and the environment increases, it is observed that the model-learning algorithms learn a reliable representation of the environment [12]. However, global convergence of the learning algorithms requires sufficient exploration of the environment and finer discretization. In the proposed indirect learning algorithm, the parameter λ controls the exploration, and it aids in the learning process (see Fig. 10 in the supplementary material). Furthermore, note that the number of clusters is not predefined in our algorithm, and the agent forms clusters online. This online clustering process influences the learning wherein the number of clusters depends on the complexity of the environment, and as the complexity increases, the agents will store the model information using more clusters automatically.

Note that it is possible for an RL agent to perform indirect learning using a model that may not be fully accurate. In the proposed algorithm, the model information includes (12), which indicates that as the number of visits increases, using the virtual experiences from the cluster with higher number of visits can be very useful. If the number of visits to a cluster is low, the agent requests information from its teammates and uses a knowledge-sharing scheme (see Section IV). This reduces the possibility of an agent trying to improve its Q-table over an imprecise environment. The agent despite using the model for indirect learning, continues to refine the model by using actual experiences via direct learning.

Next, the proposed model learning scheme for the RL agent and the experience replay algorithm are extended to a more general case involving multiagent system.

IV. Knowledge Sharing in Multiagent System

A sharing architecture for multiple agents is shown in Fig. 4. In the multiagent architecture, m homogeneous agents have their own Q-table for policy learning and their own model for model learning. The cooperative learning process is separated into two modes: 1) a learning mode and 2) a sharing mode. In the learning mode, an agent must collect experiences that are used for policy learning and model learning. In a learning task, especially, in a large state space, there may be many areas in the state space unvisited by any agent.

Fig. 4.

Sharing architecture for multiple agents.

For example, when an agent obtains a continuous state x_t+1,1 and an action, it tries to use the model to predict the next continuous state and the reward. However, in some situations, the cell of the model has insufficient information, namely, it has not been visited by the agent or the visiting times of the cell V are lower than a prescribed threshold V_th, where the visiting time is defined by

$$V=\sum _{l=1}^{\left|C\right|}{N}_l$$ (14)

with |C| being the number of clusters that is stored in the cell of the model.

If V is lower than V_th, the cell is viewed as one with insufficient information. In such a case, to compensate for the insufficient model information, this agent must switch to the sharing mode and the process of information exchange between agents is shown in Fig. 5. In this example, first, agent 2 with insufficient information broadcasts a request including the continuous state for which the information is needed. Upon receiving the request, agents 1 and 3 find the corresponding cell in their models and send the corresponding model information back to agent 2 in the second step. Agent 2 then fuses its original model information with the received model information from the other agents. Here, we introduce a fusing mechanism to achieve this goal. Let C_sh = [c₁, c₂, …, c_q] be the clusters received by an agent from its teammates and let C_or = [c₁, c₂, …, c_o] be its own clusters corresponding to that cell with the respective cardinality denoted as |C_sh| and |C_or|. Before the fusing procedure, the variance of the variation of the continuous states ${\left({\underset{\_}{s}}_s^a\right)}^2$ and the variance of the reward ${\left({\underset{\_}{s}}_r^a\right)}^2$ for all the clusters in the sets, |C_sh| and |C_or|, are calculated as

$${\left({\underset{\_}{s}}_s^a\right)}^2={\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^{\prime }-{\left(\mathrm{\Delta }{\overline{x}}_N^a\right)}^2,{\left({\underset{\_}{s}}_r^a\right)}^2={\left({\overline{r}}_N^a\right)}^{\prime }-{\left({\overline{r}}_N^a\right)}^2.$$ (15)

Next, to initiate the fusing procedure, T-statistic [43] is used wherein the similarity between any two clusters is quantified based on a t-value. For each cluster pair (one from C_or and the other from C_sh), a t-value vector composed of t_v = [t₁, t₂, …, t_n], where n is the dimension of the state space, and t_r corresponding to the reward is calculated as

$$t_v=\frac{\left|\mathrm{\Delta }{\overline{x}}_{N,or}^a-\mathrm{\Delta }{\overline{x}}_{N,sh}^a\right|}{\sqrt{\left(\frac{{\left({\underset{\_}{s}}_{s,or}^a\right)}^2}{N_{i,or}}\right)+\left(\frac{{\left({\underset{\_}{s}}_{s,sh}^a\right)}^2}{N_{l,sh}}\right)}}$$ (16)

$$t_r=\frac{|{\overline{r}}_{N,or}^a-{\overline{r}}_{N,sh}^a|}{\sqrt{\left(\frac{{\left({\underset{\_}{s}}_{r,or}^a\right)}^2}{N_{i,or}}\right)+\left(\frac{{\left({\underset{\_}{s}}_{r,sh}^a\right)}^2}{N_{l,sh}}\right)}}$$ (17)

where N_i and N_l are the activation number of the ith, lth cluster from C_or and C_sh, respectively.

Fig. 5.

Illustration of the sharing process.

Note that in the proposed algorithm, the agent receives clusters C_sh from all its teammates. During the implementation of this algorithm, the agent, upon requesting information from its teammates, will receive clusters from all its teammates, which are stored in a buffer. Then, the agent will retrieve these clusters from the buffer and will update the model in batches. Also, if the agents do not have sufficient information (determined based on V and V_th), no new information will be sent to the agent which initiated a request, and the buffer will be empty.

If one element of the t-value vector t_n and t_r are larger than the threshold t_th, then, these two clusters will not be merged into one cluster. Otherwise, if all the elements of t_n and t_r are smaller than the threshold, it implies that the two clusters have similar sample distributions and they can be merged into one cluster. In such a case, the information of this cluster is updated. In particular, the average of the variation of the continuous state and of the reward will be updated as

$$\begin{gathered} \mathrm{\Delta }{\overline{x}}_{N_{\text{total},\text{or}}}^a=\frac{N_{i,or}}{N_{\text{total}}}\times \mathrm{\Delta }{\overline{x}}_{N,or}^a+\frac{N_{l,sh}}{N_{\text{total}}}\times \mathrm{\Delta }{\overline{x}}_{N,sh}^a \\ {\overline{r}}_{N_{\text{total},\text{or}}}^a=\frac{N_{i,or}}{N_{\text{total}}}\times {\overline{r}}_{N,or}^a+\frac{N_{l,sh}}{N_{\text{total}}}\times {\overline{r}}_{N,sh}^a \end{gathered}$$ (18)

and the new average of the squared variation of the continuous state and the reward will be updated as

$$\begin{gathered} {\left(\mathrm{\Delta }{\overline{x}}_{N,or}^a\right)}^{\prime }=\frac{N_{i,or}}{N_{\text{total}}}\times {\left(\mathrm{\Delta }{\overline{x}}_{N,or}^a\right)}^{\prime }+\frac{N_{i,sh}}{N_{\text{total}}}\times {\left(\mathrm{\Delta }{\overline{x}}_{N,sh}^a\right)}^{\prime } \\ {\left({\overline{r}}_{N,or}^a\right)}^{\prime }=\frac{N_{i,or}}{N_{\text{total}}}\times {\left({\overline{r}}_{N,\text{or}}^a\right)}^{\prime }+\frac{N_{l,sh}}{N_{\text{total}}}\times {\left({\overline{r}}_{N,sh}^a\right)}^{\prime } \end{gathered}$$ (19)

where N_total = N_i,or + N_l,sh. We observe that the weights ([N_i,or/N_total], [N_i,sh/N_total]) in these update rules can be chosen between (0, 1). In our simulations, we update the model information using the convex combination of the agents own knowledge and the knowledge shared by the teammates [as in (18) and (19)]. Thus, by using the clustering approach, the introduced fusing mechanism circumvents the resampling scheme required in the existing algorithm [37]. This is achieved because of the cluster-based model learning, which avoids the heterogeneous tree structures.

Next, simulation analysis for the proposed methods is presented.

V. Simulations

In this section, first, a mountain car simulation is used to compare the learning efficiency for a single agent. The proposed method is compared with Q-learning [41], backward Q-learning (BQ-learning) [21], Q-learning with experience replay based on reward and time [QExp-learning (Time) and QExp-learning (Reward)] [20], and Dyna-Q learning [12].

Next, we evaluate the framework where multiple agents learn the state-space model and a policy. To this end, we apply the proposed knowledge-sharing algorithm to the multiagent system studied in the goal search problem and in a maze environment and compare the performance resulting from our method with other commonly used sharing methods in [33] and [37]. Here, we do not consider tasks involving co-operative or competitive games. Therefore, the agents choose their actions independently. In all these examples, the learning task is complete when each robot in the team learns to reach the goal. We do assume that there is no loss due to communication during knowledge sharing. In Examples 2 and 3, we analyze the knowledge-sharing algorithm, and we present a comparative analysis between the proposed methods and some existing methods, in terms of the learning steps. Since each algorithms use different learning structure to approximate the environment, the information stored in each algorithm is different, and the balance between exploration and exploitation by each agent in the environment is different among the algorithms. Therefore, the comparison between algorithms in terms of communication instances would not be fair. However, the termination criterion for each episode is fixed in the comparative analysis with different methods and, therefore, the convergence of the algorithms in terms of the episodes implicitly provide a qualitative comparison on the time for convergence of the learning algorithms.

To evaluate the statistical significance of the results, we perform a Monte-Carlo analysis and record the average number of steps required for convergence of the algorithm (over multiple episodes) and its standard deviation. For each example, we include the average convergence and comparative analysis and present detailed implementation procedure in the main text, and the standard deviation of the number of steps required for convergence over multiple episodes are recorded in the supplementary material.

A. Mountain Car

In the mountain car simulation (Fig. 6), a car in the valley has three actions, that is, throttle forward (+1), throttle reverse (−1), or zero throttle (0). However, these actions cannot output sufficient power to drive the car to reach the hill. As a result, the car must learn a policy to save the potential energy by applying a sequence of actions so that it can be propelled to reach the goal. The dynamics of the mountain car are

$$\begin{gathered} p_{t+1}=p_t+{\dot{p}}_{t+1} \\ {\dot{p}}_{t+1}={\dot{p}}_t+0.01a_t+0.01{\omega }_t-0.0025\cos \left(3.0\times p_t\right) \end{gathered}$$ (20)

where p_t ∈ [−1.2,0.5] is the position of the car, ${\dot{p}}_t\in [-0.07,0.07]$ is the velocity of the car, and ω_t is a Gaussian random noise process [1]. In the process of learning, if the car reached the goal, an immediate reward of +1.0 was returned by the environment. Otherwise, the car received a reward of −1.0. There were 30 trials for the simulation, and each trail included 200 episodes with 1500 steps per episode. If the car reached the hill or it cannot reach the hill within 1500 steps, then the episode was terminated, and a new episode was started. In each episode, the initial position and velocity of the car were set to −0.5 and 0.0, respectively. The setting of the parameters are listed in Table I in the supplementary material. The resolution of the Q-table and the proposed model were selected as 50 × 40 discretized grids.

Fig. 6.

Mountain car simulation.

The average numbers of the steps for the car to reach the goal are shown in Fig. 7. The x-axis represents the number of episodes and the y-axis represents the number of steps. In this figure, six curves represent the results from Q-learning, BQ-learning, Q-learning with experience replay based on time and reward, Dyna-Q, and the proposed methods (Algorithms 2 and 3), respectively. In this comparison shown in Fig. 7, the proposed method reduced the average steps required to reach the goal to be lower than 300 and accelerated the learning process in comparison with the existing methods.

Fig. 7.

Simulation results in mountain car.

B. Cooperative Goal Searching Problem

In this example, three cooperative robots are placed in the environment as shown in Fig. 8. The size of the environment is 300 × 300 and surrounded by walls, with obstacles. The three robots are placed in different areas labeled as Area 1, Area 2, and Area 3. Robot 1 (R1), robot 2 (R2), and robot 3 (R3) start from their respective initial areas R1, R2, and R3 (painted with green color). The size of these initial areas is 10 × 10 units. If a robot finishes an episode, it will randomly start from its initial position (the area marked in green) in the next episode. The red area is the goal.

Fig. 8.

Cooperative goal search.

In this simulation, robot 2 is the nearest to the goal, and hence it is easier for it to reach the goal, while robots 1 and 3 need to explore more exhaustively to find a path to avoid the obstacle and eventually reach the goal. Therefore, in this cooperative task, robot 2 obtains useful knowledge easily, and when the robots 1 and 3 enter R2 (Area 2), the robot 2 can share its own knowledge to help the other robots.

In this task, all the three robots have the same action space, {moving up, down, left, and right}. The moving distance for each action is set as 5 in the presence of the Gaussian noise with zero mean and standard deviation 0.5. After the robots take an action, they met with three types of situations: 1) if a robot hits the wall or the obstacle, it received a reward of −100.0; 2) if a robot reached the goal, it received a reward of 100.0; and 3) else, it received a reward of −10⁻⁶.

In the process of learning, when the number of steps reached 2000 or if the robot touched the goal, an episode was ended. There were 500 episodes per trial and 40 trials. The parameters for the cooperative searching goal problem are listed in Table II in the supplementary material. The resolution of the Q-table and the proposed model was selected as 100 × 100 discretized grids. Using the proposed sharing mechanism, robots 1 and 3 will decrease the required exploration by using the knowledge gained by robot 2. The proposed method is compared with some other knowledge-sharing methods. In the algorithm presented by Tan [33], all the robots shared their experiences with each other, while in [37], three sharing mechanisms based on a tree structure were proposed. Based on the simulation results, the sharing method, sharing under request for ranges, increased the learning efficiency considerably in comparison with the other schemes.

In Fig. 9, the x-axis is the number of episodes and the y-axis is the average number of steps. In Fig. 9(a), the proposed method decreased the average steps required for robot 1 to reach the goal down to 200 in the 36th episode. The sharing under request decreased the average steps to 200 in the 70th episode. Dyna-Q with sharing experiences decreased the average steps in 169th episode, and the Dyna-Q without sharing experiences decreased the average steps until the 196th episode.

Fig. 9.

Simulation results in cooperative searching goal problem. (a) Learning curves for robot 1. (b) Learning curves for robot 2. (c) Learning curves for robot 3.

Since robot 2 was near the goal, the difference of the average steps was not noticeable as shown in Fig. 9(b). The learning curve of robot 3 is shown in Fig. 9(c), where it can be observed that the proposed method decreased the average steps for robot 3 to reach the goal to 200 in the 43rd episode. The sharing under request decreased the average steps down to 200 in the 70th episode while the Dyna-Q with sharing experiences decreased the average steps in the 195th episode, and the Dyna-Q without sharing experiences in the 197th episode.

According to these simulation results, the Dyna-Q with sharing experiences reduced the average steps required for convergence in the initial episodes, but not in the later episodes, while the sharing under request for ranges decreased the average steps slower than the Dyna-Q with sharing in the initial episodes, but as the learning episodes increased, as a team, the multiagent framework improved the knowledge gathered, and as a consequence, the convergence was improved in the later episodes. On the other hand, with the proposed method, the average steps required both in the initial and the later episodes decreased. The third simulation example and additional analyses are presented in the Supplementary Material.

VI. Conclusion

In this article, a model-based RL scheme was proposed wherein the collected experiences are not only used for policy learning but also used for model learning to approximate the environment (deterministic and stochastic). By employing the learnt model, the agent simulated virtual experiences for indirect learning and planning. The clustering method introduced in this article was employed in tandem with the proposed incremental model learning method to evaluate the transition probability and encode this in the model. The simulation results and comparative analysis demonstrated that the proposed methods require fewer learning steps to reach the goal and as a consequence improve the sample efficiency.

Moreover, to solve the complex learning tasks in a large state space, the proposed methods we extended to incorporate knowledge sharing in a multiagent system. In other words, the cost of exploration can be reduced by leveraging the benefits of knowledge sharing and the simulation results indicated that the learning steps are considerably reduced with the proposed method. Furthermore, due to the clustering method employed in the model learning, the knowledge fusing procedure circumvents the computationally intensive resampling methods.

In this article, we did not consider problems with cooperative/competitive games, where actions of an agent affects other agents. These problems require learning joint state and joint action spaces and are practically significant, and the proposed algorithms can be expanded to incorporate such tasks. The second line of inquiry would involve theoretical analysis of the proposed learning algorithms and, finally, explicitly accounting for the communication losses, such as delay, data drop out, noisy data in the knowledge-sharing scheme, and circumventing the discretization step would further expand the application of the proposed algorithms.

Biography

Wei-Cheng Jiang received the B.S. degree in computer science and information engineering and the M.S. degree in electro-optical and materials science from National Formosa University, Yunlin, Taiwan, in 2007 and 2009, respectively, and the Ph.D. degree in electric engineering from National Chung Cheng University, Chiayi, Taiwan, in 2013.

He was a Postdoctoral Fellow with the Electrical Engineering Department, National Sun Yat-sen University, Kaohsiung, Taiwan. He was a Visiting Scholar with the Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, MO, USA. Since 2019, he has been an Assistant Professor with the Department of Electrical Engineering, Tunghai University, Taichung, Taiwan. His research interests include machine learning, multiagent systems, and intelligent control.

Vignesh Narayanan (M’17) received the B.Tech. degree from SASTRA University, Thanjavur, India, the M.Tech. degree from the National Institute of Technology Kurukshetra, Kurukshetra, India, and the Ph.D. degree from the Missouri University of Science and Technology, Rolla, MO, USA, in 2017.

He is currently working as a Postdoctoral Research Associate with Washington University in St. Louis, St. Louis, MO, USA. His research interests include control, neural networks, learning, and adaptation in systems theory.

Jr-Shin Li (M’06–SM’18) received the B.S. and M.S. degrees from National Taiwan University, Taipei, Taiwan, and the Ph.D. degree in applied mathematics from Harvard University, Cambridge, MA, USA, in 2006.

He is currently a Professor of electrical and systems engineering with the Joint Appointment Division of Biology and Biomedical Sciences, Washington University in St. Louis, St. Louis, MO, USA. His research interests are control theory, computational mathematics, optimization, learning, and complex networks. His current work involves developing model-based and data-driven methods for dynamical systems and control of large-scale complex systems with applications ranging from neuroscience and biology to quantum physics.

Prof. Li has been a recipient of the NSF Career Award in 2007 and the AFOSR Young Investigator Award in 2009.