Algorithm 2.

Incremental Model Learning for Stochastic Environment (ModelLearning)

(1)	Calculate the variation of the continuous state and the reward using equation (3)
(2)	A normalized procedure is employed to calculated a normalized variation vector, $v\leftarrow \left(\mathrm{\Delta }{x}_N^a,r_N^a\right)$
(3)	s ← ϕ(xt)
(4)	Retrieve all clusters, C = {c1, c2, …, cl}, from the cell of the model, M(s, a) using state-action pair, (s, a), |C| is the total number of cluster in this cell.
(5)	if |C| == 0 then
(6)	The first variation vector is set as the center of the first cluster, i.e., c1 = v, c1 ∈ C
(7)	else if |C| > 0 then
(8)	$d_{\underset{\_}{c_l}}\leftarrow arg{\min }_{c_l\in C}D\left(c_l,v\right)$
(9)	if $d_{\underset{\_}{c_l}}>D_{th}^a$ then
(10)	Create a new cluster, cl+1 = v, cl+1 ∈ C
(11)	else
(12)	Activate the cluster, $\underset{\_}{c_l}$, and retrieve the information from this cluster, $\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}}}^a$, ${\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}}}^a\right)}^{\prime }$, $\left({\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)$ and ${\left({\overline{r}}_{N_{\underset{\_}{c_l}}}^a\right)}^{\prime }$
(13)	Calculate the new $\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a$, the new ${\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a\right)}^{\prime }$, the new $\left({\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a\right)$ and the new ${\left({\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a\right)}^{\prime }$ using the equations (5)–(8).
(14)	$N_{\underset{\_}{c_l}}\leftarrow N_{\underset{\_}{c_l}+1}$
(15)	end if
(16)	end if
(17)	Update the variation transition function, $T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right)$, and the reward function, Rp(s′, s, a) $$\begin{gathered} T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right)\leftarrow \mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a \\ {R}_p\left(s^{\prime },s,a\right)\leftarrow {\overline{r}}_{N_{\underset{\_}{c_l}+1}}^a \end{gathered}$$
(18)	Store these two functions in the model, M(s, a). $$M(s,a)\leftarrow \left(T_p\left(\mathrm{\Delta }{\overline{x}}_{N_{\underset{\_}{c_l}+1}}^a|s,a\right),R_p\left(s^{\prime },s,a\right)\right)$$
(19)	return M(s, a)