Algorithm 1: Actor-Critic in the proposed model

Input: Iteration T, time step $\alpha$, discount factor $\gamma$, hypermeter for policy network $\theta$ Process: Initialize observations of states $\phi \left(s_1\right),\phi \left(s_2\right),...,\phi \left(s_n\right)$ for $i=1,2,...,T$: $R_i$,${\phi }^{\prime }\left(s_{i+1}\right)$,$A_{i+1}$ = Actor($\phi \left(s_i\right)$,$A_i$) $V\left(s_i\right)$,$V^{\prime }\left(s_i\right)$ = Critic($\phi \left(s_i\right)$,${\phi }^{\prime }\left(s_i\right)$) Update TD Error by $\delta \leftarrow R_i+V^{\prime }\left(s_i\right)-V\left(s_i\right)$ Update Critic by $\omega \leftarrow \omega +\beta \delta V\left(s_i\right)$ Update Actor by $\theta \leftarrow \theta +\alpha {▿}_{\theta }log{\pi }_{\theta }(S_t,A_i)\delta$ Update State by $\phi \left(s_i\right)={\phi }^{\prime }\left(s_{i+1}\right)$ end for Output: $\omega ,\theta$