Algorithm 1 Discretized soft-actor-critic algorithm.

1:Initialize the experience buffer D 2:Initialize the weight $\alpha$ with 1 3:Initialize the actor network ${\pi }_{\varphi }$ with random parameter $\varphi$ 4:Initialize main critic networks $Q_{{\theta }_i}$ with random parameters ${\theta }_i$ for $i\in \left\{1,2\right\}$ 5:Initialize target critic networks $Q_{{\widehat{\theta }}_i}$ with parameters ${\widehat{\theta }}_i$ as main critic networks $Q_{{\theta }_i}$ 6:for each training episode do 7:   Observe initial state $s_t$ 8:   for each step $t=1,2,...,T$ do 9:     Generate the action $a_t={\pi }_{\varphi }\left(s_t\right)$ 10:     Execute the action $a_t$ 11:     Observe the next state $s_{t+1}$ and the reward $r_t$ 12:     Store the experience $(s_t,a_t,r_t,s_{t+1})$ at the experience buffer D 13:     Sample a mini-batch $\overline{D}$ of a few experiences from the buffer D 14:     Calculate the target state value $V_{\widehat{\theta }}\left(s^{\prime }\right)$ based on Equation (9) 15:     Update the main critic network $Q_{{\theta }_i}$ based on the gradient ${\nabla }_{{\theta }_i}{J}_Q\left({\theta }_i\right)$ in Equation (10) 16:     Update the actor network ${\pi }_{\varphi }$ based on the gradient ${\nabla }_{\varphi }{J}_{\pi }\left(\varphi \right)$ in Equation (12) 17:     Update the weight $\alpha$ based on the gradient ${\nabla }_{\alpha }J\left(\alpha \right)$ in Equation (13) 18:     For every B steps, use soft update for the target critic networks based on Equation (14) 19:   end for 20:end for