Algorithm 1.

Actor’s update.

1: if δt > 0 then

2:   for i: = 1 to $⌈{\mathrm{\delta }}_t∕\sqrt{\mathit{va}{r}_t}⌉$ step 1 do

3:    ${\mathrm{\theta }}_{i,t+1}^A={\mathrm{\theta }}_{i,t}^A+\mathrm{\alpha }\left(a_t^{\ast }-A\left(s_t\right)\right)\frac{\mathrm{\partial A}\left(s_t\right)}{\partial {\mathrm{\theta }}_{i,t}^A}$

4:   end for

5:  end if