Algorithm 1 Proximal policy optimization (PPO) for trajectory planning

Require: Start ${\mathit{\theta }}_{start}$, end angle ${\mathit{\theta }}_{end}$, hyperparameters of actor and critic networks, learning rate $\alpha$, discount factor $\gamma$, generalized advantage estimation (GAE) parameter $\lambda$, clipping parameter $ϵ$, entropy coefficient $c_{ent}$, number of iterations N, number of epochs E, batch size B, episode length T, accuracy setting $d_e$. 1:for $n=1,2,\dots ,N$ do 2:      Collect T time steps transition $\left\{\begin{array}{cccc}{s}_t& a_t& r_t& s_{t+1}\end{array}\right\}$ using the running policy ${\pi }_{\theta }$ 3:      if $f_a\left(\mathit{\theta }\right)<d_e$ then 4:          $T\leftarrow T_{\mathrm{current}}$ 5:      end if 6:      Compute advantage estimates ${\widehat{A}}_t$ 7:      for $e=1,2,\dots ,E$ do 8:            Randomly sample a batch of B time steps 9:            Update the critic network parameters by minimizing the loss: 10:          $L\left(\varphi \right)=\frac{1}{B}{\sum }_{t=1}^B{(V_{\varphi }\left(s_t\right)-{\widehat{R}}_t)}^2$ 11:          Update the actor network parameters using the PPO-clip objective and the entropy bonus: 12:          ${\theta }_{k+1}=arg{max}_{\theta }\frac{1}{B}{\sum }_{t=1}^B{L}^{CLIP}\left(\theta \right)+c_{ent}H\left({\pi }_{\theta }(·|s_t)\right)$ 13:          $L^{CLIP}\left(\theta \right)=min\left(\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_k}\left(a_t\right|s_t)}{\widehat{A}}_t,clip\left(\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_k}\left(a_t\right|s_t)},1-ϵ,1+ϵ\right){\widehat{A}}_t\right)$ 14:      end for 15:end for