Algorithm 1 Proximal Policy Optimization (PPO)

Require: Initialize the policy parameters $\theta$ and the value function parameters $\varphi$   while not converged, do    for each training iteration, do     for each environment step, do       $s_t\leftarrow$ current state       $a_t\leftarrow$ action sampled from policy ${\pi }_{\theta }$       $r_t\leftarrow$ reward received       $s_{t+1}\leftarrow$ next state after executing $a_t$       Store $(s_t,a_t,r_t,s_{t+1})$ in buffer     end for     compute the advantage estimates ${\widehat{A}}_t$ using Generalized Advantage Estimation (GAE)     for each policy update, step do       $L\left(\theta \right)\leftarrow \mathbb{E}\left[\min \left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)\right]$       update policy $\theta$ using gradient ascent on $L\left(\theta \right)$$L_V\left(\varphi \right)\leftarrow {(V_{\varphi }\left(s_t\right)-R_t)}^2$       update value function $\varphi$ by minimizing $L_V\left(\varphi \right)$     end for    end for   end while