Algorithm 2 CLI-PPO End-to-End Training Framework

1:Initialize PPO policy parameters $\theta$, LSTM parameters ${\theta }_{lstm}$, ICM parameters ${\theta }_{ICM}=\{{\theta }_{\varphi },{\theta }_f,{\theta }_g\}$, rollout buffer $\mathcal{B}$ and success buffer ${\mathcal{B}}_{success}$   2:Set $T_{max}=5\times {10}^7$, $N=24$, $n_{steps}=2048$   3:Set learning rate $lr=1\times {10}^{-5}$, discount factor $\gamma =0.999$, intrinsic reward coefficient $\beta =0.1$, and curriculum threshold $\rho =0.7$   4:Initialize curriculum boundary ratio $\alpha =0.3$   5:for $iter=1$ to ${\displaystyle \frac{T_{max}}{N·n_{steps}}}$ do   6:    for $step=1$ to $n_{steps}$ do   7:        for $i=1$ to N do   8:            Obtain observation: $o_t^i\leftarrow \mathrm{GetObservation}\left(En{v}_i\right)$   9:            Update recurrent hidden states: $h_t^i,c_t^i\leftarrow \mathrm{LSTM}(o_t^i,h_{t-1}^i,c_{t-1}^i;{\theta }_{lstm})$ 10:            Sample continuous action: $a_t^i\sim {\pi }_{\theta }\left(a_t^i\right|h_t^i)$ 11:            Execute action: $(o_{t+1}^i,r_{ext}^i,don{e}^i)\leftarrow \mathrm{Step}(En{v}_i,a_t^i)$ 12:            Compute intrinsic curiosity reward: $r_{int}^i\leftarrow \mathrm{ComputeICM}(o_t^i,o_{t+1}^i,a_t^i;{\theta }_{ICM})$ 13:            Compute total reward: $r_t^i=r_{ext}^i+\beta r_{int}^i$ 14:            Store transition: $\mathcal{B}\leftarrow (o_t^i,h_t^i,a_t^i,r_t^i,o_{t+1}^i)$ 15:        end for 16:    end for 17:    Compute ICM loss: ${\mathcal{L}}_{ICM}\leftarrow \mathrm{ComputeICMLoss}\left(\mathcal{B}\right)$ 18:    Update ICM parameters: ${\theta }_{ICM}\leftarrow \mathrm{AdamUpdate}({\theta }_{ICM},{\mathcal{L}}_{ICM})$ 19:    Update PPO policy: $(\theta ,{\theta }_{lstm})\leftarrow \mathrm{PPOUpdate}(\theta ,{\theta }_{lstm},\mathcal{B})$ 20:    Update success buffer: ${\mathcal{B}}_{success}\leftarrow \mathrm{UpdateSuccessBuffer}\left(done\right)$ 21:    if $\mathrm{Mean}\left({\mathcal{B}}_{success}\right)\ge \rho$ then 22:        Expand curriculum boundary: $\alpha \leftarrow \mathrm{ExpandBoundary}\left(\alpha \right)$ 23:        Update environment boundary: $\mathrm{SetBoundary}(Env,\alpha )$ 24:    end if 25:    Clear rollout buffer: $\mathcal{B}\leftarrow \varnothing$ 26:end for 27:return $\theta ,{\theta }_{lstm}$