Algorithm 2. Proximal Policy Optimization (PPO) with Pre-Trained MN Reward

Input: Pre—date set $L^{\prime }={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^k$, load the parameter $\chi$ of pre-trained MN

1: Initialize PPO parameter $\theta$ and policy ${\pi }_{\theta }$

2: For each iteration $I$ do

3:  While the robot does not reach the target position do

4:    Initializes the robot position

5:    Get state $s_t$ (including laser scan state $o_t$ and relative position $d_t$)

6:    Run ${\pi }_{\theta }$ generate action $a_t$

7:    Add $s_t$ to the queue ${\left(s_1,s_2,\dots ,s_{t-1}\right)}_I$

8:    Input ${\left(s_1,s_2,\dots ,s_{t-1}\right)}_I$ into MN

9:    Get reward $r_t$ from MN and environment

10:    Collect $\left\{s_t{,\mathrm{a}}_t{,\mathrm{r}}_t\right\}$

11:  Add ${\left(s_1,s_2,\dots ,s_{t-1}\right)}_I$ and task result (Label) to the MN’s data set $L^{\prime }$

12:  If $I$ is a multiple of 3 then

13:    Update parameter $\chi$ of MN by $L^{\prime }$ with few epochs

14:  Compute estimated advantage $A_t$

15:  Update $\theta$ with K epochs