Algorithm 2. Behavior cloning.

Function Behavior_Cloning(${\mathsf{\pi }}_{\mathsf{\theta }}$, $\mathrm{d}$) /* the policy network ${\mathsf{\pi }}_{\mathsf{\theta }}$ the demo data $\mathrm{d}=〈\left(s_1^E,a_1^E\right), \left(s_2^E,a_2^E\right),\dots , \left(s_l^E,a_l^E\right)〉$ */ Initialize the episodic buffer $\mathrm{B}$ to be empty for t = 1,…,$l$ timesteps do Sample an action ${\mathrm{a}}_{\mathrm{t}}~{\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{s}}_{\mathrm{t}}^{\mathrm{E}}\right)$             ${\mathrm{p}}_{\mathrm{t}}=\mathsf{{\rm P}}({\mathrm{a}}_{\mathrm{t}}|{\mathrm{s}}_{\mathrm{t}}^{\mathrm{E}};{\mathsf{\pi }}_{\mathsf{\theta }})$           $\mathrm{B}\leftarrow \mathrm{B}\cup \left\{\left({\mathrm{s}}_{\mathrm{t}}^{\mathrm{E}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}},{\mathrm{a}}_{\mathrm{t}},{\mathrm{p}}_{\mathrm{t}}\right)\right\}$ end for Estimate the BC loss ${\mathrm{L}}_{\mathrm{BC}}$           ${\mathrm{L}}_{\mathrm{BC}}={\displaystyle {\displaystyle \sum }_{\left({\mathrm{s}}_{\mathrm{t}}^{\mathrm{E}},{\mathrm{a}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}}\right)\in \mathrm{B}}}\Vert {\mathrm{a}}_{\mathrm{t}}-{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}}\Vert {}_2^2$ Calculate the degree of cloning ${\mathrm{d}}_{conv}$           ${\mathrm{d}}_{\mathrm{conv}}=\frac{1}{l}{\displaystyle {\displaystyle \sum }_{\mathrm{t}=1}^l}{\mathrm{p}}_{\mathrm{t}}$ return ${\mathrm{L}}_{\mathrm{BC}},{\mathrm{d}}_{\mathrm{conv}}$