Algorithm 4. Pretraining the dynamics network.

Function Pretrain_Dynamics_Network (${\mathrm{f}}_{\mathsf{\varphi }},$ $\mathrm{D}$) /* the dynamics network ${\mathrm{f}}_{\mathsf{\varphi }}$, the demo dataset $\mathrm{D}$ */ for e = 0,…,E epochs do   for i = 0,…,|D| do Initialize the episodic buffer $\mathrm{B}$ to be empty Sample $\mathrm{d}=〈\left({\mathrm{s}}_1^{\mathrm{E}},{\mathrm{a}}_1^{\mathrm{E}}\right),\left({\mathrm{s}}_2^{\mathrm{E}},{\mathrm{a}}_2^{\mathrm{E}}\right),\dots ,\left({\mathrm{s}}_{il}^{\mathrm{E}},{\mathrm{a}}_{il}^{\mathrm{E}}\right)〉$ from $\mathrm{D}$ Initialize the environment to initial state ${\mathrm{s}}_1^{\mathrm{E}}$ for t = 1,….$il-1$ timesteps do Execute an action ${\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}}$ and perceive the next state ${\mathrm{s}}_{\mathrm{t}+1}$ B $\leftarrow$ B$\cup${$\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}},{\mathrm{s}}_{\mathrm{t}+1}\right)\}$//append each state transition to B     end for Update the dynamics network parameters $\mathsf{\varphi }$ by gradient descent        $\mathsf{\varphi }\leftarrow \mathsf{\varphi }-{\nabla }_{\mathsf{\varphi }}\left({\displaystyle {\displaystyle \sum }_{\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}},{\mathrm{s}}_{\mathrm{t}+1}\right)\in \mathrm{B}}}\Vert {\mathrm{f}}_{\mathsf{\varphi }}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{E}}\right)-{\mathrm{s}}_{\mathrm{t}+1}\Vert {}_2^2\right)$    end for end for return ${\mathrm{f}}_{\mathsf{\varphi }}$