Algorithm 1.

Train DLSM.

Input: Model weights $\psi$, $\varphi$, experience replay buffer ${ℰ}^{\mu }$, and learning rate $\alpha$.
Begin:
1:	Initialize $\psi$, $\varphi$
2:	for iter in 1 : max_iter do
3:	Sample a trajectory $\left[\left(s_0,a_0,r_0,s_1\right),\dots ,\left(s_{T-1},a_{T-1},r_{T-1},s_T\right)\right]~{ℰ}^{\mu }$
4:	$z_0^{\varphi }~q_{\varphi }\left(z_0\mid s_0\right)$
5:	$z_0^{\psi }~p_{\psi }\left(z_0\right)$
6:	Run forward pass of DLSM following (13) and (15) for $t=1:T$, and collect all variables needed to evaluate the all terms within the expectation in ${ℒ}_{ELBO}$, which is denoted as ${\widetilde{ℒ}}_{ELBO}$.
7:	$\psi \leftarrow \psi +\alpha {\nabla }_{\psi }{\widetilde{ℒ}}_{ELBO}$
8:	$\varphi \leftarrow \varphi +\alpha {\nabla }_{\varphi }{\widetilde{ℒ}}_{ELBO}$
9:	end for