Algorithm 1: V-learning.
1 Initialize a class of policies, $\Pi =\left\{{\pi }_{\beta }:\beta \in \mathcal{B}\right\}$, and a model, V (π, s; θπ);
2 Set k = 1 and initialize β1 to a starting value in $\mathcal{B}$;
3 while Not converged do
4  Estimate ${\widehat{\theta }}_n^{{\pi }_{{\beta }^k}}=\text{arg}{\text{min}}_{{\theta }^{{\pi }_{{\beta }^k}}\in \Theta }\left\{{\Lambda }_n{\left({\pi }_{{\beta }^k},{\theta }^{{\pi }_{{\beta }^k}}\right)}^{⊺}\Omega {\Lambda }_n\left({\pi }_{{\beta }^k},{\theta }^{{\pi }_{{\beta }^k}}\right)+{\lambda }_n\mathcal{P}\left({\theta }^{{\pi }_{{\beta }^k}}\right)\right\}$;
5  Evaluate ${\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)={\displaystyle \int V\left({\pi }_{{\beta }^k},S;{\widehat{\theta }}_n^{{\pi }_{{\beta }^k}}\right)}\text{d}\mathcal{R}\left(s\right)$;
6  Set ${\beta }^{k+1}={\beta }^k+{\alpha }^k{\nabla }_{{\beta }_k}{\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$ for some step size, αk, where ${\nabla }_{{\beta }_k}{\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$ is the gradient of ${\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$;
7 end