Algorithm 2 Bellman EM algorithm (BEM)

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z};{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     $F\left({\mathit{\theta }}_k\right)\leftarrow {(I-\gamma P\left({\mathit{\theta }}_k\right))}^{-1}p\left({\mathit{\nu }}_k\right)$     $V\left({\mathit{\theta }}_k\right)\leftarrow {\left({(I-\gamma P\left({\mathit{\theta }}_k\right))}^{-1}\right)}^{T}r\left({\mathit{\pi }}_k\right)$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$