Algorithm 3 Modified Bellman EM algorithm (MBEM)

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. $F(x,z;{\mathit{\theta }}_{k-1})\leftarrow p_0(x,z;{\mathit{\nu }}_k)$ $V(x,z;{\mathit{\theta }}_{k-1})\leftarrow \overline{r}(x,z;{\mathit{\pi }}_k)$ while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     $F_0(x,z)\leftarrow F(x,z;{\mathit{\theta }}_{k-1})$     $V_0(x,z)\leftarrow F(x,z;{\mathit{\theta }}_{k-1})$     $L\leftarrow 0$    repeat        $F_{L+1}(x,z)\leftarrow A_{{\mathit{\theta }}_k}{F}_L(x,z)$        $V_{L+1}(x,z)\leftarrow B_{{\mathit{\theta }}_k}{V}_L(x,z)$        $L\leftarrow L+1$    until $max\{\parallel F_L-F_{L-1}{\parallel }_1,\parallel V_L-V_{L-1}{\parallel }_{\infty }\}<\frac{1-\gamma }{\gamma }\epsilon$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$