Algorithm 1 EM algorithm for DEC-POMDP

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. $T_{max}\leftarrow \lceil (log(1-\gamma )\epsilon )/log\gamma -1\rceil$ while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     ${\alpha }_0(x,z;{\mathit{\theta }}_k)\leftarrow p_0(x,z;{\mathit{\nu }}_k)$     ${\beta }_0(x,z;{\mathit{\theta }}_k)\leftarrow \overline{r}(x,z;{\mathit{\pi }}_k)$    for  $t=1,2,\dots ,T_{max}$  do        ${\alpha }_t(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{x^{\prime },z^{\prime }}p(x,z|x^{\prime },z^{\prime };{\mathit{\theta }}_k){\alpha }_{t-1}(x^{\prime },z^{\prime };{\mathit{\theta }}_k)$        ${\beta }_t(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{x^{\prime },z^{\prime }}{\beta }_{t-1}(x^{\prime },z^{\prime };{\mathit{\theta }}_k)p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$    end for     $F(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,z;{\mathit{\theta }}_k)$     $V(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{t=0}^{T_{max}}{\gamma }^t{\beta }_t(x,z;{\mathit{\theta }}_k)$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$