Algorithm 2:Value iteration algorithm.

Initialize V arbitrarily Repeat       $\Delta \leftarrow 0$       For each $s\in S$             $v\leftarrow V\left(s\right)$             $V\left(s\right)\leftarrow ma{x}_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma V\left(s^{\prime }\right)]$             $\Delta \leftarrow max(\Delta ,|v-V\left(s\right)\left|\right)$ until $\Delta <\theta$ (a small positive number) output a deterministic policy, $\pi$, such that $\pi \left(s\right)=argma{x}_a{\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)[r+\gamma V\left(s^{\prime }\right)]$