Figure 1.

Value, decision information and free energy plots in a 5 × 5 gridworld with cardinal (Manhattan) actions $\mathcal{A}\hspace{0.17em}:\hspace{0.17em}\{\uparrow ,\leftarrow ,\downarrow ,\to \}$. The goal g = #12 is in the centre and is coloured yellow in the grid plots. The arrow lengths are proportional to the conditional probability π(a|s) in the indicated direction. The relevant prior, i.e. the joint state and action distribution marginalized over all transient states, $\widehat{\hspace{0.17em}p}(a;\pi )$ is shown in the yellow goal state. (a) The policy displayed is the optimal value policy ${\pi }_V={\arg \hspace{0.17em}\max \hspace{0.17em}}_{\pi }{V}_g^{\pi }(s)$ for all $s\in \mathcal{S}$. The heatmap and annotations show the negative optimal value function $-V_g^{{\pi }_V}(s)$ for each state. (b) The policy presented is optimal with respect to free energy, i.e ${\pi }_{\mathcal{F}}={\arg \hspace{0.17em}\min \hspace{0.17em}}_{\pi }{\mathcal{F}}_g^{\pi }(s;\beta )$ for all $s\in \mathcal{S}$. The heatmap and annotations show decision information ${{\mathrm{\Im }}_D}^{{\pi }_{\mathcal{F}}}(s)$ with β = 100. (c) The policy displayed is again ${\pi }_{\mathcal{F}}$ with β = 100. The heatmap and annotations show free energy ${\mathcal{F}}_g^{{\pi }_{\mathcal{F}}}(s;\beta )$. (d) Graph showing the numbering of states in the gridworld, the goal is coloured in green and the other colours indicate levels radiating from the centre.