Fig. 7.

This figure illustrates the functional anatomy implied by a simple message passing scheme based on variational Bayes, and generative models based upon Markov decision processes; see (Friston et al., 2014) for details. It includes the following variables: observations (o_t), expected states of the world (${\stackrel{⌢}{s}}_t$), action (a_t), expected action sequences or policies ($\stackrel{⌢}{\pi }$) and their precision ($\stackrel{⌢}{\gamma }$). Q represents the quality of a policy scored in terms of its (epistemic) value or expected free energy. The equations corresponds to (variational) Bayesian updates, where A and B are probability transition matrices mapping hidden states to observations and hidden states to hidden states under different actions respectively. σ is a softmax function. Left panel: here, we have associated the Bayesian updates of hidden states of the world with perception, control states (policies) with action selection and expected precision with incentives salience. Right panel: this shows the results of a simulation in terms of simulated dopamine discharges, of the kind that is usually associated with reward prediction errors (Schultz et al., 1997), but which can also be modelled under an Active Inference scheme. The key thing to note is that the responses to an informative cue (or conditioned stimulus CS – blue) pre-empt subsequent responses to the reward (or unconditioned stimulus US – red). In this simulation, the agent was shown a cue that resolved uncertainty (i.e., had epistemic value) about where to find a reward in a simple T-maze (upper right panel). In this context, dopaminergic responses appear to transfer from the US when it is encountered without (middle right panel) and with (lower right panel) a preceding CS.