Figure 6. Description of a model selecting action intensity.

(A) Details of the algorithm. The update rules for the variance parameters can be obtained by computing derivatives of $F$, giving ${\delta }_g^2-1/{\mathrm{\Sigma }}_g$ and ${\delta }_h^2/{\mathrm{\Sigma }}_h^2-1/{\mathrm{\Sigma }}_h,$ but to simplify these expressions, we scale them by ${\mathrm{\Sigma }}_g^2$ and ${\mathrm{\Sigma }}_h^2$, resulting in Equations 6.5. Such scaling does not change the value to which the variance parameters converge because ${\mathrm{\Sigma }}_g^2$ and ${\mathrm{\Sigma }}_h^2$ are positive. (B) Mapping of the algorithm on network architecture. Notation as in Figure 5B. This network is very similar to that shown in Figure 5B, but now the projection to output nuclei from the habit system is weighted by its precision $1/{\mathrm{\Sigma }}_h$ (to reflect the weighting factor in Equation 6.2), and also the rate of decay (or relaxation to baseline) in the output nuclei needs to depend on ${\mathrm{\Sigma }}_h$. One way to ensure that the prediction error in goal-directed system is scaled by ${\mathrm{\Sigma }}_g$ is to encode ${\mathrm{\Sigma }}_g$ in the rate of decay or leak of these prediction error neurons (Bogacz, 2017). Such decay is included as the last term in orange Equation 6.7 describing the dynamics of prediction error neurons. Prediction error evolving according to this equation converges to the value in orange Equation 6.3 (the value in equilibrium can be found by setting the left hand side of orange Equation 6.7 to 0, and solving for ${\delta }_g$). In Equation 6.7, total reward $R$ was replaced according to Equation 1.1 by the sum of instantaneous reward $r$, and available reward $v$ computed by the valuation system. (C) Dynamics of the model.