Figure I.

Probabilistic Kalman–Filter [80] Models of the Environment. The reward outcome r at time-step t is sampled from a normal distribution whose mean $v_t^s$ is specific to the current state. (A) For a particular state, depicted here, changes in the mean follow a random walk with normally distributed noise. (B) A general environmental factor affects multiple states. At each time-step t, a general factor g_t is sampled from a normal distribution whose mean is zero, and is then added to multiple state means ($v_t^s$). (C) Changes in reward follow an underlying momentum. The mean reward $v_t^s$ of a state is sampled from a normal distribution whose mean is the sum of the previous mean $v_{t-1}^s$ and the current momentum m_t. Changes in momentum follow a random walk.