Figure 2:

Discrete and continuous generative models. (a) This graphic shows a Markov decision process in factor graph form (Loeliger et al., 2007). Blue squares indicate probability distributions (factors of the generative distribution). Below, these factors are expressed in terms of probability matrices. “Cat” denotes a categorical distribution. The implicit mean-field factorization of the approximate distribution, $Q$, is shown. (b) This shows the equivalent structure for a continuous state-space model. Heuristically, we can think of these (generative) models as an algorithm that generates data. This would involve drawing variables from their prior distributions (e.g., $\pi ,s_1,\tilde{v})$, and using these variables to sample dependent variables (e.g., $s_2,\tilde{x})$ from conditional distributions. Finally, the data ($\tilde{o},\tilde{y})$ can then be generated from the likelihood. In short, a generative model is just a probabilistic specification of how data are caused. Note that the prior mean for $\tilde{v}$ is derived from the outcomes of the discrete model. See the main text and Figure 1 for an explanation of the variables.