FIG. 69.

KL-divergences between the data distribution p_data and the model p_θ. Data is drawn from a bimodal Gaus-sian distribution with unit variances peaked at ±∆ with ∆ = 2.0 and the model p_θ(x) is a Gaussian with mean zero and same variance as p_θ(x). (Top) p_data and p_θ for ∆ = 2. (Bottom) D_KL(p_data||p_θ) (Data-Model) and D_KL(p_θ||p_data) (Model-Data) as a function of ∆. Notice that D_KL(p_data||p_θ) is insensitive to placing weight in the model distribution in regions where p_data ≈ 0 whereas D_KL(p_θ||p_data) punishes this harshly.