Fig 4. Parameter dependence of tempering effectiveness.
(a, c) Visible and label layer activity of an LIF network trained on the MNIST dataset, with (b,d) showing the corresponding mode duration distributions (the active mode corresponds to the image class and is determined by the most active label neuron). The network with oscillatory background (red) moves quickly between modes, with correspondingly short mode durations, whereas the network with constant background activity (blue) switches to the “6” mode after two samples and remains there until the last of the 103 collected samples. (e) Kullback-Leibler divergence (DKL) between the distribution of sampled modes and the uniform distribution. The sampled distribution quickly becomes significantly more uniform for the oscillatory (red) compared to the constant (blue) background. (f) Indirect sampling likelihood (ISL) as a measure of image quality and diversity for the two background settings and, for orientation, for the optimal sampling (OPT, orange) and the product of marginals (POM, gray). Under this measure, the averaged MNIST images described by the POM are more similar to the entire dataset than the near-unimodal distribution generated under constant background at T = 1. Similarly, the network with oscillatory background needs several samples to produce a distribution that is diverse enough to overtake the POM. The mean (solid lines) and standard deviation (shades) over 10 runs of 103 samples are plotted. (g) Average mode duration for different oscillation parameters: The peak background rate νmax represents the most critical parameter and needs to be high enough to enable good mixing. The minimum background rate νmin and the oscillation frequency fosc are less important. (h) Same as (g) for the ISL values. The image quality remains consistently high across a wide range of parameter configurations. The data used for (a-f) corresponds to the simulations marked by the red and blue crosses, respectively. Values represent averages over 10 runs of 104 samples.