Figure - PMC

Skip to main content

View full-text article in PMC

. Author manuscript; available in PMC: 2015 May 1.

Published in final edited form as: Artif Intell. 2014 Feb 24;210:78–122. doi: 10.1016/j.artint.2014.02.004

Figure 9.9 — Histogram of the difference between the dropout variance of $O_{i}^{l}$ and its approximate upperbound $W_{i}^{l} (W_{i}^{l})$ in a MNIST classifier network before and after training. Histograms are obtained by taking all non-input neurons and aggregating the results over 10 random input vectors. Note that at the beginnning of learning, with random small weights, $E (O_{i}^{l}) \approx W_{i}^{l} \approx 0.5$ , and thus $V a r (O_{i}^{l}) \approx 0$ whereas $W_{i}^{l} (1 - W_{i}^{l}) \approx 0.25$ .