Figure 1:

For the MNIST dataset, we validate that (a) the proposed approximate objectives ĝ(w) and $\tilde{g}(w)$ are close to the true objective g(w), and (b) training on the approximate objectives leads to similar predictions as training on the true objective. We plot the relative difference between the proposed approximations and the true augmented objective, in terms of difference in objective value (1a) and resulting test prediction disagreement (1b), using the non-augmented objective as a baseline. The 2nd-order approximation closely matches the true objective, particularly in terms of the resulting predictions. We observe that the accuracy of the approximations remains stable throughout training (1c). Full experiments are provided in Appendix E.