Fig 1. Model architecture and visualization of Fisher information.

(A) Preparation of dataset and general model architecture: circles of different radii were sliced out of CIFAR-10 images and binarized. The pixels corresponded to the visible units of a standard RBM with one hidden layer. (B) Fisher information for an exemplary over-parameterized RBM initialized with n_v = 13 visible units and n_h = 70 hidden units. The FIM is sparse, indicating many irrelevant parameters. (C) Importance of each parameter, as summarized by the value for each parameter in the vector of steepest curvature in the FIM (i.e. the leading eigenvector). The rectangular plot shows the normalized importance for all weights w_ij connecting visible units (v_i; vertical axis) and hidden units (h_j; horizontal axis). The importance for biases for the hidden (b^h) and visible (b^v) units is shown above (horizontal) and to the right (vertical), respectively. (D) Normalized parameter importance directly estimated from the diagonal entries of the FIM. (FIM = Fisher Information Matrix).