Figure 8. Conditions on feedback synapses for effective learning.

(A) Diagram of a one hidden layer network trained in B, with 80% of feedback weights set to zero. The remaining feedback weights ${\displaystyle {\mathit{Y}}^{\mathrm{\prime }}}$ were multiplied by five in order to maintain a similar overall magnitude of feedback signals. (B) Plot of test error across 60 epochs for our standard one hidden layer network (gray) and a network with sparse feedback weights (red). Sparse feedback weights resulted in improved learning performance compared to fully connected feedback weights. Right: Spreads (min – max) of the results of repeated weight tests ($n=20$) after 60 epochs for each of the networks. Percentages indicate mean final test errors for each network (two-tailed t-test, regular vs. sparse: $t_{38}=16.43$, ${\displaystyle p=7.4\times {10}^{-19}}$). (C) Diagram of a one hidden layer network trained in D, with feedback weights that are symmetric to feedforward weights ${\displaystyle {\mathit{W}}^1}$, and symmetric but with added noise. Noise added to feedback weights is drawn from a normal distribution with variance $\sigma =0.05$. (D) Plot of test error across 60 epochs of our standard one hidden layer network (gray), a network with symmetric weights (red), and a network with symmetric weights with added noise (blue). Symmetric weights result in improved learning performance compared to random feedback weights, but adding noise to symmetric weights results in impaired learning. Right: Spreads (min – max) of the results of repeated weight tests ($n=20$) after 60 epochs for each of the networks. Percentages indicate means (two-tailed t-test, random vs. symmetric: $t_{38}=18.46$, ${\displaystyle p=4.3\times {10}^{-20}}$; random vs. symmetric with noise: $t_{38}=-71.54$, ${\displaystyle p=1.2\times {10}^{-41}}$; symmetric vs. symmetric with noise: $t_{38}=-80.35$, ${\displaystyle p=1.5\times {10}^{-43}}$, Bonferroni correction for multiple comparisons).

Figure 8—figure supplement 1. Importance of weight magnitudes for learning with sparse weights.

Plot of test error across 20 epochs of training on MNIST of a one hidden layer network, with regular feedback weights (gray), sparse feedback weights that were amplified (red), and sparse feedback weights that were not amplified (blue). The network with amplified sparse feedback weights is the same as in Figure 8A and B, where feedback weights were multiplied by a factor of 5. While sparse feedback weights that were amplified led to improved training performance, sparse weights without amplification impaired the network’s learning ability. Right: Spreads (min – max) of the results of repeated weight tests ($n=20$) after 20 epochs for each of the networks. Percentages indicate means (two-tailed t-test, regular vs. sparse, amplified: $t_{38}=44.96$, ${\displaystyle p=4.4\times {10}^{-34}}$; regular vs. sparse, not amplified: $t_{38}=-51.30$, ${\displaystyle p=3.2\times {10}^{-36}}$; sparse, amplified vs. sparse, not amplified: $t_{38}=-100.73$, ${\displaystyle p=2.8\times {10}^{-47}}$, Bonferroni correction for multiple comparisons).