Figure 3:

(a) Eigenvalues of $B^{T}B$ as a function of the expansion ratio ($1/\lambda$). The shaded region shows the standard deviation of the eigenvalues. (b) Backward-forward path alignment between $W$ and ${(RB)}^T$ as a function of the expansion ratio ($1/\lambda$), where we randomly sampled $W$ and set $R^T=BW$ (expected to hold after the effect of the weight initialization has fully decayed). This simplification is consistent with the path alignment after training observed in simulations.