Fig. 3.

Generalization error (red; Eq. 12), approximation error (blue; Eq. 9), and estimation error (green; Eq. 11) at L_x = 50, $N=30,000$, for various hidden-layer sizes L_h. Lines are analytical results; points are from numerical simulations (see SI Appendix, section 7.4 for details). Solid and dashed vertical lines are the minima of the generalization error from theory and simulations, respectively. Here, and in all figures except Fig. 4 D and E, both g_t and g_s are rectified linear functions [$g_t(u)=g_s(u)=\max (0,u)$]. In all figures except Fig. 6 we use ${\sigma }_t^2=0.1$ for the noise in the teacher circuit, and in all figures the hidden-layer size of the teacher network is fixed at $L_t=500$. Error bars represent the SD over 10 simulations.