Fig. 1.

Top graphs are associated to functions. Each Bottom diagram (Insets) depicts the ideal network approximating the function above. (Inset A) A shallow universal network in 8 variables and $N$ units approximates a generic function of 8 variables $f\left(x_1,\dots ,x_8\right)$. (Inset B) A hierarchical network at the bottom in $n=8$ variables, which approximates well functions of the form $f\left(x_1,\dots ,x_8\right)=h_3\left(h_{21}\left(h_{11}\left(x_1,x_2\right),h_{12}\left(x_3,x_4\right)\right),h_{22}\left(h_{13}\left(x_5,x_6\right),h_{14}\left(x_7,x_8\right)\right)\right)$ as represented by the binary graph above. In the approximating network each of the $n-1$ nodes in the graph of the function corresponds to a set of ReLU units. Similar to the shallow network, a hierarchical network is universal; that is, it can approximate any continuous function; the text discusses how it can approximate a subclass of compositional functions exponentially better than a shallow network. Redrawn from ref. 23.