Archetypal landscapes for deep neural networks

Significance

Deep neural networks have reached impressive predictive capability for many challenging tasks, yet it remains unclear why they work. Training neural networks involves minimizing a complex, high-dimensional, nonconvex loss function, yet, empirically, it proves possible to produce useful models without rigorous global optimization. To provide insight into this observation, we analyze the structure of the loss-function landscape of deep neural networks and show that it features either a single funnel or low barriers between minima. Such landscapes are relatively easy to optimize and are qualitatively different from the energy landscape of a structural glass. More generally, our results demonstrate how the methodology developed for exploring molecular energy landscapes can be exploited to extend our understanding of machine learning.

Abstract

The predictive capabilities of deep neural networks (DNNs) continue to evolve to increasingly impressive levels. However, it is still unclear how training procedures for DNNs succeed in finding parameters that produce good results for such high-dimensional and nonconvex loss functions. In particular, we wish to understand why simple optimization schemes, such as stochastic gradient descent, do not end up trapped in local minima with high loss values that would not yield useful predictions. We explain the optimizability of DNNs by characterizing the local minima and transition states of the loss-function landscape (LFL) along with their connectivity. We show that the LFL of a DNN in the shallow network or data-abundant limit is funneled, and thus easy to optimize. Crucially, in the opposite low-data/deep limit, although the number of minima increases, the landscape is characterized by many minima with similar loss values separated by low barriers. This organization is different from the hierarchical landscapes of structural glass formers and explains why minimization procedures commonly employed by the machine-learning community can navigate the LFL successfully and reach low-lying solutions.

Introduction

The performance of deep neural networks (DNNs) continues to evolve to increasingly impressive levels (1), surpassing skilled human operators in many challenging tasks, such as gameplay. However, it is not clear why DNNs work so well. DNNs include a huge number of tunable parameters, which, together with the loss function, define the loss-function landscape (LFL). This landscape is high-dimensional and nonconvex, and, thus, one might expect global optimization to be a challenging task. However, empirically, even simple gradient descent with stochastic noise manages to avoid high-loss local minima and locate useful models. Resolving this optimizability puzzle will suggest strategies for designing more robust machine-learning architectures, as well as scalable optimization algorithms.

Previous investigations have developed simplified analytical models or performed numerical experiments to understand this optimizability puzzle. On the analytical front, ref. 2 analytically examines the LFL of a single hidden-layer neural network, and ref. 3 mapped DNNs to spin-glass models, which required a series of significant approximations. On the numerical front, earlier works focus on local geometric properties, such as the width of minima (4–7). Those insights led to modified optimization algorithms that bias the search toward solutions with favorable geometric properties (8, 9). Going beyond individual minima, more recent work focused on the connectivity of the landscape. In particular, we have analyzed the landscape for single hidden-layer neural networks using the disconnectivity graph (10, 11) formalism developed for molecular energy landscapes (12–14). Beyond single hidden-layer networks, ref. 15 visualized the LFL of DNNs by projecting to three dimensions and suggested that architectures such as skip-connections lead to smoother LFLs, while ref. 16 considered pairs of minima in DNNs and showed that the barrier along the minimum energy path is much lower than the linear path in weight space that connects the two minima. While these results are intriguing, they lack a global view of the LFL as a function of network depth and number of training data.

In this paper, we examine the number of minima and transition states, the connectivity of minima, and the relationship between minima width and generalization error, all as a function of the number of hidden layers and the amount of training data. Our approach is to benchmark small networks, where local minima and transition states can be characterized accurately using computational energy landscapes techniques (17). This perspective reveals the structure of the underlying solution space, in terms of the minimal barriers that any optimization procedure, including stochastic gradient descent, must navigate to pass between the catchment basins of different local minima (17, 18) and reach low-lying minima. Our most important conclusion is that the LFLs generally exhibit low barriers between local minima in terms of both local and global organization, which explains why DNNs can be optimized sufficiently to produce useful predictions.

Methods

DNNs.

We briefly define the DNNs considered in this paper (Fig. 1). We label layers using $l\in \left\{0,\dots ,H+1\right\}$, with $H$ being the number of hidden layers, $l=0$ the input layer, $l=H+1$ the output layer, and $n_l$ the number of nodes in layer $l$. The activations $a_i^{\left(l\right)}$ are obtained from the $z_j^{\left(l-1\right)}$ using:

$$a_i^{\left(l\right)}=\sum _{j=1}^{n_{l-1}}{w}_{ij}^{\left(l\right)}{z}_j^{\left(l-1\right)}+{\theta }_i^{\left(l\right)}=\sum _{j=0}^{n_{l-1}}{w}_{ij}^{\left(l\right)}{z}_j^{\left(l-1\right)}\hspace{0.17em}\text{for}\hspace{0.17em}1\le i\le n_l\hspace{0.17em},$$ [1]

where in the last step we have absorbed the bias weights into the link weight matrix by setting $w_{i0}^{\left(l\right)}={\theta }_i^{\left(l\right)}$ and $z_0^{\left(l-1\right)}=1$ for all $l$, giving these matrices the shape $n_l\times \left(n_{l-1}+1\right)$. The number of variables is then given by $v={\sum }_{l=1}^{H+1}{n}_l\times \left(n_{l-1}+1\right)$. The activations are turned into signals through $z_i^l={\varphi }_l\left(a_i^{\left(l\right)}\right)$ with activation functions ${\varphi }_l$. Here, we set ${\varphi }_l=\tanh$ for all $l$.

Fig. 1.

A DNN. Blue, input; red, output; green, hidden nodes.

We define the loss function (the analogue of the energy in a potential energy landscape [PEL]) from the softmax classification probabilities:

$$L=-\sum _d^{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}\ln \left(p_{c\left(d\right)}\right)+L_{\mathrm{r}\mathrm{e}\mathrm{g}}\hspace{0.17em}\hspace{0.17em}\text{with}\hspace{0.17em}\hspace{0.17em}{p}_c=e^{a_c^{\left(H+1\right)}}/\sum _{a=0}^3{e}^{a_a^{\left(H+1\right)}},$$ [2]

where $c\left(d\right)$ is the known outcome for data item $d$, $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is the number of training data, and $L_{\mathrm{r}\mathrm{e}\mathrm{g}}=\lambda \hspace{0.17em}\Vert {\mathbf{w}\Vert }^2$ is an L2 regularization term with regularization parameter $\lambda$, and $\mathbf{w}=\left(w^{\left(1\right)},\dots ,w^{\left(H+1\right)}\right)$ is the vector of all weights. Unless stated otherwise, the regularization used in this work is $\lambda =10^{-4}$.

Training Datasets.

We analyze the LFLs for three datasets: LJAT19 (SI Appendix), OPTDIG (19), and WINE (20), as summarized below.

The LJAT19 dataset was generated specifically for the assessment of DNN loss landscapes. It is based on the outcome predictions of geometry optimization of the ${\mathrm{L}\mathrm{J}\mathrm{A}\mathrm{T}}_3$ problem and was used in previous work (13, 21, 22). Details of its creation are reported in the SI Appendix. Readers primarily interested in LFLs need simply note that this is a fourfold classification problem with three inputs. Here, we only employ two of these inputs for training and testing to make the problem harder. This benchmark is appealing because we can generate arbitrary amounts of training and testing data, and it has practical importance in chemistry, where calculations of molecular configuration volumes and densities of states are of interest.

OPTDIG is a set of optical data for handwritten digits, with the target being digit recognition/classification. This resource is similar to the widely known MNIST (Modified National Institute of Standards and Technology) dataset, but with inputs of size 8 × 8. The WINE dataset is a list of red and white wines, with the target being the classification of their quality as “good,” “medium,” or “bad” based on 11 physicochemical tests (such as acidity, sulfates, alcohol, etc.). Both OPTDIG and WINE are “real-life” data and were obtained from the University of California, Irvine, Machine Learning Repository (23).

We primarily focus our studies on the LJAT19 dataset because we can generate as much data as is needed for benchmarking, which allows us to produce disconnectivity graphs with up to $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}=\mathrm{100,000}$ training data. In contrast, the WINE and OPTDIG datasets only have up to $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}=\mathrm{5,000}$ entries, so we use them to validate and confirm our principal conclusions. The computational cost of our methods is limited by the dimensionality of the LFL, and, hence, the datasets we study are a compromise between complexity and feasibility.

The performance of any point in the LFL, defined by a set of parameters $\mathbf{w}$, can be quantified using standard area under the curve (AUC) receiver operating characteristic metrics. AUC values range between 0 and 1, where random classification yields an AUC value of 0.5 and a perfect prediction yields 1. In addition, we visualize the prediction outcomes of points in the landscape of the LJAT19 dataset by coloring a representative subspace of the inputs according to the classification indices (Figs. 2 and 3).

Fig. 2.

Disconnectivity graphs for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in$ {100, 1,000, 2,000} training data (from top to bottom) for the DNNs with $H\in \left\{\mathrm{1,2,3}\right\}$ hidden layers (from left to right), labeled as “DATASET-#HL-#” on the top of each panel, where the two digits indicate, respectively, the number of hidden layers, $H$, and the amount of training data, $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$. Only the lowest 2,000 minima (or all, if fewer than 2,000 were identified) are shown, and the vertical scale has been adjusted to span the range of loss-function values within this set. Included as Insets below each disconnectivity graph are graphical visualizations of the performance of the global minimum (see SI Appendix, section S2 for details), as well as a plot of the training (horizontal axis) versus testing (vertical axis) loss values of all minima. It is apparent from these graphs that, in each case, the structure of the LFL is either funneled or comprises many minima with similar loss values connected by low barriers.

Fig. 3.

Continuation of Fig. 2 with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in \left\{\mathrm{10,000,100,000}\right\}$.

Exploring the LFL.

The LFL (12–14, 24) is the analogue of the PEL in molecular science, where the local minima correspond to isomers, and the pathways between them are atomic rearrangements. While the PEL is defined by the energy as a function of the atomic coordinates, the LFL is defined by the edge weights between nodes in training. We explore and visualize the LFL using geometry optimization tools developed for the analysis of PELs in molecular science; details are available in several reviews (17, 22, 25–27), and a brief overview is given here.

For each LFL, we first employ basin-hopping (BH) global optimization (28–30) to locate the putative global minimum. This technique is very effective in navigating the loss function and is not subject to the exponential slow-down suggested by spin-glass models (3). Local minima in themselves do not define a landscape; to determine the connectivity and barriers between solutions, we need to locate transition states, defined as stationary points with precisely one negative Hessian eigenvalue (31), which connect pairs of minima via steepest-descent paths (17, 18). Databases of minima and transition states were created and expanded using the doubly-nudged (32, 33) elastic band (34, 35) (DNEB) approach to identify points for accurate transition state refinement by hybrid eigenvector-following (36, 37). The connectivity between minima was established by calculating approximate steepest-descent paths for each transition state using a custom limited memory quasi-Newton Broyden (38), Fletcher (39), Goldfarb (40), Shanno (41) (LBFGS) routine. The resulting database of stationary points is the analogue of a kinetic transition network for a PEL (42–45), which is employed to analyze structure, dynamics, and thermodynamics in molecular systems. This computational energy landscapes framework has been applied to a wide variety of problems, and most of the standard procedures for expanding stationary point databases (17, 22, 25–27) carry over directly to the landscapes considered in the present contribution. Additional care is needed in tightening convergence criteria and in characterizing highly asymmetric pathways (46).

We note that ref. 16 employed nudged elastic band (NEB) interpolations to investigate barrier heights in neural network LFLs. However, in that study, the transition states were not refined accurately, and the connectivity between stationary points was not established. The discrete images of an NEB or DNEB interpolation can straddle barriers and intervening stationary points. To establish the true topology, it is important to refine transition states accurately using hybrid eigenvector-following (36, 37) and identify connected minima via approximate steepest-descent pathways. Gaps in the profile between two target minima can then be addressed using the missing connection algorithm (47) to identify the best candidates for additional double-ended searches, as in the present work.

We visualized the LFL using disconnectivity graphs (10, 11), which provide a faithful representation of the barrier heights, coarse-grained over a regular sequence of energy thresholds. The loss increases on the vertical axis, and the bottom of every branch corresponds to the value of the loss for a particular local training minimum. At each threshold, a superbasin analysis is performed to determine which groups of minima can interconvert without exceeding the threshold. Superbasins and the corresponding branches merge together as the loss threshold increases. We chose the position of the branches on the horizontal axis to avoid crossings and to produce a clear representation of the solution space.

Results

First, we studied the LFL of DNNs using the LJAT19 dataset. We characterized the landscapes as a function of training data ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in$ {100, 1,000, 2,000, 10,000, 100,000}) and number of hidden layers ($H=\mathrm{1,2,3}$). We chose $n_l=\mathrm{10,5,4}$ for each hidden layer $l$, which yielded an approximately equal number of weight variables ($v=\mathrm{74,69,72}$). The input and output dimensions were $n_0=2$ and $n_{H+1}=4$, respectively. In each case, we calculated databases of local minima and the transition states that connect them. We report the number of minima and transition states, visualize the LFL using disconnectivity graphs, and analyze the correlation between train error, test error, and minima width.

The above values of $v$ are small compared to many of the DNNs in typical applications, but they support loss functions with many local minima and allow us to identify clear trends where all of the optimization procedures can be converged accurately. We note that the lowest minimum discovered from BH remained the lowest in all cases on expanding the database, except for $H=3$ and $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}=100$. Our results therefore enabled us to focus on the fundamental organization of the landscape, independent of any optimization protocol.

After our study of the LJAT19 dataset, we confirmed that our key results also applied to the OPTDIG ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in${1,500, 5,000}, $H\in \left\{\mathrm{1,3}\right\}$) and WINE ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 1,500, $H\in \left\{\mathrm{1,3}\right\}$) datasets, where we simply report the disconnectivity graphs for comparison.

All of the results are labeled as “DATASET-#HL-#,” where the two digits indicate, respectively, the number of hidden layers, $H$, and the amount of training data, $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$.

Number of Minima.

Two key observations for LJAT19 are easily summarized (Table 1): The number of local minima, $n_{\mathrm{m}\mathrm{i}\mathrm{n}}$, and the number of transition states, $n_{\mathrm{t}\mathrm{s}}$, decreased rapidly with increasing $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$. All of the values reported in Table 1 are lower bounds; we expect the largest databases for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}=100$ with $H=\mathrm{2,3}$ to be the least complete, but the low-lying regions of the LFL should be well sampled. The ranges of loss values of all located minima followed the same pattern (note the scale bars in Figs. 2 and 3). The number of minima for given $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ also increased rapidly with $H$. For $H=1$, which has the fewest stationary points, we also compared the landscapes for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in${250, 500, 3,000, 4,000, 5,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000} to confirm the observed trends. Note that the disconnectivity graphs in Figs. 2 and 3 show only the lowest 2,000 minima for clarity.

Table 1.

Number of local minima and transition states for DNNs with varying number of hidden layers, $H$, and training data, $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$

	$H=1$		$H=2$		$H=3$
$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$	$n_{\mathrm{m}\mathrm{i}\mathrm{n}}$	$n_{\mathrm{t}\mathrm{s}}$	$n_{\mathrm{m}\mathrm{i}\mathrm{n}}$	$n_{\mathrm{t}\mathrm{s}}$	$n_{\mathrm{m}\mathrm{i}\mathrm{n}}$	$n_{\mathrm{t}\mathrm{s}}$
100	649	10,426	299,484	508,471	585,730	1,126,877
1,000	7	33	13,336	41,837	85,150	263,615
2,000	5	21	487	1,583	3,027	35,363
10,000	7	43	49	393	150	804
100,000	5	11	33	200	23	96

The number of local minima was further examined for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in${100, 1,000} and networks with $H=1$ containing $n_1\in \left\{\mathrm{3,4,5,6,10,15}\right\}$ nodes. We found five or fewer minima in each case, with AUC values for both testing and training data between 0.79 and 0.80. Hence, it does not require many hidden nodes to reduce the solution space to a few minima with similar accuracy when we provide enough training data.

Connectivity of Minima.

Based on the disconnectivity graphs for the LJAT19 dataset (Figs. 2 and 3), it is evident that for sufficiently large $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$, the loss function has a relatively simple structure corresponding to a funneled landscape, where finding the global minimum is straightforward (17). This “palm tree” (11) organization is especially clear in panels 1HL-100, 2HL-2,000, and 3HL-10,000 (and those with higher $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$): Pathways from all of the minima with higher loss to the global minimum encountered relatively low downhill barriers. The structure for small $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ with $H=2$ or $H=3$ is rather different: We see many competing low-lying minima with similar loss values. The observation of low barriers between the minima agrees with previous calculations (16). The PELs visualized for structural glasses support many amorphous structures, corresponding to minima with similar potential energy (48–52). However, these minima are separated by large barriers compared to the relevant thermal energy, producing a landscape (11) with a hierarchical structure, where metabasins can be defined in terms of cage-breaking rearrangements (48–52). In contrast, the barriers between the low-lying minima for 2HL-100 and 3HL-100 were relatively low; the structure at the bottom of these landscapes is more reminiscent of a mangrove swamp, and we do not see the hierarchical organization characteristic of structural glasses (48–52). Instead, the overall funneled structure probably agrees with the conclusion that local optimization is likely to be guided downhill toward low-lying minima (53).

Visualizations of the predictions of the DNN global minimum are included in Figs. 2 and 3; a description is provided below the figure with further details in SI Appendix. We see how the best solution converges to the same pattern as sufficient training data are supplied. The best AUC value here is around 0.8 and corresponds to convergence of the relative probabilities for predictions in the subspace of one missing input variable. Note that the class index colored in red is entirely absent in these visualizations, reflecting the difficulty of distinguishing the red from the gray class index in the ${\mathrm{L}\mathrm{J}\mathrm{A}\mathrm{T}}_3$ problem without knowledge of all three inputs.

The same optimal solution was obtained for $H\in \left\{\mathrm{1,2,3}\right\}$, but more training data are required for larger $H$ (even though the number of variable weights that are optimized is very similar in each case). The enhanced expressibility of the DNNs with higher $H$ may be reflected in the increased complexity of the patterns in the visualizations (see especially $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in${100, 1,000}).

In Figs. 2 and 3, we also plot the training versus testing loss of all minima. The graphs for $H\in \left\{\mathrm{2,3}\right\}$ reveal the following trends: There exists a weak anticorrelation between the train-loss and test-loss values of low-lying minima in the case of little training data ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\le$1,000). This result suggests overfitting: Minima with loss values lower than the optimal value obtained for the large training-data limit must gain their advantage in a way that does not generalize well to testing data. In this regime, the corresponding DNN models likely yield no good learning capability when trained, due to the lack of train-test loss correlation and the resulting poor generalizability of the model, despite the fact that it can easily be optimized, and low-lying minima can easily be identified. For a medium amount of training data ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 2,000), there is no clear correlation in the graphs, while for large amounts of training data ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\ge$10,000), there is a positive correlation. Interestingly, the lack of bare train-test loss correlation for medium ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 2,000)—but not small ($N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}=100$)—amounts of training data can be overcome by considering the correlation with the geometrical properties of the minimum, as discussed in the next section.

Relating Test Error and Basin Geometry.

To further investigate the correlation of train and test loss, $L_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and $L_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, as well as how the loss of each minimum correlates with local curvatures, we performed a fit of the following two functions to the data:

$$L_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(1\right)}\left(L_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\right)=a_1+b_1{L}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$$ , [3]

$$L_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(2\right)}\left(L_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},S\right)=a_2+b_2{L}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}+c_{2}S,$$ [4]

where $S$ is the log product of all Hessian eigenvalues of the respective minimum and is defined analogously to the vibrational entropy in molecular systems. The optimal fit parameters and values of the adjusted coefficient of determination, $r^2$, are reported in Table 2 (where results for $H=1$ and $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\ge$1,000 are omitted because $n_{min}\le 7$, so that proper statistical analysis is impossible). First, the trend of negative to positive correlation between $L_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and $L_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ with increasing $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is confirmed by the values of $b_1$ and $b_2$. Second, while adding the term proportional to $S$ seems to be irrelevant in the case of $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\ge$10,000 (as $c_2$ is small, $b_2$ is similar to $b_1$, and $r^2$ changes only slightly), the results are very different for small $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$: The parameter $c_2$ increases by up to three orders of magnitude with decreasing $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$; the optimal values of $b_1$ and $b_2$ differ drastically; and the $r^2$ value increases significantly between the two fits. The change between the fits is most pronounced for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in${1,000, 2,000}. We even find that for $H=2$ and $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 2,000, the correlation between train and test loss changes from negative ($b_1<0$) to positive ($b_2>0$), and $r^2$ increases from 4.7 × 10⁻³ to 4.0 × 10⁻¹.

Table 2.

Fitting parameters for correlation between training and testing loss of minima

$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$	$b_1$	$r^2$	$b_2$	$c_2$	$r^2$
$H=1$
100	$-7.8\pm 1.1$	0.068	$-4.7\pm 1.1$	$8.39\pm 0.88$	0.18
$H=2$
100	$-5.5\pm 0.027$	0.097	$-4.858\pm 0.027$	$5.57\pm 0.05$	0.12
1,000	$-2.529\pm 0.038$	0.25	$-0.308\pm 0.033$	$0.4495\pm 0.0038$	0.63
2,000	$-0.155\pm 0.085$	0.0047	$0.231\pm 0.07$	$0.151\pm 0.0085$	0.4
10,000	$1.07\pm 0.1$	0.68	$0.9\pm 0.13$	$0.035\pm 0.016$	0.71
100,000	$0.94\pm 0.057$	0.89	$0.944\pm 0.058$	$0.0034\pm 0.0047$	0.89
$H=3$
100	$-4.05\pm 0.017$	0.09	$-4.678\pm 0.016$	$6.736\pm 0.03$	0.16
1,000	$-2.998\pm 0.011$	0.48	$-1.083\pm 0.013$	$0.5131\pm 0.0025$	0.65
2,000	$-1.059\pm 0.045$	0.15	$-0.063\pm 0.035$	$0.2122\pm 0.0037$	0.6
10,000	$0.673\pm 0.075$	0.35	$0.672\pm 0.071$	$0.0267\pm 0.0069$	0.41
100,000	$0.945\pm 0.023$	0.99	$0.939\pm 0.025$	$-0.0019\pm 0.0027$	0.99

This result suggests that knowledge of the curvature of a minimum (encoded in the entropy parameter $S$) may enhance the prediction of the performance of training LFL minima on testing data. However, we note that information on the curvature is only helpful when the model has the potential to overfit the data, hence explaining why the additional fitting parameter delivers little benefit for $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\ge$10,000. Moreover, we note that $S$ summarizes the information contained in the eigenvalues of all of the $v=\mathrm{74,69,72}$ (for $H=\mathrm{1,2,3}$) dimensions of the Hessian matrix. In future work, it would be interesting to investigate the distribution of those eigenvalues (55) that play a key role in determining the structure of the heat capacity analogue for the LFL (13, 14).

Performance of Minima.

The performance of all of the LJAT19 training minima can be analyzed for the accompanying test data in terms of the loss function and AUC. For consistency, we summarize results for training and testing datasets of equal size (Table 3); our general conclusions are not sensitive to which test data are chosen. The average barrier heights for individual transition states and for the lowest barrier pathway to the global minimum are collected in SI Appendix, Tables S1 and S2.

Table 3.

Average loss function and AUC values for train and test data of all minima found in training for databases listed in Table 1

	$H=1$				$H=2$				$H=3$
	Train		Test		Train		Test		Train		Test
$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$	Loss	AUC	Loss	AUC	Loss	AUC	Loss	AUC	Loss	AUC	Loss	AUC
100	0.297	0.953	1.453	0.547	0.166	0.995	2.344	0.557	0.121	0.994	2.424	0.564
1,000	0.519	0.810	0.552	0.796	0.511	0.823	0.572	0.787	0.502	0.829	0.596	0.779
2,000	0.539	0.806	0.548	0.795	0.534	0.809	0.550	0.792	0.531	0.812	0.556	0.788
10,000	0.546	0.801	0.559	0.801	0.542	0.802	0.557	0.801	0.542	0.803	0.557	0.801
100,000	0.547	0.797	0.551	0.796	0.543	0.798	0.548	0.797	0.543	0.798	0.548	0.797

Comparison with Other Datasets.

To check that our results are transferable to very different prediction problems, we analyzed the LFL for the OPTDIG and WINE datasets and present their disconnectivity graphs in Fig. 4. The number of weight variables for OPTDIG with $n_0=64$, $n_{H+1}=10$, $H=\mathrm{1,3}$, and $n_l=\mathrm{5,4}$ are $v=\mathrm{385,350}$ and for WINE with $n_0=11$, $n_{H+1}=3$, $H=\mathrm{1,3}$, and $n_l=\mathrm{5,3}$ are $v=\mathrm{78,72}$ (where $l$ are hidden layers). The structure of these landscapes appears to be qualitatively similar to LJAT19. There are no large barriers separating low-lying local minima, which would hinder efficient optimization.

Fig. 4.

Disconnectivity graphs for the training datasets OPTDIG (with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\in$ {1,500, 5,000} and $H\in \left\{\mathrm{1,3}\right\}$) and WINE (with $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ = 1,500 and $H\in \left\{\mathrm{1,3}\right\}$). Only the lowest 2,000 minima (or all of them if fewer than 2,000 were found) are shown. The vertical scale is adjusted to span the range of loss-function values within this set.

Conclusions

The main result of this study is that the LFLs of DNNs have archetypal structures, depending on the network depth and the amount of data employed. The single-layer networks generally feature single-funneled landscapes, and in the high-$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ regime, the number of minima is very low (fewer than 10 in some cases). Such networks are easy to optimize.

This situation changes significantly when we move to a multilayer system (while keeping the number of variables approximately fixed), where the number of minima grows rapidly with increased depth. In the high-$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ limit, the LFL remains funneled with low barriers, and it is still easy to locate low-lying solutions. The landscapes are rather different when few data points are provided. There exist many minima with similar loss values, but they are connected by low barriers. This organization suggests that a reasonable optimization procedure will still reach low-lying minima, since we do not expect to see the slow dynamics characteristic of structural glasses, where the PEL has a hierarchical organization (48–52). The structure that our analysis reveals could be interpreted as a somewhat perturbed convex function, which corresponds to a funneled landscape (17).

We stress that these results are relevant for the optimizability of DNNs. One might expect that training of DNNs, which involves the optimization of complex, high-dimensional, nonconvex loss functions, could suffer from trapping in local minima. However, our study shows that the absence of high barriers between minima allows techniques such as stochastic gradient descent to reach low-lying solutions in the LFL.

Unsurprisingly, when few data are supplied, the large number of minima in the training LFL of DNNs spread across a wide range of loss values, and the same is true for testing data. In this limit, the system is prone to overfitting, which is manifested in anticorrelation between training and testing loss values. We find a strong correlation between the local curvatures (represented by the Hessian eigenvalues and the corresponding entropy) and the generalizability of minima in the deep medium-$N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ regime. Hence, we conclude that entropic contributions to the optimization dynamics may guide the system to minima that generalize well.

Extending our study to larger networks (3) is not straightforward, because the expected exponential growth in the number of stationary points with the number of variables (55, 56) makes enumeration impossible. However, the trends for small networks observed in this contribution, especially the arrangement of low-lying minima separated by small barriers, appear to be consistent with theory based on spin-glass models (3).

Previous studies by Baity-Jesi et al. (57) of DNN training dynamics were used to infer properties of the LFL, whereas our work directly interrogates the organization of the landscape. Our results are consistent with ref. 57, confirming that the dynamics of DNNs, unlike structural glasses, do not suffer from high loss barriers. However, our “microscopic” view of the energy landscape revises the hypothesis of flat basins and, instead, would attribute the observed slow dynamics to the proliferation of multiple shallow minima.

There still remain many interesting open questions concerning the properties of DNNs, and the various theoretical and computational tools developed to understand and explore PELs in chemical physics (17, 22, 25–27) could provide new insights. For example, the number of local minima and transition states in an atomic system are expected to grow exponentially with the number of degrees of freedom, $D$, as $\exp \left(\gamma D\right)$ and $D\exp \left(\gamma D\right)$, respectively (55, 56). The exponential factor $\gamma$ is larger for short-range interatomic potentials (58–61), because more local minima are possible when the effect of distant atoms is smaller. The number of stationary points may increase with the number of hidden layers for the same reason: Nodes in nonadjacent layers are not directly connected, so weights that correspond to local stability may combine almost combinatorially. A more quantitative analysis of this effect has been presented for atomic systems using catastrophe theory (60), and the same ideas should apply to the properties of multilayer neural networks. We will present further analysis of this effect elsewhere.

Data Availability.

All of the energy-landscapes exploration software, documentation, examples, and further resources are available for download under the Gnu Public License from the Cambridge Landscape Database (62). The input data and program instructions used to produce the results for the present work have been made available on the Apollo repository, https://doi.org/10.17863/CAM.55772 (63).