Homophily modulates double descent generalization in graph convolution networks

Significance

Graph neural networks (GNNs) have been applied with great success across science and engineering, but we do not understand why they work so well. Motivated by experimental evidence of a rich phase diagram of generalization behaviors, we analyzed simple GNNs on a community graph model and derived precise expressions for generalization error as a function of noise in the graph, noise in the features, proportion of labeled data, and the nature of interactions in the graph. Computer experiments show that the analysis also qualitatively explains large “production-scale” networks and can thus be used to improve performance and guide hyperparameter tuning. This is significant both for the downstream science and for the theory of deep learning on graphs.

Abstract

Graph neural networks (GNNs) excel in modeling relational data such as biological, social, and transportation networks, but the underpinnings of their success are not well understood. Traditional complexity measures from statistical learning theory fail to account for observed phenomena like the double descent or the impact of relational semantics on generalization error. Motivated by experimental observations of “transductive” double descent in key networks and datasets, we use analytical tools from statistical physics and random matrix theory to precisely characterize generalization in simple graph convolution networks on the contextual stochastic block model. Our results illuminate the nuances of learning on homophilic versus heterophilic data and predict double descent whose existence in GNNs has been questioned by recent work. We show how risk is shaped by the interplay between the graph noise, feature noise, and the number of training labels. Our findings apply beyond stylized models, capturing qualitative trends in real-world GNNs and datasets. As a case in point, we use our analytic insights to improve performance of state-of-the-art graph convolution networks on heterophilic datasets.

Graph neural networks (GNNs) recently achieved impressive results on problems as diverse as weather forecasting (1), predicting forces in granular materials (2), or understanding biological molecules (3–5). They have become the de facto machine learning model for datasets with relational information such as interactions in protein graphs or friendships in a social network (6–9). These remarkable successes triggered a wave of research on better, more expressive GNN architectures for diverse tasks, yet there is little theoretical work that studies why and how these networks achieve strong performance.

In this paper, we study generalization in graph neural networks for transductive (semi-supervised) node classification: given a graph $G=(V,E)$, node features $\mathit{x}:V\to {\mathbb{R}}^F$, and labels $y:V_{\text{train}}\to \{-1,1\}$ for a “training” subset of nodes $V_{\text{train}}\subset V$, we want to learn a rule which assigns labels to nodes in $V_{\text{test}}=V\setminus V_{\text{train}}$. This setting exhibits a richer generalization phenomenology than the usual supervised learning: in addition to the quality and dimensionality of features associated with data, the generalization error is affected by the quality of relational information (are there missing or spurious edges?), the proportion of observed labels $|V_{\text{train}}|/|V|$, and the specifics of interaction between the graph and the features. Additional complexity arises because links in different graphs encode qualitatively distinct semantics. Interactions between proteins are heterophilic; friendships in social networks are homophilic (10). They result in graphs with different structural statistics, which in turn modulate interactions between the graphs and the features (11, 12). Whether and how these factors influence learning and generalization is currently not understood. Outstanding questions include the role of overparameterization and the differences in performance on graphs with different levels of homophily or heterophily. Despite much work showing that in overparameterized models the traditional bias–variance tradeoff is replaced by the so-called double descent, there have been no reports nor analyses of double descent in transductive graph learning. Recent work speculates that this is due to implicit regularization (13).

Toward addressing this gap, we derive a precise characterization of generalization in simple graph convolution networks (GCNs) in semi-supervised^* node classification on random community graphs. We motivate this setting by first presenting a sequence of experimental observations that point to universal behaviors in a variety of GNNs on a variety of domains.

In particular, we argue that in the transductive setting a natural way to “diagnose” double descent is by varying the number of labels available for training (Section 1). We then design experiments that show that double descent is in fact ubiquitous in GNNs: There is often a counterintuitive regime where more training data hurts generalization (14). Understanding this regime has important implications for the (often costly) label collection and questions of observability of complex systems (15). While earlier work reports similar behavior in standard supervised learning, our transductive version demonstrates it directly (14, 16). On the other hand, we indeed find that for many combinations of relational datasets and GNNs, double descent is mitigated by implicit or explicit regularization. Interestingly, the risk curves are affected not only by the properties of the models and data (14), but also by the level of homophily or heterophily in the graphs.

Motivated by these findings we then present our main theoretical result: a precise analysis of generalization on the contextual stochastic block model (CSBM) with a simple GCN. We combine tools from statistical physics and random matrix theory and derive generalization curves either in closed form or as solutions to tractable low-dimensional optimization problems. To carry out our theoretical analysis, we formulate a universality conjecture which states that in the limit of large graphs, the risks in GCNs with polynomial filters do not change if we replace random binary adjacency matrices with random Gaussian matrices. We empirically verify the validity of this conjecture in a variety of settings; we think it may serve as a starting point for future analyses of deep GNNs.

These theoretical results allow us to effectively explore a range of questions: For example, in Section 3 we show that double descent also appears when we fix the (relative) number of observed labels, and vary relative model complexity (Fig. 5). This setting is close but not identical to the usual supervised double descent (17). We also explain why self-loops improve performance of GNNs on homophilic (18) but not heterophilic (11, 12) graphs, as empirically established in a number of papers, but also that negative self-loops benefit learning on heterophilic graphs (19, 20). We then go back to experiment and show that building negative self-loop filters into state-of-the-art GCNs can further improve their performance on heterophilic graphs. This can be seen as a theoretical GCN counterpart of recent observations in the message passing literature (19, 20) and an explicit connection with heterophily for architectures such as GraphSAGE which can implement analogous logic (9).

Fig. 5.

Test risk as a function of relative model complexity $\alpha ={\gamma }^{-1}$: different levels of homophily lead to distinct types of double descent in CSBM. Plots from Left to Right (with increasing $\lambda$) show curves for graphs of decreasing randomness. Varying model complexity in GNNs yields nonmonotonic curves similar to those in the earlier studies of double descent studies in supervised (inductive) learning. Note that the overall shape of the curve is strongly modulated by the degree of homophily in the graph.

Existing studies of generalization in graph neural networks rely on complexity measures like the VC-dimension or Rademacher complexity but they result in vacuous bounds which do not explain the observed phenomena (21–23). Further, they only indirectly address the interaction between the graph and the features. This interaction, however, is of key importance: an Erdős–Renyi graph is not likely to be of much use in learning with a graph neural network. In reality both the graph and the features contain information about the labels; learning should exploit the complementarity of these two views.

Instead of applying the “big hammers” of statistical learning theory, we adopt a statistical mechanics approach and study performance of simple graph convolution networks on the CSBM (24). We derive precise expressions for the learning curves which exhibit a rich phenomenology.

The two ways to think about generalization, statistical learning theory and statistical mechanics, have been contrasted already in the late 1980s and the early 1990s. Statistical mechanics of learning, developed at that time by Gardner, Opper, Sejnowski, Sompolinsky, Tishby, Vallet, Watkin, and many others—an excellent account is given in the review paper by Watkin et al. (25)—must make more assumptions about the data and the space of admissible functions, but it gives results that are more precise and more readily applied to the practice of machine learning.

These dichotomies have been revisited recently in the context of deep learning and highly overparameterized models by Martin and Mahoney (26), in reaction to Zhang et al.’s thought provoking “Understanding deep learning requires rethinking generalization” (27) which shows, among other things, that modern deep neural networks easily fit completely random labels. Martin and Mahoney explain that such seemingly surprising new behaviors can be effectively understood within the statistical mechanics paradigm by identifying the right order parameters and related phase diagrams. We explore these connections further in Section 4—Discussion.

Outline.

We begin by describing the motivational experimental findings in Section 1. We identify the key trends to explain, such as the dependence of double descent generalization on the level of noise in features and graphs. In Section 2, we introduce our analytical model: a simple GCN on the contextual stochastic block model. Section 3 then explores the implications of some of the analytical findings about self-loops and heterophily on the design of state-of-the-art GCNs. We follow this by a discussion of our results in the context of related work in Section 4. In Section 5, we explain the analogies between GCNs and spin glasses which allow us to apply analysis methods from statistical physics and random matrix theory. We follow with a few concluding comments in Section 6.

1. Motivation: Empirical Results

Given an $N$-vertex graph $G=(V,E)$ with an adjacency matrix $\mathit{A}\in {\{0,1\}}^{N\times N}$ and features $\mathit{X}\in {\mathbb{R}}^{N\times F}$, a node classification GNN is a function $(\mathit{A},\mathit{X})\mapsto \mathit{h}(\mathit{w};\mathit{A},\mathit{X})$ insensitive to vertex ordering: for any node permutation $\pi$, $\mathit{h}(\mathit{w};\pi \mathit{A}{\pi }^{⊺},\pi \mathit{X})=\pi \mathit{h}(\mathit{w};\mathit{A},\mathit{X})$. We are interested in the behavior of train and test risk,

$$\begin{array}{c}\hfill R_N(S)=\frac{1}{|S|}{\displaystyle \sum _{i\in S}}\ell \left({\mathit{y}}_i,{\mathit{h}}_i({\mathit{w}}^{\ast };\mathit{A},\mathit{X})\right),\end{array}$$ [1]

with $S\in \{V_{\text{train}},V_{\text{test}}\}$ and $\ell (·,·)$ a loss metric such as the mean-squared error (MSE) or the cross-entropy. The optimal network parameters ${\mathit{w}}^{\ast }$ are obtained by minimizing the regularized loss

$$\begin{array}{c}\hfill L_N(\mathit{w})=\frac{1}{|V_{\text{train}}|}{\displaystyle \sum _{i\in V_{\text{train}}}}\ell \left({\mathit{y}}_i,{\mathit{h}}_i(\mathit{w};\mathit{A},\mathit{X})\right)+r_N(\mathit{w}),\end{array}$$ [2]

where $r_N(\mathit{w})$ is a regularizer.

A. Is Double Descent Absent in GNNs?

We start by investigating the lack of reports of double descent in transductive learning on graphs. Double descent is a scaling of test risk with model complexity which is rather different from the textbook bias–variance tradeoff (16, 28). Up to the interpolation point, where the model has sufficient capacity to fit the training data without error, things behave as usual, with the test risk first decreasing together with the bias and then increasing with the variance due to overfitting. But increasing complexity beyond the interpolation point—into an overparameterized region characteristic for modern deep learning—may make the test risk decrease again.

This generalization behavior has been identified already in the 90s by applying analytical tools from statistical mechanics to problems of machine learning; see for example figure 10 in the paper by Watkin et al. (25) or figure 1 in Opper et al. (29) which show the generalization ability of the so-called pseudoinverse algorithm to train a Boolean linear classifier (30). It is implicit in work on phase diagrams of generalization akin to those for magnetism or the Sherrington–Kirkpatrick model (31, 32).

While these works are indeed the first to observe double descent, its significance for modern machine learning has been recognized by a line of research starting with (33). Double descent has been observed in complex deep neural networks (14) and theoretically analyzed for a number of machine learning models (17, 25, 30, 34, 35). There are, however, scarcely any reports of double descent in graph neural networks. Oono and Suzuki (13) speculate that this may be due to implicit regularization in relation to the so-called oversmoothing (36).

B. Generalization in Supervised vs. Transductive Learning.

When illustrating double descent the test error is usually plotted against model complexity. For this to make sense, the amount of training data must be fixed, so the complexity on the abscissa is really relative complexity; denoting the size of the dataset (node of nodes) by $N$ and the number of parameters by $F$ we let this relative complexity be $\alpha :=F/N$. An alternative is to plot the risk against $\gamma ={\alpha }^{-1}$: Starting from a small amount of data (small $\gamma$), we first go through a regime in which increasing the amount of training data leads to worse performance. In our context this can be interpreted as varying the size of the graph while keeping the number of features fixed.

In transductive node classification, we always observe the entire graph $\mathit{A}$ and the features associated with all vertices $\mathit{X}$, but only a subset of $M$ labels. It is then more natural to vary $\tau :=M/N$ than ${\alpha }^{-1}$, with $M$ being the number of observed labels. Although the resulting curves are slightly different, they both exhibit double descent; in the terminology of Martin and Mahoney, both $\tau$ and ${\alpha }^{-1}$ may be called load-like parameters (26); see also ref. 37.^† In particular, they both have the interpolation peak at $\tau ={\alpha }^{-1}$, or $M=F$, when the system matrix becomes square and poorly conditioned. The key aspect of double descent is that the generalization error decreases on both sides of the interpolation peak.

Using $\tau$ instead of ${\alpha }^{-1}$ is convenient for several reasons: In real datasets, the number of input features is fixed; we cannot vary it. Further, there is no unique way to increase the number of parameters in a GNN and different GNNs are parameterized differently which complicates comparisons. Varying depth may lead to confounding effects such as oversmoothing which is implicit regularization. Varying $\tau$ is a straightforward and clean way to compare different architectures in analogous settings. We can, however, easily vary $\alpha ={\gamma }^{-1}$ in our analytic model described in Section 2; we show the related results in Fig. 5.

C. Experimental Observation of Double Descent in GNNs.

Armed with this understanding, we design an experiment as follows: We study the homophilic citation graph Cora (38) and the heterophilic graphs of Wikipedia pages Chameleon (39) and university web pages Texas (11). We apply different graph convolution networks with different losses, with and without dropout regularization.

Results are shown in Fig. 1. Importantly, we plot both the test error (red) and the test accuracy (black) in node classification against a range of training label ratios $\tau$. In the first column, we use a one-layer GCN similar to the one we analyze theoretically in Section 2, but with added degree normalization, self-loops, and multiple classes; in the second column, we use a two-layer GCN; in the third column we add dropout; in the fourth, we use the cross-entropy loss instead of the MSE. This last model is used in the pytorch-geometric node classification tutorial.^‡

Fig. 1.

Double descent generalization for different GNNs, different losses, with and without explicit regularization, on datasets with varying levels of noise. We plot both the test error (red) and the test accuracy (black) against different training label ratios $\tau$ on the abscissa on a logarithmic scale. First column: one linear layer trained by MSE loss; second column: a two-layer GCN with ReLU activations and MSE loss; third column: a two-layer GCN with ReLU activation function, dropout and MSE loss; fourth column: a two-layer GCN with ReLU activations, dropout and cross-entropy loss; Each experimental data point is averaged over 10 random train–test splits; the shadow area represents the SD. The Right ordinate axis shows classification accuracy; we suppress the Left-axis ticks due to different numerical ranges. We observe that double descent is ubiquitous across datasets and architectures when varying the ratio of training labels: there often exists a regime where more labels impair generalization.

First, with a one-layer network, one can clearly observe transductive double descent on Cora in both the test risk and accuracy. The situation is markedly different on the heterophilic Texas, which contains only 183 nodes but 1,703 features per node which yields relative model complexity $\alpha =F/N$ much higher than for other datasets. Here the test accuracy decreases near-monotonically, consistently with our theoretical analysis in Section 2 (cf. Fig. 4D). In this setting, strong regularization improves performance.

Fig. 4.

Four typical generalization curves in CSBM model. The solid lines represent theoretical results of test risk (black) and accuracy (red) computed via Eq. 17. We also plot the mean and variance of test output ${\mathit{h}}_i({\mathit{w}}^{\ast })$ where $i\in V_{\text{test}}$. This illustrates how the tradeoff of mean–variance leads to different double descent curves. Note we only display results for nodes with label ${\mathit{y}}_i=1$; the result for the ${\mathit{y}}_i=-1$ class simply has opposite mean and identical variance. (A) monotonic $\mathrm{ACC}$ (increasing) and $R_{\text{test}}$ (decreasing) when regularization $r$ is large; (B) A typical double descent with small regularization $r$; (C) slight double descent with relative model complexity $\alpha$ close to $1$; (D) (near-monotonically) decreasing $\mathrm{ACC}$ and increasing $R_{\text{test}}$ with large relative model complexity $\alpha =1/\gamma$. The parameters are chosen as (A) $\mu =1,\lambda =2,\gamma =5,r=2$; (B) $\mu =1,\lambda =2,$-pagination $\gamma =5,r=0.1$; (C) $\mu =1,\lambda =2,\gamma =1.2,r=0.05$; (D) $\mu =5,\lambda =1,\gamma =0.1,r=0.005$. The solid circles and vertical bars represent the mean and SD of risk and accuracy from experiment results. Each experimental data point is averaged over $10$ independent trials; the SD is indicated by vertical bars. We use $N$ = 5,000 and $d=30$ for A, B and C, and $N=500$ and $d=20$ for (D). In all cases, we use the symmetric binary adjacency matrix ensemble ${\mathcal{A}}^{\text{bs}}$.

With a two-layer network the double descent still “survives” in the test error on Cora, but the accuracy is almost monotonically increasing except on Texas. These results corroborate the intuition that dropout and nonlinearity alleviate GNN overfitting on node classification, especially for large training label ratios.

We then explore the role of noise in the graph and in the features by manually adding noise to Cora. We randomly remove $30\%$ of the links and add the same number of random links, and randomize $30\%$ of the entries in $\mathit{X}$; results are shown in the fourth row of Fig. 1. The double descent in test error appears even with substantial regularization. Comparing the first and the fourth row affirms that double descent is more prominent with noisy data; this is again consistent with our analysis (Section 3). In the last row, we apply the networks to the synthetic CSBM. Observing the same qualitative behavior also in this case lends credence to the choice of CSBM for our precise analysis in Section 2.

In Fig. 2, we further focus on the strongly heterophilic Chameleon which does not clearly show double descent in Fig. 1. We randomly perturb different percentages of edges and in addition to GCNs also use the considerably more powerful FSGNN (40), which achieves current state-of-the-art results on Chameleon. Again, we see that double descent (a nonmonotonic risk curve) emerges at higher noise (weaker heterophily). It is noteworthy that more expressive architectures do seem to mitigate double descent; conversely, a one-layer GCN exhibits double descent even without additional noise. We analytically characterize this phenomenon in Section 2 and illustrate it in Fig. 3. Beyond GCNs, we show that double descent occurs in more sophisticated GNNs like graph attention networks (41), GraphSAGE (9) and Chebyshev graph networks (7); see SI Appendix, 5 for details.

Fig. 2.

Test error with different training label ratios for different GCNs on chameleon (heterophilic) datasets. (A) FSGNN (40); (B) two-layer GCN with ReLU activations and cross-entropy loss; (C) one layer GCN with cross-entropy loss; (D) one layer GCN with MSE loss. We interpolate between the original dataset shown in blue ($0\%$ noise), and an Erdős–Rényi random graph shown in red ($100\%$ noise) by adding noise in increments of $20\%$. Noise is introduced by first randomly removing a given proportion of edges and then adding the same number of new random edges. The node features are kept the same. Each data point is averaged ten times, and the abscissa is on a logarithmic scale. We see that graph noise accentuates double descent, which is consistent with our theoretical results (Fig. 3B). Similarly, better GNNs attenuate the effect where additional labels hurt generalization.

Fig. 3.

Theoretical results computed by the replica method (solid line) versus experimental results (solid circles) on CSBM, with $\mathit{P}(\mathit{A})=\mathit{A}$, for varying training label ratios $\tau$. (A) training and test risks with $\lambda =\mu =1$, $\gamma =5$ and $r=0$. (For $\tau <0.2$, we use the pseudoinverse in Eq. 11 in numerics and $r={10}^{-5}$ for the theoretical curves). We further study the impact of varying $\lambda$ in (B) and $r$ in (C). We set $r=0.02$, $\gamma =2$, $\mu =1$ in (B) and $\lambda =3$, $\mu =1$, $\gamma =2$ in (C). In all experiments we set $N$ = 5,000 and $d=30$. We work with the symmetric binary adjacency matrix ensemble ${\mathcal{A}}^{\text{bs}}$. Each experimental data point is averaged over $10$ independent trials; their SD is shown by vertical bars. The theoretical curves agree perfectly with experiments but also qualitatively with the phenomena we observed on real data in Section 1.

In summary, transductive double descent occurs in a variety of graph neural networks applied to real-world data, with noise and implicit or explicit regularization being the key determinants of the shape of generalization curves. Understanding the behavior of generalization error as a function of the number of training labels is of great practical value given the difficulty of obtaining labels in many domains. For some datasets like Texas, using too many labels seems detrimental for some architectures.

2. A Precise Analysis of Node Classification on CSBM with a Simple Graph Convolution Network

Motivated by the above discussions, we turn to a theoretical study of the performance of GCNs on random community graphs where we can understand the influence of all the involved parameters. We have seen in Section 1 that the generalization behavior in this setting qualitatively matches generalization on real data.

Graph convolution networks are composed of graph convolution filters and nonlinear activations. Removing the activations results in a so-called simple GCN (42) or a spectral GNN (43, 44). For a graph $G=(V,E)$ with adjacency matrix $\mathit{A}$ and features that live on the nodes $\mathit{X}$,

$$\begin{array}{c}\hfill \mathit{h}(\mathit{w};\mathit{A},\mathit{X})=\mathit{P}(\mathit{A})\mathit{X}\mathit{w}\text{where}\mathit{P}(\mathit{A})={\displaystyle \sum _{k=0}^K}{c}_k{\mathit{A}}^k,\end{array}$$ [3]

where $\mathit{w}\in {\mathbb{R}}^F$ are trainable parameters and $K$ is the filter support size in terms of hops on the graph. We treat the neighborhood weights $c_k$ at different hops as hyperparameters. We let ${\mathit{A}}^0\stackrel{\text{def}}{=}{\mathit{I}}_N$ so that the model Eq. 3 reduces to ordinary linear regression when $K=0$.

In standard feed-forward networks, removing the activations results in a linear end-to-end mapping. Surprisingly, GCNs without activations such as SGC (42) or with activations only in the output such as FSGNN (40) and GPRGNN (12) achieve state-of-the-art performance in many settings.^§

We will derive test risk expressions for the above graph convolution network in two shallow cases: $\mathit{P}(\mathit{A})=\mathit{A}$ and $\mathit{P}(\mathit{A})=\mathit{A}+c\mathit{I}$. We will also state a universality conjecture for general polynomial filters. Starting with this conjecture, we can in principle extend the results to all polynomial filters using routine but tedious computation. We provide an example for the training error of a two-hop network in SI Appendix, 3. As we will show, this analytic behavior closely resembles the motivational empirical findings from Section 1.

A. Training and Generalization.

We are interested in the large graph limit $N\to \infty$ where the training label ratio $|V_{\text{train}}|/N\to \tau$. We fit the model parameters $\mathit{w}$ by ridge regression $w^{\ast }:=\text{arg}\text{m}\mathrm{\text{i}}{\text{n}}_w{L}_{A,X}(w)$, where

$$\begin{array}{c}\hfill L_{\mathit{A},\mathit{X}}(\mathit{w})=\frac{1}{|V_{\text{train}}|}{\displaystyle \sum _{i\in V_{\text{train}}}}{({\mathit{y}}_i-\left({\mathit{h}}_i(\mathit{w};\mathit{A},\mathit{X})\right))}^2+\frac{r}{N}{\Vert \mathit{w}\Vert }_2^2.\end{array}$$ [4]

We are interested in the training and test risk in the limit of large graphs,

$$\begin{array}{c}\hfill R_{\mathrm{train}}={\displaystyle \underset{N\to \infty }{lim}}\mathbb{E}{R}_N(V_{\mathrm{train}}),R_{\mathrm{test}}={\displaystyle \underset{N\to \infty }{lim}}\mathbb{E}{R}_N(V_{\mathrm{test}}),\end{array}$$ [5]

as well as in the expected accuracy,

$$\begin{array}{c}\hfill \mathrm{ACC}={\displaystyle \underset{N\to \infty }{lim}}\mathbb{E}\left[\frac{1}{|V_{\text{test}}|}{\displaystyle \sum _{i\in V_{\text{test}}}}\mathbf{1}\{{\mathit{y}}_i=\mathrm{sign}({\mathit{h}}_i\left({\mathit{w}}^{\ast }\right))\}\right].\end{array}$$ [6]

We will sometimes write $R_{\mathrm{train}}(\mathcal{A})$, $R_{\mathrm{test}}(\mathcal{A})$, $\mathrm{ACC}(\mathcal{A})$ to emphasize that the matrix $\mathit{A}$ in Eq. 3 follows a distribution $\mathcal{A}$, $\mathit{A}\sim \mathcal{A}$. The expectations are over the random graph adjacency matrix $\mathit{A}$, random features $\mathit{X}$, and the uniformly random test–train partition $V=V_{\mathrm{train}}\cup V_{\mathrm{test}}$. Our analysis in fact shows that the quantities all concentrate around the mean for large $N$ (and $M$ and $F$): In the language of statistical physics, they are self-averaging. This proportional asymptotics regime where $F,M$, and $N$ all grow large at constant ratios is more challenging to analyze than the regimes where dataset size or model complexity is constant, but it results in phenomena we see with production-scale machine learning models on real data; see also refs. 26 and 34.

B. Contextual Stochastic Block Model.

We apply the GCN to the CSBM. CSBM adds node features to the stochastic block model (SBM)—a random community graph model (24) where the probability of a link between nodes depends on their communities. The lower triangular part of the adjacency matrix ${\mathit{A}}^{\text{bs}}$ has distribution

$$\begin{array}{c}\hfill \mathbb{P}\left({\mathit{A}}_{\mathit{ij}}^{\text{bs}}=1\right)=\left\{\begin{array}{cc}{c}_{\text{in}}/N\hfill & \text{if} i\le j \text{and} {\mathit{y}}_i={\mathit{y}}_j\hfill \\ c_{\text{out}}/N\hfill & \text{if} i\le j \text{and} {\mathit{y}}_i\ne {\mathit{y}}_j.\hfill \end{array}\right.\end{array}$$ [7]

A convenient parameterization is

$$\begin{array}{c}\hfill c_{\text{in}}=d+\sqrt{d}\lambda ,c_{\text{out}}=d-\sqrt{d}\lambda ,\end{array}$$

where $d$ is the average node degree and the sign of $\lambda$ determines whether the graph is homophilic or heterophilic; $|\lambda |$ can be regarded as the graph signal noise ratio (SNR).

We will also study a directed SBM (45, 46) with adjacency matrix distributed as

$$\begin{array}{c}\hfill \mathbb{P}\left({\mathit{A}}_{\mathit{ij}}^{\text{bn}}=1\right)=\left\{\begin{array}{cc}{c}_{\text{in}}/N\hfill & \text{if} {\mathit{y}}_i={\mathit{y}}_j\hfill \\ c_{\text{out}}/N\hfill & \text{if} {\mathit{y}}_i\ne {\mathit{y}}_j.\hfill \end{array}\right.\end{array}$$ [8]

Many real graphs have directed links, including chemical connections between neurons, the electric grid, followee–follower relation in social media, and Bayesian graphs. In our case the directed SBM facilitates analysis with self-loops while exhibiting the same qualitative behavior and phenomenology as the undirected one.

The features of CSBM follow the spiked covariance model,

$$\begin{array}{c}\hfill {\mathit{x}}^i=\sqrt{\frac{\mu }{N}}{\mathit{y}}_i\mathit{u}+{\mathit{\xi }}^i,\end{array}$$ [9]

where $\mathit{u}\sim \mathcal{N}(0,{\mathit{I}}_F/F)$ is the $F$-dimensional hidden feature and ${\mathit{\xi }}^i\sim \mathcal{N}(0,{\mathit{I}}_F/F)$ are i.i.d. Gaussian noise; the parameter $\mu$ is the feature SNR. We work in the proportional scaling regime where $\frac{N}{F}\to \gamma$, with $\gamma$ being the inverse relative model complexity, and ascribe feature vectors to the rows of the data matrix $\mathit{X}$,

$$\begin{array}{c}\hfill \mathit{X}={[{\mathit{x}}^1,\cdots ,{\mathit{x}}^N]}^{⊺}=\sqrt{\frac{\mu }{N}}\mathit{y}{\mathit{u}}^{⊺}+{\mathbf{\Xi }}^x.\end{array}$$ [10]

We assume throughout that the two communities are balanced; without loss of generality we let ${\mathit{y}}_i=1$ for $i=1,2,\dots ,N/2$ and ${\mathit{y}}_i=-1$ for $i>N/2$.

We will show that CSBM is a comparatively tractable statistical model to characterize generalization in GNNs. Intuitively, when $N\to \infty$, the risk should concentrate around a number that depends on five parameters:

$$\begin{array}{cc}\lambda & \text{Degree of homophily}\mathrm{(Graph\; SNR),}\hfill \\ \mu & \text{Feature SNR,}\hfill \\ \alpha & \text{Relative model complexity}(={\gamma }^{-1}),\hfill \\ \tau & \text{Label ratio},\hfill \\ r& \text{Ridge regularization parameter.}\hfill \end{array}$$

We emphasize that we study the challenging weak-signal regime where $\lambda$, $\mu$ and $\gamma$ do not scale with $N$ (but $F$ does). This stands in contrast to recent machine learning work on CSBM (47, 48) which studies the low-noise regime where $\mu$ or ${\lambda }^2$ scale with $N$, or even the noiseless regime where the classes become linearly separable after applying a graph filter or a GCN. We argue that the weak-signal regime is closer to real graph learning problems which are neither too easy (as in linearly separable) nor too hard (as with a vanishing signal). The fact that we identify phenomena which occur in state-of-the-art networks and real datasets supports this claim.

We outline our analysis in Section 5 and provide the details in SI Appendix. But first, in the following section, we show that the derived expressions precisely characterize generalization of shallow GCNs on CSBM and also give a correct qualitative description of the behavior of “big” graph neural networks on complex datasets, pointing to interesting phenomena and interpretations.

3. Phenomenology of Generalization in GCNs

We focus on the behavior of the test risk under various levels of graph homophily, emphasizing two main aspects: i) different levels of homophily lead to different types of double descent; ii) self-loops, standard in GCNs, create an imbalance between heterophilic and homophilic datasets; negative self-loops improve the handling of heterophilic datasets.

A. Double Descent in Shallow GCNs on CSBM.

As we show in Section 5 and SI Appendix, 1, the expression for the test risk for unregularized regression ($r=0$) with shallow GCN can be obtained in closed form as

$$R_{\mathrm{test}}=\frac{\gamma \tau (\gamma +\mu )}{(\gamma \tau -1)\left(\gamma +{\lambda }^2(\mu +1)+\mu \right)}$$

when $\gamma \tau >1$. It is evident that the denominator vanishes as $\gamma \tau$ approaches 1. When this happens, the system matrix ${\mathit{I}}_{\mathrm{train}}\mathit{P}(\mathit{A})\mathit{X}$, where ${\mathit{I}}_{\mathrm{train}}$ selects the rows for which we have labels; see Section 5, Eq. 12, is square and near-singular for large $N$, which leads to the explosion of $R_{\mathrm{test}}$ (Fig. 3A). When relative model complexity is high, i.e., $\gamma =N/F<1$ is low, $\tau \gamma$ is always less than $1$. In such cases, no interpolation peak appears, which is consistent with our experimental results for the Texas dataset where $\gamma =0.11$; cf. Fig. 4D and the third row of Fig. 1.

At the other extreme, for strongly regularized training (large $r$) the double descent disappears (Fig. 3C); it has been shown that this happens at optimal regularization (35, 49). The absolute risk values in Fig. 3 B and C show the same behavior.

Fig. 3B shows that when the graph is very noisy ($\lambda$ is small) the test error starts to increase as soon as the training label ratio $\tau$ increases from $0$. When $\lambda$ is large, meaning that the graph is discriminative, the test error first decreases and then increases. Similar behavior can be observed when varying the feature SNR $\mu$ instead of $\lambda$. Double descent also appears in test accuracy (Fig. 4).

While these curves all illustrate double descent in the sense that they all have the interpolation peak on both sides of which the error decreases, they are qualitatively different. The emergence of these different shapes can be explained by looking at the distribution of the predicted $i$th label ${\mathit{h}}_i({\mathit{w}}^{\ast })$. As we show in SI Appendix, 1, ${\mathit{h}}_i({\mathit{w}}^{\ast })$ is normally distributed with mean and variance given by the solutions of a saddle point equation outlined in Section 5. The test accuracy can thus be expressed by the error function (cf. Eq. 17).

As we increase the number of labels, the mean $\mathbb{E}[{\mathit{h}}_i({\mathit{w}}^{\ast })]$ approaches ${\mathit{y}}_i$ monotonically. However, the variance $\mathrm{Var}[{\mathit{h}}_i({\mathit{w}}^{\ast })]$ behaves differently for different model complexities $\alpha =\frac{1}{\gamma }$ and regularizations $r$, resulting in distinct double descent curves.

For example, when $r\to 0$ and $\tau \to \frac{1}{\gamma }$, the variance of ${\mathit{h}}_i({\mathit{w}}^{\ast })$ for $i\in V_{\text{test}}$ diverges and the accuracy approaches $50\%$, a random guess. On the other hand, when $r$ is large, the variance is small and double descent is mild or absent, as shown in Fig. 4A. Fig. 4B shows a typical double descent curve with two regimes where additional labels hurt generalization. In Fig. 4C we also see a mild double descent when the relative model complexity is close to $1$: this is consistent with experimental observations on Cora in Fig. 1. In certain extremal cases, for example when $\gamma$ is very small, the test accuracy continuously decreases after a very small ascent around $\tau =0$ (Fig. 4D); this is consistent with our experimental observations for the Texas dataset.

B. Double Descent as a Function of the Relative Model Complexity.

As mentioned earlier, the theoretical model makes it easy to study double descent as we vary the model complexity $\alpha =1/\gamma$ rather than $\tau$; this is closer to the traditional reports of double descent in supervised learning. The resulting plots follow a similar logic: as shown in Fig. 5, adding randomness in the graph (low $|\lambda |$), makes the double descent more prominent. Conversely, for a highly homophilic graph (large $\lambda$), the test risk decreases monotonically as the relative model complexity $\alpha$ grows.

C. Heterophily, Homophily, and Positive and Negative Self-Loops.

GCNs often perform worse on heterophilic than on homophilic graphs. An active line of research tries to understand and mitigate this phenomenon with special architectures and training strategies (12, 50, 51). We now show that it can be understood through the lens of self-loops.

Strong GCNs ubiquitously employ self-loops of the form $\mathit{P}(\mathit{A})=\mathit{A}+{\mathit{I}}_N$ on homophilic graphs (8, 12, 41, 42, 52).^¶ Self-loops, however, deteriorate performance on heterophilic networks. CSBM is well suited to study this phenomenon since $\lambda$ allows us to transition between homophilic and heterophilic graphs.

We allow the self-loop strength $c$ to vary continuously so that the effective adjacency matrix becomes $\mathit{A}+c{\mathit{I}}_N$. Importantly, we also allow $c$ to be negative (SI Appendix, 2). In Fig. 6 we plot the test risk as a function of $c$ for both positive and negative $c$. We find that a negative self-loop ($c<0$) results in much better performance on heterophilic data ($\lambda <0$). We sketch a signal-processing interpretation of this phenomenon in SI Appendix, 4.

Fig. 6.

Train and test risks on CSBM for different intensities of self loops. (A) train and test risk for $\tau =0.8$ and $\lambda =-1$ (heterophilic). (B) test risks for $\gamma =0.8$, $\tau =0.8$, $\mu =0$ under different $\lambda$. (C) training risk for different $\mu$ when $\tau =\lambda =1$. Each data point is averaged over $10$ independent trials with $N$ = 5,000, $r=0$, and $d=30$. We use the nonsymmetric binary adjacency matrix ensemble ${\mathcal{A}}^{\text{bn}}$. The solid lines are the theoretical results predicted by the replica method. In (B) we see that the optimal generalization performance requires adapting the self-loop intensity $c$ to the degree of homophily.

D. Negative Self-Loops in State-of-the-Art GCNs.

It is remarkable that this finding generalizes to complex state-of-the-art graph neural networks and datasets. We experiment with two common heterophilic benchmarks, Chameleon and Squirrel, first with a two-layer ReLU GCN. The default GCN (for example in pytorch-geometric) contains self-loops of the form $\mathit{A}+\mathit{I}$; we immediately observe in Fig. 7 that removing them improves performance on both datasets. We then make the intensity of the self-loop adjustable as a hyper-parameter and find that a negative self-loop with $c$ between $-1.0$ and $-0.5$ results in the highest accuracy on both datasets. It is notable that the best performance in the two-layer ReLU GCN with $c=-0.5$ (76.29%) is already close to state-of-the-art results by the Feature Selection Graph Neural Network (FSGNN) (40) (78.27%). FSGNN uses a graph filter bank $\mathcal{B}=\{{\mathit{A}}^k,{(\mathit{A}+\mathit{I})}^k\}$ with careful normalization. Taking a cue from the above findings, we show that a simple addition of negative self-loop filters ${(\mathit{A}-0.5\mathit{I})}^k$ to FSGNN results in a better performance (78.96%) than previous state of the art; see also Table 1.

Fig. 7.

Test accuracy (black) and test error (red) in node classification with GCNs on real heterophilic graphs with different self-loop intensities. We implement a two-layer ReLU GCN with $128$ hidden neurons and an additional self-loop with strength $c$. Each setting is averaged over different training–test splits taken from ref. 11 (60% training, 20% validation, 20% test). The relatively large SD (vertical bars) is mainly due to the randomness of the splits. The randomness from model initialization and training is comparatively small. The optimal test accuracy for these two datasets is obtained for self-loop intensity $-0.5<c^{\ast }<-1$.

Table 1.

Comparison of test accuracy when negative self-loop is absent (first and third column) or present (second and fourth column)

	GCN ($c=0$)	GCN ($c^{\ast }$)	FSGNN	FSGNN ($c^{\ast }$)
Chameleon	75.81 $\pm$ 1.69	76.29 $\pm$ 1.22	78.27 $\pm$ 1.28	78.96 $\pm$ 1.05
Squirrel	67.19 $\pm$ 1.48	68.62 $\pm$ 2.13	74.10 $\pm$ 1.89	74.34 $\pm$ 1.21

The datasets and splits are the same as Fig. 7.

4. Discussion

Before delving into the details of the analytical methods in Section 5 and conceptual connections between GNNs and spin glasses, we discuss the various interpretations of our results in the context of related work.

A. Related Work on Theory of GNNs.

Most theoretical work on GNNs addresses their expressivity (53, 54). A key result is that the common message-passing formulation is limited by the power of the Weisfeiler–Lehman graph isomorphism test (55). This is of great relevance for computational chemistry where one must discriminate between the different molecular structures (56), but it does not explain how the interaction between the graph structure and the node features leads to generalization. Indeed, simple architectures like GCNs are far from being universal approximators but they often achieve excellent performance on real problems with real data.

Existing studies of generalization in GNNs leverage complexity measures such as the Vapnik–Chervonenkis dimension (57–59) or the Rademacher complexity (21). While the resulting bounds sometimes predict coarse qualitative behavior, a precise characterization of relevant phenomena remains elusive. Even the more refined techniques like PAC-Bayes perform only marginally better (22). It is striking that only in rare cases do these bounds explicitly incorporate the interaction between the graph and the features (23). Our results show that understanding this interaction is crucial to understanding learning on graphs.

Indeed, recall that a standard practice in the design of GNNs is to build (generalized) filters from the adjacency matrix or the graph Laplacian and then use these filters to process data. But if the underlying graph is an Erdős–Rényi random graph, the induced filters will be of little use in learning. The key is thus to understand how much useful information the graph provides about the labels (and vice-versa), and in what way that information is complementary to that contained in the features.

B. A Statistical Mechanics Approach: Precise Analysis of Simple Models.

An alternative to the typically vacuous^# complexity-based risk bounds for graph neural networks (21–23) is to adopt a statistical mechanics perspective on learning; this is what we do here. Indeed, one key aspect of learning algorithms that is not easily captured by complexity measures of statistical learning theory is the emergence of qualitatively distinct phases of learning as one varies certain key “order parameters”. Such phase diagrams emerge naturally when one views machine learning models in terms of statistical mechanics of learning (26, 37).

Martin and Mahoney (26) demonstrate this elegantly by formulating what they call a very simple deep learning model, and showing that it displays distinct learning phases reminiscent of many realistic, complex models, despite abstracting away all but the essential “load-like” and “temperature-like” parameters. They argue that such parameters can be identified in machine learning models across the board.

The statistical mechanics paradigm requires one to commit to a specific model and do different calculations for different models (25), but it results in sharp characterizations of relevant phases of learning.

Important results within this paradigm, both rigorous and heuristic, were derived over the last decade for regularized least-squares (60–62), random-feature regression (17, 34, 49, 63), and noisy Gaussian mixture and spiked covariance models (64–66), using a variety of analytical techniques from statistical physics, high-dimensional probability, and random matrix theory. Not all of these works explicitly declare adherence to the statistical mechanics tradition. It nonetheless seems appropriate to categorize them thus since they provide precise analyses of learning in specific models in terms of a few order parameters.

Even though these papers study comparatively simple models, many key results only appeared in the last couple of years, motivated by the proliferation of over-parameterized models and advances in analytical techniques. One should make sure to work in the correct scaling of the various parameters (34); while this may complicate the analysis it leads to results which match the behavior of realistic machine learning systems. We extend these recent results by allowing the information to propagate on a graph; this gives rise to interesting phenomena of some relevance for the practitioners. In order to obtain precise results we similarly study simple graph networks, but we also show that the salient predictions closely match the behavior of state-of-the-art networks on real datasets. We precisely traced the connection between generalization, the interaction type (homophilic or heterophilic) and the parameters of the GCN architecture and the dataset for a specific graph model. Experiments show that the learned lessons apply to a broad class of GNNs and can be used constructively to improve the performance of state-of-the-art graph neural networks on heterophilic data.

Finally, let us mention that phenomenological characterizations of phase diagrams of risk are not the only way to apply tools from statistical mechanics and more broadly physics to machine learning and neural networks. These tools may help address a rather different set of “design” questions, as reviewed by Bahri et al. (67).

C. Negative Self-Loops in Other Graph Learning Models.

Recent theoretical work (19, 20) shows that optimal message passing in heterophilic datasets requires aggregating neighbor messages with a sign opposite from that of node-specific updates. Similarly, in earlier GCN architectures such as GraphSAGE (9), node and neighbor features are extracted using different trainable functions. This immediately allows the possibility of aggregating neighbors with an opposite sign in heterophilic settings. We show that self-loops with sign and strength depending on the degree of heterophily improve performance both in theory and in real state-of-the-art GCNs. The notion of self-loops in the context of GCNs usually indicates an explicit connection between a node and itself, $\mathit{A}\leftarrow \mathit{A}+\mathit{I}$.

D. GCNs with a Few Labels Outperform Optimal Unsupervised Detection.

One interpretation of our results is that they quantify the value of labels in community detection, traditionally approached with unsupervised methods. These approaches are subject to fundamental information-theoretic detection limits which have drawn considerable attention over the last decade (64, 68, 69). The most challenging and most realistic high-dimensional setting is when the signal strength is comparable to that of the noise for both the graph and the features (24, 68, 70). The results of Deshpande et al. indicate that when ${\mu }^2/\gamma +{\lambda }^2<1$, no unsupervised estimator can detect the latent structure $\mathit{y}$ from $\mathit{A}$ and $\mathit{X}$ (24). Our analysis shows that even a small fraction of revealed labels allow a simple GCN to break this unsupervised barrier.

In Fig. 8, we compare the accuracy of a one-layer GCN with unsupervised belief propagation (BP) (24). We first run BP with $\mu =\lambda =\gamma =1$ and record the achieved accuracy. We then plot the smallest training label ratio $\tau$ for which the GCN achieves the same accuracy. We repeat this procedure for different feature SNRs $\mu$ and graph SNRs $\lambda$. The black solid line indicates the information-theoretic threshold for detecting the latent structure from $\mathit{A}$ and $\mathit{X}$.

Fig. 8.

Training label ratio when a one-layer GCN matches the performance of unsupervised belief propagation at $\mu =\lambda =\gamma =1$. The black solid line denotes the information-theoretic detection threshold in the unsupervised setting where no label information is available (i.e., when we use only $\mathit{A}$, $\mathit{X}$). If given a small number of labels, a simple, generally sub-optimal estimator matches the performance of the optimal unsupervised estimator.

Earlier analyses of belief propagation in the SBM without features uncover a detectability phase transition (71). Our analysis shows that no such transition happens with GCNs. Indeed, our primary interest is in understanding GCNs, which are a general tool for a variety of problems, but unlike belief propagation, GCNs need not be near-optimal for community detection. For the optimal inference strategy, the phase transition may not be destroyed by revealing labels.

5. Generalization in GCNs via Statistical Physics

The optimization problem Eq. 4 has a unique minimizer as long as $r>0$. Since it is a linear least-squares problem in $\mathit{w}$, we can write down a closed-form solution,

$$\begin{array}{c}\hfill {\mathit{w}}^{\ast }={(r{\mathit{I}}_F+{(\mathit{P}(\mathit{A})\mathit{X})}^T{\mathit{I}}_{\text{train}}\mathit{P}(\mathit{A})\mathit{X})}^{-1}{(\mathit{P}(\mathit{A})\mathit{X})}^T{\mathit{I}}_{\text{train}}\mathit{y},\end{array}$$ [11]

where

$$\begin{array}{c}\hfill {\left({\mathit{I}}_{\text{train}}\right)}_{\mathit{ij}}=\left\{\begin{array}{cc}1\hfill & \text{if}i=j\in V_{\text{train}}\hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}\right.\end{array}$$ [12]

Analyzing generalization is, in principle, as simple as substituting the closed-form expression Eq. 11 into Eq. 5 and Eq. 6 and calculating the requisite averages. The procedure is, however, complicated by the interaction between the graph $\mathit{A}$ and the features $\mathit{X}$ and the fact that $\mathit{A}$ is a random binary adjacency matrix. Further, for a symmetric $\mathit{A}$, ${\mathit{I}}_{\text{train}}\mathit{P}(\mathit{A})$ is correlated with ${\mathit{I}}_{\text{test}}\mathit{P}(\mathit{A})$ even in a shallow GCN (and certainly in a deep one).

A. The Statistical Physics Program.

We interpret the (scaled) loss function as an energy, or a Hamiltonian, $H(\mathit{w};\mathit{A},\mathit{X})=\tau N{L}_{\mathit{A},\mathit{X}}(\mathit{w})$. Corresponding to this Hamiltonian is the Gibbs measure over the weights $\mathit{w}$,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathrm{d}{\mathbb{P}}_{\beta }(\mathit{w};\mathit{A},\mathit{X})& =\frac{exp(-\beta H(\mathit{w};\mathit{A},\mathit{X}))\mathrm{d}\mathit{w}}{Z_{\beta }(\mathit{A},\mathit{X})}\hfill \\ \hfill \text{where}{Z}_{\beta }(\mathit{A},\mathit{X})& =\int \mathrm{d}\mathit{w}exp\left(-\beta H\left(\mathit{w};\mathit{A},\mathit{X}\right)\right),\hfill \end{array}\end{array}$$

$\beta$ is the inverse temperature and $Z_{\beta }$ is the partition function. At infinite temperature ($\beta \to 0$), the Gibbs measure is diffuse; as the temperature approaches zero $(\beta \to \infty )$, it converges to an atomic measure concentrated on the unique solution of Eq. 4, ${\mathit{w}}^{\ast }={\displaystyle \underset{\beta \to \infty }{lim}}\int \mathit{w}{\mathbb{P}}_{\beta }(\mathit{w};\mathit{A},\mathit{X})\mathrm{d}\mathit{w}.$ In this latter case, the partition function is similarly dominated by the minimum of the Hamiltonian. The expected loss can thus be computed from the free energy density $f_{\beta }$,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{E}}_{\mathit{A},\mathit{X}}[L_{\mathit{A},\mathit{X}}({\mathit{w}}^{\ast })]=& \frac{1}{\tau }{\displaystyle \underset{\beta \to \infty }{lim}}{f}_{\beta }\hfill \\ \hfill \text{where}{f}_{\beta }:=& -{\displaystyle \underset{N\to \infty }{lim}}\frac{1}{N\beta }{\mathbb{E}}_{\mathit{A},\mathit{X}}ln{Z}_{\beta }(\mathit{A},\mathit{X}).\hfill \end{array}\end{array}$$

Since the quenched average $\mathbb{E}ln{Z}_{\beta }$ is usually intractable, we apply the replica method (72) which allows us to take the expectation inside the logarithm and compute the annealed average,

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathit{A},\mathit{X}}ln{Z}_{\beta }\left(\mathit{A},\mathit{X}\right)={\displaystyle \underset{n\to 0}{lim}}\frac{ln{\mathbb{E}}_{\mathit{A},\mathit{X}}{Z}_{\beta }^n\left(\mathit{A},\mathit{X}\right)}{n}.\end{array}$$

The gist of the replica method is to compute ${\mathbb{E}}_{\mathit{A},\mathit{X}}{Z}_{\beta }^n$ for integer $n$ and then “pretend” that $n$ is real and take the limit $n\to 0$. The computation for integer $n$ is facilitated by the fact that $Z_{\beta }^n$ normalizes the joint distribution of $n$ independent copies of $\mathit{w}$, ${\{{\mathit{w}}^a\}}_{a=1}^n$. We obtain

$$\begin{array}{l}{\mathbb{E}}_{A,X}{Z}_{\beta }^n(A,X)\\ ={\mathbb{E}}_{A,X}{\left(Z_{\beta }(A,X)\right)}_1\times \cdots \times {\left(Z_{\beta }(A,X)\right)}_n\\ ={\displaystyle \int {\displaystyle \prod _{a=1}^n}d{w}^a{\mathbb{E}}_{A,X}\exp }\left({\displaystyle \sum _{a=1}^n}\left(-\beta {\Vert I_{\text{}train}AX{w}^a-I_{\text{}train}y\Vert }_2^2\right)\right)\\ \times \exp (-\beta \tau r\Vert w^a{\Vert }_2^2).\end{array}$$ [13]

Instead of working with the product $\mathit{A}\mathit{X}$, replica allows us to express the free energy density as a stationary point of a function where the dependence on $\mathit{A}$ and $\mathit{X}$ is separated (see SI Appendix, 1 for details),

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f_{\beta }=& \frac{1}{\beta }\text{extr}{\displaystyle \underset{n\to 0}{lim}\underset{N\to \infty }{lim}}\frac{1}{\mathit{nN}}\left({\mathbb{E}}_{\mathit{A}}\left[c\left(\mathit{P}(\mathit{A})\right)\right]+{\mathbb{E}}_{\mathit{X}}\left[e(\mathit{X})\right]\right)\hfill \\ \hfill & +D(m,p,q,\widehat{m},\widehat{p},\widehat{q})\hfill \\ \hfill =& \frac{1}{\beta }\underset{\begin{array}{c}m,p,q\\ \widehat{m},\widehat{p},\widehat{q}\end{array}}{\text{extr}}C(m,p,q)+E(\widehat{m},\widehat{p},\widehat{q})+D(m,p,q,\widehat{m},\widehat{p},\widehat{q}),\hfill \end{array}\end{array}$$ [14]

where we defined $C\stackrel{\text{def}}{=}\frac{1}{\mathit{nN}}{\mathbb{E}}_{\mathit{A}}\left[c\left(\mathit{P}(\mathit{A})\right)\right]$, $E\stackrel{\text{def}}{=}\frac{1}{\mathit{nN}}{\mathbb{E}}_{\mathit{X}}\left[e(\mathit{X})\right]$, which in the limit $N\to \infty$, $n\to 0$ only depend on the so-called order parameters $m,p,q$ and $\widehat{m},\widehat{p},\widehat{q}$. The separation thus allows us to study the influence of the distribution of $\mathit{A}$ in isolation; we provide the details in SI Appendix, 1. The risks (called the observables in physics) can be obtained from $f_{\beta }$.

B. Gaussian Adjacency Equivalence.

A challenge in computing the quantities in Eq. 13 and Eq. 14 is to average over the binary adjacency matrix $\mathit{A}$. We argue that $f$ in Eq. 14 does not change if we instead average over the Gaussian ensemble with a correctly chosen mean and covariance. For a one-layer GCN ($\mathit{P}(\mathit{A})=\mathit{A}$), we show that replacing ${\mathbb{E}}_{{\mathit{A}}^{\text{bs}}}c(\mathit{P}({\mathit{A}}^{\text{bs}}))$ by ${\mathbb{E}}_{{\mathit{A}}^{\text{gn}}}c\left(\mathit{P}({\mathit{A}}^{\text{gn}})\right)$ will not change $f$ in Eq. 14 with ${\mathit{A}}^{\text{gn}}$ being a spiked nonsymmetric Gaussian random matrix,

$$\begin{array}{c}\hfill {\mathit{A}}^{\text{gn}}=\frac{\lambda }{N}\mathit{y}{\mathit{y}}^T+{\mathbf{\Xi }}^{\text{gn}},\end{array}$$ [15]

with ${\mathbf{\Xi }}^{\text{gn}}$ having i.i.d. centered normal entries with variance $1/N$. This substitution is inspired by the universality results for the disorder of spin glasses (73–75) and the universality of mutual information in CSBM (24). Deshpande et al. (24) showed that the binary adjacency matrix in the stochastic block model can be replaced by

$$\begin{array}{c}\hfill {\mathit{A}}^{\text{gs}}=\frac{\lambda }{N}\mathit{y}{\mathit{y}}^T+{\mathbf{\Xi }}^{\text{gs}},\end{array}$$ [16]

where ${\mathbf{\Xi }}^{\text{gs}}\in {\mathbb{R}}^{N\times N}$ is a sample from the standard Gaussian orthogonal ensemble, without affecting the mutual information between $\mathit{y}$ (which they modeled as random) and ($\mathit{A},\mathit{X}$) when $N\to \infty$ and $d\to \infty$.

Our claim refers to certain averages involving $\mathit{A}$; we record it as a conjecture since our derivations are based on the nonrigorous replica method. We first define four probability distributions:

• ${\mathcal{A}}^{\mathrm{bs}}$: The distribution of adjacency matrices in the undirected CSBM (cf. Eq. 7) scaled by $1/\sqrt{d}$, $\frac{1}{\sqrt{d}}{\mathit{A}}^{\mathrm{bs}}\sim {\mathcal{A}}^{\mathrm{bs}}$;

• ${\mathcal{A}}^{\mathrm{bn}}$: the distribution of adjacency matrices in the directed CSBM (cf. Eq. 8), scaled by $1/\sqrt{d}$;

• ${\mathcal{A}}^{\mathrm{gs}}$: the distribution of spiked Gaussian orthogonal ensemble (cf. Eq. 16);

• ${\mathcal{A}}^{\mathrm{gn}}$: the distribution of spiked Gaussian random matrices (cf. Eq. 15).

With these definitions in hand we can state

Conjecture 1.

Assume that $d$ scales with $N$ so that $1/d\to 0$ and $d/N\to 0$ when $N\to \infty$. Let $\mathit{P}(\mathit{A})$ be a polynomial in $\mathit{A}$ used to define the GCN function in Eq. 3). It then holds that

$$\begin{array}{cc}\hfill R_{\mathrm{train}}({\mathcal{A}}^{\mathrm{b}\bullet })& =R_{\mathrm{train}}({\mathcal{A}}^{\mathrm{g}\bullet }),\hfill \\ \hfill R_{\mathrm{test}}({\mathcal{A}}^{\mathrm{b}\bullet })& =R_{\mathrm{test}}({\mathcal{A}}^{\mathrm{g}\bullet }),\hfill \\ \hfill \mathrm{ACC}({\mathcal{A}}^{\mathrm{b}\bullet })& =\mathrm{ACC}({\mathcal{A}}^{\mathrm{g}\bullet }),\hfill \\ \hfill \end{array}$$

with $\bullet \in \{\mathrm{s},\mathrm{n}\}$. When $\mathit{P}(\mathit{A})=\mathit{A}$, the above quantities for symmetric and nonsymmetric distributions also coincide.

In the case when $\mathit{P}(\mathit{A})=\mathit{A}$ we justify Conjecture 1. by the replica method (SI Appendix, 1). In the general case we provide abundant numerical evidence in Fig. 9. We first consider the case when $\mathit{P}(\mathit{A})=\mathit{A}$. Fig. 9 A and D show estimates of $R_{\text{train}}$ and $R_{\text{test}}$ averaged over $100$ independent runs. The SD over independent runs is indicated by the shading. We see that the means converge and the variance shrinks as $N$ grows.

Fig. 9.

Numerical validation of Conjecture 1.. In (A and D) we show training and test risks with different numbers of nodes for $\mathit{P}(\mathit{A})=\mathit{A}$. The parameters are set to $\gamma =\lambda =\mu =2,r=0.01,\tau =0.8$ and $d=\sqrt{N}/2$. In (B and E) we show the absolute difference of the risks between binary and Gaussian adjacency as a function of $N$, using the same data in (A and D). The solid lines correspond to a linear fit in the logarithmic scale, which shows that the error scales as $|\mathrm{\Delta }|\sim N^{-0.5}$. In (C and F) we show the training and test risks when $\mathit{P}(\mathit{A})={\mathit{A}}^2$ under different average node degrees $d$. Other parameters are set to $\lambda =\mu =1,\gamma =2,N=2,000,\tau =0.8$ and $r=0.01$. In these settings, the conjecture empirically holds up to scrutiny.

We also show absolute differences between the averages of $R_{\text{train}}$ and $R_{\text{test}}$ in Fig. 9 B and E. We find that the values of $R_{\text{train}}$ and $R_{\text{test}}$ can be well fitted by a linear relationship in the logarithmic scale, suggesting that the absolute differences approach zero exponentially fast as $N\to \infty$. We next consider $\mathit{P}(\mathit{A})={\mathit{A}}^2$. In Fig. 9 C and F we can see that for intermediate values of $d$, $R_{\text{train}}$ and $R_{\text{test}}$ corresponding to ${\mathit{A}}^{\text{bs}}$ and ${\mathit{A}}^{\text{bn}}$ are both close to that corresponding to ${\mathit{A}}^{\text{gs}}$ and ${\mathit{A}}^{\text{gn}}$. This is consistent with the results shown in Figs. 3 and 6 where the theoretical results computed by the replica method and ${\mathit{A}}^{\text{gn}}$ perfectly match the numerical results with ${\mathit{A}}^{\text{bn}}$ (for $\mathit{P}(\mathit{A})=\mathit{A}+c{\mathit{I}}_N$) and ${\mathit{A}}^{\text{bs}}$ (for $\mathit{P}(\mathit{A})=\mathit{A}$), further validating the conjecture.

C. Solution to the Saddle Point Equation.

We can now solve the saddle point Eq. 14 by averaging over ${\mathit{A}}^{\text{gn}}$. In the general case the solution is easy to obtain numerically. For an one-layer GCN with $\mathit{P}(\mathit{A})=\mathit{A}$ we can compute a closed-form solution. Denoting the critical point in Eq. 14 by $(m^{\ast },p^{\ast },q^{\ast })$ we obtain

$$\begin{array}{cc}\hfill R_{\text{train}}& =\frac{{(\lambda m^{\ast }-1)}^2+p^{\ast }}{{(2q^{\ast }+1)}^2},\hfill \\ \hfill R_{\text{test}}& ={(\lambda m^{\ast }-1)}^2+p^{\ast },\hfill \\ \hfill \mathrm{ACC}& =\frac{1}{2}\left(1+\text{erf}\left(\frac{\lambda m^{\ast }}{\sqrt{2p^{\ast }}}\right)\right),\hfill \end{array}$$ [17]

where $\text{erf}$ is the usual error function. While the general expressions are complicated (SI Appendix, 1), in the ridgeless limit $r\to 0$ we can compute simple closed-form expressions for train and test risks,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill R_{\text{train}}=& \frac{(\gamma +\mu )(\gamma \tau -1)}{\gamma \tau \left(\gamma +{\lambda }^2(\mu +1)+\mu \right)},\hfill \\ \hfill R_{\text{test}}=& \frac{\gamma \tau (\gamma +\mu )}{(\gamma \tau -1)\left(\gamma +{\lambda }^2(\mu +1)+\mu \right)},\hfill \end{array}\end{array}$$ [18]

assuming that $\tau \gamma >1$.

D. A Rigorous Solution.

We note that for a one-layer GCN risks can be computed rigorously using random matrix theory provided that Conjecture 1. holds and we begin with a Gaussian “adjacency matrix” instead of the true binary SBM adjacency matrix. We outline this approach in SI Appendix, 3; in particular, for $r=0$, the result of course coincides with that in Eq. 18.

6. Conclusion

We analyzed generalization in graph neural networks by making an analogy with a system of interacting particles: particles correspond to the data points and the interactions are specified by the adjacency relation and the learnable weights. The latter can be interpreted as defining the “interaction physics” of the problem. The best weights correspond to the most plausible interaction physics, coupled in turn with the network formation mechanism.

The setting that we analyzed is maybe the simplest combination of a graph convolution network and data distribution which exhibits interesting, realistic behavior. In order to theoretically capture a broader spectrum of complexity in graph learning we need to work on new ideas in random matrix theory and its neural network counterparts (76). While very deep GCNs are known to suffer from oversmoothing, there exists an interesting intermediate-depth regime beyond a single layer (77). Our techniques should apply simply by replacing $\mathit{A}$ by any polynomial $\mathit{P}(\mathit{A})$ before solving the saddle point equation, but we will need a generalization of existing random matrix theory results for HCIZ integrals. Finally, it is likely that these generalized results could be made fully rigorous if “universality” in Conjecture 1. could be established formally.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Previously published data were used for this work (11, 38, 39).