Learnable Filters for Geometric Scattering Modules

Abstract

We propose a new graph neural network (GNN) module, based on relaxations of recently proposed geometric scattering transforms, which consist of a cascade of graph wavelet filters. Our learnable geometric scattering (LEGS) module enables adaptive tuning of the wavelets to encourage band-pass features to emerge in learned representations. The incorporation of our LEGS-module in GNNs enables the learning of longer-range graph relations compared to many popular GNNs, which often rely on encoding graph structure via smoothness or similarity between neighbors. Further, its wavelet priors result in simplified architectures with significantly fewer learned parameters compared to competing GNNs. We demonstrate the predictive performance of LEGS-based networks on graph classification benchmarks, as well as the descriptive quality of their learned features in biochemical graph data exploration tasks. Our results show that LEGS-based networks match or outperforms popular GNNs, as well as the original geometric scattering construction, on many datasets, in particular in biochemical domains, while retaining certain mathematical properties of handcrafted (non-learned) geometric scattering.

I. Introduction

GEOMETRIC deep learning has recently emerged as an increasingly prominent branch of deep learning [1]. At the core of geometric deep learning is the use of graph neural networks (GNNs) in general, and graph convolutional networks (GCNs) in particular, which ensure neuron activations follow the geometric organization of input data by propagating information across graph neighborhoods [2]–[7]. However, recent work has shown the difficulty in generalizing these methods to more complex structures, identifying common problems and phrasing them in terms of so-called oversmoothing [8], underreaching [9] and oversquashing [10].

Using graph signal processing terminology from [4], these issues can be partly attributed to the limited construction of convolutional filters in many commonly used GCN architectures. Inspired by the filters learned in convolutional neural networks, GCNs consider node features as graph signals and aim to aggregate information from neighboring nodes. For example, [4] presented a typical implementation of a GCN with a cascade of averaging (essentially low pass) filters. We note that more general variations of GCN architectures exist [3], [5], [6], which are capable of representing other filters, but as investigated in [10], they often have difficulty in learning long-range connections.

Recently, an alternative approach was presented to provide deep geometric representation learning by generalizing Mallat’s scattering transform [11], originally proposed to provide a mathematical framework for understanding convolutional neural networks, to graphs [12]–[14] and manifolds [15]–[17]. The geometric scattering transform can represent nodes or graphs based on multi-scale diffusions, and differences between scales of diffusions of graph signals (i.e., node features). Similar to traditional scattering, which can be seen as a convolutional network with non-learned wavelet filters, geometric scattering is defined as a GNN with handcrafted graph filters, constructed with diffusion wavelets over the input graph [18], which are then cascaded with pointwise absolute-value nonlinearities. The efficacy of geometric scattering features in graph processing tasks was demonstrated in [12], with both supervised learning and data exploration applications. Moreover, their handcrafted design enables rigorous study of their properties, such as stability to deformations and perturbations, and provides a clear understanding of the information extracted by them, which by design (e.g., the cascaded band-pass filters) goes beyond low frequencies to consider richer notions of regularity [19], [20].

However, while geometric scattering transforms provide effective universal feature extractors, their handcrafted design does not allow the automatic task-driven representation learning that is so successful in traditional GNNs and neural networks in general. Here, we combine both frameworks by incorporating richer multi-frequency band features from geometric scattering into GNNs, while allowing them to be flexible and trainable. We introduce the geometric scattering module, which can be used within a larger neural network. We call this a learnable geometric scattering (LEGS) module and show it inherits properties from the scattering transform while allowing the scales of the diffusion to be learned. Moreover, we show that our framework is differentiable, allowing for backpropagation through it in a standard reverse mode auto differentiation library.

The benefits of our construction over standard GNNs, as well as pure geometric scattering, are discussed and demonstrated on graph classification and regression tasks in Sec. VI. In particular, we find that our network maintains the robustness to small training sets present in fixed geometric scattering [21], while improving performance on biological graph classification and regression tasks. In particular, we find that in tasks where the graphs have a large diameter relative to their size, learnable scattering features improve performance over competing methods. We show that our construction performs better on tasks that require whole-graph representations with an emphasis on biochemical molecular graphs, where relatively large diameters and non-planar structures usually limit the effectiveness of traditional GNNs. We also show that our network maintains performance in social network and other node classification tasks where state-of-the-art GNNs perform well.

A previous short version of this work appeared in the IEEE Workshop on Machine Learning and Signal Processing 2021 [22]. We expand on that work first by incorporating additional theory including Theorem 3 which generalizes existing theory for nonexpansive scattering operators. Furthermore, we add additional experiments on molecular data, as well as ablation studies on both amount of training data and ensembling with other models.

The remainder of this paper is organized as follows. In Section II, we review related work on graph scattering and literature on the challenges of modern GNNs. In Section III, we review some of the concepts of geometric scattering. In Section IV, we present expanded theory on geometric scattering with task-driven tuning. This theory establishes that our LEGS module retains the theoretical properties of scattering while increasing expressiveness. In Section V, we present the architecture and implementation details of the LEGS module. We examine the empirical performance of LEGS architectures in Section VI and conclude in Section VII.

II. Related Work

A widely discussed challenge for many modern GNN approaches is so-called oversmoothing [8], [23]. This is a result of the classic message passing in GNNs that is based on cascades of local node feature aggregations over node neighborhoods. This increasingly smooths graph signals, which in turn renders the graph nodes undistinguishable. From a spectral point of view, this phenomenon is due to most GNN filters being low-pass filters [24] that mostly preserve the low-frequency spectrum. A related phenomenon is so-called underreaching [9], which is a result of the limited spatial support of most GNN architectures. Most models can only relate information from nodes within a distance equal to the number of layers. Hence, they cannot represent long-range interactions as the before mentioned oversmoothing typically prohibits the design of truly “deep” GNN architectures. Lastly, oversquashing [10] is yet another consequence of typical message passing. As the number of nodes in the receptive field of each node grows exponentially in the number of GNN layers, a huge amount of information needs to be compressed into a vector of fixed size. This makes it difficult to represent meaningful relationships between nodes, as the contribution of features of single nodes becomes marginal.

We note that efforts to improve the capture of long-range connections in graph representation learning have recently yielded several spectral approaches based on using the Lancoz algorithm to approximate graph spectra [25], or based on learning in block Krylov subspaces [26]. Such methods are complementary to the work presented here, in that their spectral approximation can also be applied in the computation of geometric scattering when considering very long range scales (e.g., via spectral formulation of graph wavelet filters). However, we find that such approximations are not necessary in the datasets considered here and in other recent work focusing on whole-graph tasks, where direct computation of polynomials of the Laplacian is sufficient. Furthermore, recent attempts have also considered ensemble approaches with hybrid architectures that combine GCN and scattering channels [27], albeit primarily focused on node-level tasks, considered on a single graph at a time, rather than whole-graph tasks considered here on datasets comparing multiple graphs. Such ensemble approaches are also complimentary to the proposed approach in that hybrid architectures can also be applied in conjunction with the proposed LEGS module here as we demonstrate in Sec. VI.

III. Preliminaries: Geometric Scattering

Let $𝒢=(V,E,w)$ be a weighted graph with $V:=\left\{v_1,\dots ,v_n\right\}$ the set of nodes, $E\subset \{\left\{v_i,v_j\right\}\in \left(\genfrac{}{}{0pt}{}{V}{2}\right),i\ne j\}$ the set of (undirected) edges and $w:E\to (0,\mathrm{\infty })$ assigning (positive) edge weights to the graph edges. We define a graph signal as a function $x:V\to \mathbb{R}$ on the nodes of $𝒢$ and aggregate them in a signal vector $\mathit{x}\in {\mathbb{R}}^n$ with the i^th entry being $x\left(v_i\right)$ We define the weighted adjacency matrix $\mathit{W}\in {\mathbb{R}}^{n\times n}$ of $𝒢$ as $\mathit{W}\left[v_i,v_j\right]:=w\left(v_i,v_j\right)$ if $\left\{v_i,v_j\right\}\in E$, and 0 otherwise and the degree matrix $\mathit{D}\in {\mathbb{R}}^{n\times n}$ of $𝒢$ as $\mathit{D}:=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(d_1,\dots ,d_n\right)$ with $d_i:=\mathrm{deg}\left(v_i\right):={\sum }_{j=1}^n\mathit{W}\left[v_i,v_j\right]$ the degree of node $v_i$.

The geometric scattering transform [12] consists of a cascade of graph filters constructed from a left stochastic diffusion matrix $\mathit{P}:=\frac{1}{2}\left({\mathit{I}}_n+\mathit{W}{\mathit{D}}^{-1}\right)$, which corresponds to transition probabilities of a lazy random walk Markov process. The laziness of the process signifies that at each step it has equal probability of staying at the current node or transitioning to a neighbor. Scattering filters are defined via graph-wavelet matrices ${\mathbf{\Psi }}_j\in {\mathbb{R}}^{n\times n}$ of order $j\in {\mathbb{N}}_0$, as

$$\begin{array}{ll}& {\mathbf{\Psi }}_0:={\mathit{I}}_n-\mathit{P},\\ & {\mathbf{\Psi }}_j:={\mathit{P}}^{2^{j-1}}-{\mathit{P}}^{2^j}={\mathit{P}}^{2^{j-1}}\left({\mathit{I}}_n-{\mathit{P}}^{2^{j-1}}\right),j\ge 1.\end{array}$$ (1)

These diffusion wavelet operators partition the frequency spectrum into dyadic frequency bands, which are then organized into a full wavelet filter bank ${𝒲}_J:={\left\{{\mathbf{\Psi }}_j,{\mathbf{\Phi }}_J\right\}}_{0\le j\le J}$, where ${\mathbf{\Phi }}_J:={\mathit{P}}^{2^J}$ is a pure low-pass filter, similar to the one used in GCNs. It is easy to verify that the resulting wavelet transform is invertible since a simple sum of filter matrices in ${𝒲}_J$ yields the identity. Moreover, as discussed in [20], this filter bank forms a nonexpansive frame, which provides energy preservation guarantees, as well as stability to perturbations, and can be generalized to a wider family of constructions that encompasses the variations of scattering transforms on graphs from such as those considered in [13].

Given the wavelet filter bank ${𝒲}_J$, node-level scattering features are computed by stacking cascades of bandpass filters and element-wise absolute value nonlinearities to form

$${\mathit{U}}_p\mathit{x}:={\mathbf{\Psi }}_{j_m}\left|{\mathbf{\Psi }}_{j_{m-1}}\dots \right|{\mathbf{\Psi }}_{j_2}\left|{\mathbf{\Psi }}_{j_1}\mathit{x}\right||\dots |,$$ (2)

parameterized by the scattering path $p:=\left(j_1,\dots ,j_m\right)\in {\cup }_{m\in \mathbb{N}}{\mathbb{N}}_0^m$ that determines the filter scales of each wavelet. Whole-graph representations are obtained by aggregating node-level features via statistical moments over the nodes of the graph [12], which yields the geometric scattering moments

$${\mathit{S}}_{p,q}\mathit{x}:=\sum _{i=1}^n{\left|{\mathit{U}}_p\mathit{x}\left[v_i\right]\right|}^q,$$ (3)

indexed by the scattering path $p$ and moment order $q$. Notably, coefficients with $m=0$ correspond to simply taking moments of the input signal $\mathbf{x}$ and, the coefficients with $m=1$ correspond to moments of the wavelet transform. For $m\ge 2$, the coefficients correspond to iterated wavelet transforms with non-linearities interspersed. Finally, we note that Theorem 3.6 of [20] shows that ${\mathit{U}}_p$ is equivariant to permutations of the nodes. Therefore, it follows that the graph-level scattering transform ${\mathit{S}}_{p,q}$ is node-permutation invariant since it is defined via global summation.

IV. Adaptive Geom. Scattering Relaxation

The geometric scattering construction, described in Sec. III, can be seen as a particular GNN architecture with handcrafted layers, rather than learned ones. This provides a solid mathematical framework for understanding the encoding of geometric information in GNNs [20], while also providing effective unsupervised graph representation learning for data exploration, which also has advantages in supervised learning tasks [12].

Both [20] and [12] used dyadic scales in Eq. 1, a choice inspired by the Euclidean scattering transform [11]. Below in Theorem 1, we will show that these dyadic scales may be replaced by any increasing sequence of scales and the resulting wavelets will still form a nonexpansive frame. Later in Section V, Theorem 3 will consider scales which are learned from data and show that the learned filter bank forms a nonexpansive operator under mild assumptions. This allows us to obtain a flexible model with similar guarantees to the model considered in [12], but which is amenable to task-driven tuning provided by end-to-end GNN training.

Given an increasing sequence of integer diffusion time scales $0<t_1<\cdots <t_J$, we replace the wavelets considered in Eq. 1 with the generalized filter bank ${𝒲}_J^{\prime }:={\left\{{\mathbf{\Psi }}_j^{\prime },{\mathbf{\Phi }}_J^{\prime }\right\}}_{j=0}^{J-1}$, where

$$\begin{array}{ll}& {\mathbf{\Psi }}_0^{\prime }:={\mathit{I}}_n-{\mathit{P}}^{t_1},{\mathbf{\Phi }}_J^{\prime }:={\mathit{P}}^{t_J},\\ & {\mathbf{\Psi }}_j^{\prime }:={\mathit{P}}^{t_j}-{\mathit{P}}^{t_{j+1}},1\le j\le J-1.\end{array}$$ (4)

Since $\mathit{P}$ is not a symmetric matrix, the these wavelets are not self-adjoint operators on the standard, unweighted inner product space. Therefore, to study these wavelets, it will be convenient to consider a weighted inner product space $L^2(𝒢,{\mathit{D}}^{-\frac{1}{2}})$ of graph signals with inner product $⟨\mathit{x},\mathit{y}⟩:=⟨{\mathit{D}}^{-\frac{1}{2}}\mathit{x},{\mathit{D}}^{-\frac{1}{2}}\mathit{y}⟩$ and induced norm $\parallel \mathit{x}{\parallel }_{{\mathit{D}}^{-\frac{1}{2}}}^2={∥{\mathit{D}}^{-\frac{1}{2}}\mathit{x}∥}_2={\sum }_{i=1}^n\frac{\mathit{x}[i{]}^2}{d_i}$ for $\mathit{x},\mathit{y}\in L^2(𝒢,{\mathit{D}}^{-\frac{1}{2}})$. Importantly, we note that Lemma 1.1 of [15] implies that $\mathit{P}$ (and therefore the wavelets constructed from it) are self-adjoint on $L^2(𝒢,{\mathit{D}}^{-\frac{1}{2}})$.

The following theorem shows that ${𝒲}_J^{\prime }$ is a nonexpansive frame, similar to the result shown for dyadic scales in [20]. We note our proof that it relies upon the fact that ${\mathit{D}}^{-1/2}\mathit{P}{\mathit{D}}^{1/2}$ is a symmetric matrix, which necessitates that our results be in terms of the weighted norm $\parallel \cdot {\parallel }_{{\mathit{D}}^{-1/2}}$ rather than the standard unweighted inner product.

Theorem 1.

There exists a constant $C>0$ that only depends on $t_1$ and $t_J$ such that for all $\mathit{x}\in L^2(𝒢,{\mathit{D}}^{-1/2})$,

$$C{\Vert \mathit{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2⩽{\Vert {\mathbf{\Phi }}_J^{\prime }\mathit{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2+\sum _{j=0}^J{\Vert {\mathbf{\Psi }}_j^{\prime }\mathit{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2⩽{\Vert \mathit{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2,$$

where the norm is the one induced by the space $L^2(𝒢,{\mathit{D}}^{-1/2})$.

Proof.

Note that $\mathit{P}$ has a symmetric conjugate $\mathit{M}:={\mathit{D}}^{-1/2}\mathit{P}{\mathit{D}}^{1/2}$ with eigendecomposition $\mathit{M}=\mathit{Q}\mathbf{\Lambda }{\mathit{Q}}^{\mathrm{\top }}$ for orthogonal $\mathit{Q}$. Given this decomposition, we can write

$${\mathbf{\Phi }}_J^{\prime }={\mathit{D}}^{1/2}\mathit{Q}{\mathbf{\Lambda }}^{t_J}{\mathit{Q}}^{\mathrm{\top }}{\mathit{D}}^{-1/2},$$

$${\mathbf{\Psi }}_j^{\prime }={\mathit{D}}^{1/2}\mathit{Q}\left({\mathbf{\Lambda }}^{t_j}-{\mathbf{\Lambda }}^{t_{j+1}}\right){\mathit{Q}}^{\mathrm{\top }}{\mathit{D}}^{-1/2},0\le j\le J-1,$$

where we set $t_0=0$ to simplify notations. Therefore, we have

$${∥{\mathbf{\Phi }}_J^{\prime }\mathit{x}∥}_{{\mathit{D}}^{-1/2}}^2={⟨{\mathbf{\Phi }}_J^{\prime }\mathit{x},{\mathbf{\Phi }}_J^{\prime }\mathit{x}⟩}_{{\mathit{D}}^{-1/2}}={∥{\mathbf{\Lambda }}^{t_J}{\mathit{Q}}^{\mathrm{\top }}{\mathit{D}}^{-1/2}\mathit{x}∥}_2^2.$$

If we consider a change of variable to $\mathit{y}={\mathit{Q}}^{\mathrm{\top }}{\mathit{D}}^{-1/2}\mathit{x}$, we have $\parallel \mathit{x}{\parallel }_{{\mathit{D}}^{-1/2}}^2={∥{\mathit{D}}^{-1/2}\mathit{x}∥}_2^2=\parallel \mathit{y}{\parallel }_2^2$, while ${∥{\mathbf{\Phi }}_J^{\prime }\mathit{x}∥}_{{\mathit{D}}^{-1/2}}^2={∥{\mathrm{\Lambda }}^{t_J}\mathit{y}∥}_2^2$. Similarly, we can also reformulate the operations of the other filters in terms of diagonal matrices applied to $\mathit{y}$ as ${∥{\mathbf{\Psi }}_j^{\prime }\mathit{x}∥}_{{\mathit{D}}^{-1/2}}^2={∥\left({\mathrm{\Lambda }}^{t_j}-{\mathrm{\Lambda }}^{t_{j+1}}\right)\mathit{y}∥}_2^2$.

Given these reformulations, we can now write

$$\begin{array}{l}{∥{\mathrm{\Lambda }}^{t_J}\mathit{y}∥}_2^2+\sum _{j=0}^{J-1}{∥\left({\mathrm{\Lambda }}^{t_j}-{\mathrm{\Lambda }}^{t_{j+1}}\right)\mathit{y}∥}_2^2=\\ \sum _{i=1}^n{\mathit{y}}_i^2\cdot \left({\lambda }_i^{2t_J}+{\sum }_{j=0}^{J-1}{\left({\lambda }_i^{t_j}-{\lambda }_i^{t_{j+1}}\right)}^2\right).\end{array}$$

Since $0\le {\lambda }_i\le 1$ and $0=t_0<t_1<\cdots <t_J$, we have

$${\lambda }_i^{2t_J}+\sum _{j=0}^{J-1}{\left({\lambda }_i^{t_j}-{\lambda }_i^{t_{j+1}}\right)}^2\le {\left({\lambda }_i^{t_J}+\sum _{j=0}^{J-1}{\lambda }_i^{t_j}-{\lambda }_i^{t_{j+1}}\right)}^2\le 1,$$

which yields the upper bound ${‖{\text{Λ}}^{t_J}\mathit{y}‖}_2^2+{\sum }_{j=0}^{J-1}‖\left({\text{Λ}}^{t_j}-\right.\left.{\text{Λ}}^{t_{j+1}}\right)\mathit{y}‖{ }_2^2\le {‖\mathit{y}‖}_2^2$. On the other hand, since $t_1>0=t_0$,

$${\lambda }_i^{2t_J}+\sum _{j=0}^{J-1}{\left({\lambda }_i^{t_j}-{\lambda }_i^{t_{j+1}}\right)}^2\ge {\lambda }_i^{2t_J}+{\left(1-{\lambda }_i^{t_1}\right)}^2,$$

and thus, setting $C:={\mathrm{m}\mathrm{i}\mathrm{n}}_{0\le \xi \le 1}\left({\xi }^{2t_J}+{\left(1-{\xi }^{t_1}\right)}^2\right)>0$, we get the lower bound ${∥{\mathrm{\Lambda }}^{t_J}\mathit{y}∥}_2^2+{\sum }_{j=0}^{J-1}{∥\left({\mathrm{\Lambda }}^{t_j}-{\mathrm{\Lambda }}^{t_{j+1}}\right)\mathit{y}∥}_2^2\ge C\parallel \mathit{y}{\parallel }_2^2$. Applying the reverse change of variable to $\mathit{x}$ and $L^2(𝒢,{\mathit{D}}^{-1/2})$ yields the result of the theorem.

Theorem 1 shows that the wavelet transform is injective and stable to additive noise. Our next theorem shows that it is permutation equivariant at the node-level and permutation invariant at the graph level. This guarantees that the extracted features encode the intrinsic geometry of the graph rather than a priori indexation.

Theorem 2.

Let ${\mathit{U}}_p^{\prime }$ and ${\mathit{S}}_{p,q}^{\prime }$ be defined as in Eq. 2–3, with the filters from ${𝒲}_J^{\prime }$ with an arbitrary configuration $0<t_1<\cdots <t_J$ in place of ${𝒲}_J$. Then, for any permutation $\mathrm{\Pi }$ over the nodes and any graph signal $\mathit{x}\in L^2(𝒢,{\mathit{D}}^{-1/2})$, we have ${\mathit{U}}_p^{\prime }\mathrm{\Pi }\mathit{x}=\mathrm{\Pi }{\mathit{U}}_p^{\prime }\mathit{x}$ and ${\mathit{S}}_{p,q}^{\prime }\mathrm{\Pi }\mathit{x}={\mathit{S}}_{p,q}^{\prime }\mathit{x}$, for $p\in {\cup }_{m\in \mathbb{N}}{\mathbb{N}}_0^m,$ $q\in \mathbb{N}$, where geometric scattering implicitly considers the node ordering supporting its input signal.

Proof.

For any permutation $\mathrm{\Pi }$, we let $\overline{𝒢}=\mathrm{\Pi }(𝒢)$ be the graph obtained by permuting the vertices of $𝒢$ with $\mathrm{\Pi }$. We note that the adjacency and degree matrices on $\overline{𝒢}$ are given by $\overline{\mathit{W}}=\mathrm{\Pi }\mathit{W}{\mathrm{\Pi }}^{\mathrm{\top }}$ and $\overline{\mathit{D}}=\mathrm{\Pi }\mathit{D}{\mathrm{\Pi }}^{\mathrm{\top }}$. Additionally, the corresponding permutation operation on a graph signal $\mathit{x}\in L^2(𝒢,{\mathit{D}}^{-1/2})$ gives a signal $\mathrm{\Pi }\mathit{x}\in L^2(\overline{𝒢},{\overline{\mathit{D}}}^{-1/2})$, which we implicitly considered in the statement of the theorem, without specifying these notations for simplicity. Rewriting the statement of the theorem more formally, we let $\overline{\mathit{P}}=\frac{1}{2}({\mathit{I}}_n+{\overline{\mathit{W}\mathit{D}}}^{-1})$ denote the analog of $\mathit{P}$ on $\overline{𝒢}$ and let $\overline{{\mathbf{\Psi }}_j^{\prime }}$ denote the matrix defined according to Eq. 4 but with $\overline{\mathit{P}}$ in place of $\mathit{P}$. Then, we let ${\overline{\mathit{U}}}_p^{\prime }$ and ${\overline{\mathit{S}}}_{p,q}^{\prime }$ be the analogs of ${\mathit{U}}_p^{\prime }$ and ${\mathit{S}}_{p,q}^{\prime }$ constructed using $\overline{{\mathbf{\Psi }}_j^{\prime }}$. With the introduced notations, we aim to show that ${\overline{\mathit{U}}}_p^{\prime }\mathrm{\Pi }\mathit{x}=\mathrm{\Pi }{\mathit{U}}_p^{\prime }\mathit{x}$ and ${\overline{\mathit{S}}}_{p,q}^{\prime }\mathrm{\Pi }\mathit{x}={\mathit{S}}_{p,q}^{\prime }\mathit{x}$.

First, we notice that for any ${\mathbf{\Psi }}_j,$ $0<j<J$ we have that ${\overline{\mathbf{\Psi }}}_j\mathrm{\Pi }\mathit{x}=\mathrm{\Pi }{\mathbf{\Psi }}_j\mathit{x}$, as for $1\le j\le J-1$,

$$\begin{array}{l}{\overline{\mathbf{\Psi }}}_j\mathrm{\Pi }\mathit{x}={\left({\left(\mathrm{\Pi }\mathit{P}{\mathrm{\Pi }}^{\mathrm{\top }}\right)}^{t_j}-\left(\mathrm{\Pi }\mathit{P}{\mathrm{\Pi }}^{\mathrm{\top }}\right)\right)}^{t_{j+1}}\mathrm{\Pi }\mathit{x}\\ & =\left(\mathrm{\Pi }{\mathit{P}}^{t_j}{\mathrm{\Pi }}^{\mathrm{\top }}-\mathrm{\Pi }{\mathit{P}}^{t_{j+1}}{\mathrm{\Pi }}^{\mathrm{\top }}\right)\mathrm{\Pi }\mathit{x}=\mathrm{\Pi }{\mathbf{\Psi }}_j\mathit{x},\end{array}$$

with similar reasoning for $j\in \{0,J\}$. Note that the element-wise absolute value yields $\left|\mathrm{\Pi }\mathit{x}\right|=\mathrm{\Pi }\left|\mathit{x}\right|$ for any permutation matrix $\mathrm{\Pi }$. These two observations inductively yield

$$\begin{array}{l}{\overline{\mathit{U}}}_p^{\prime }\mathrm{\Pi }\mathit{x}={\overline{\mathbf{\Psi }}}_{j_m}^{\prime }\left|{\overline{\mathbf{\Psi }}}_{j_{m-1}}^{\prime }\dots \right|{\overline{\mathbf{\Psi }}}_{j_2}^{\prime }\left|{\overline{\mathbf{\Psi }}}_{j_1}^{\prime }\mathrm{\Pi }\mathit{x}\right||\dots |\\ & ={\overline{\mathbf{\Psi }}}_{j_m}^{\prime }\left|{\overline{\mathbf{\Psi }}}_{j_{m-1}}^{\prime }\dots \right|{\overline{\mathbf{\Psi }}}_{j_2}^{\prime }\mathrm{\Pi }\left|{\mathbf{\Psi }}_{j_1}^{\prime }\mathit{x}\right||\dots |=\cdots =\mathrm{\Pi }{\mathit{U}}_p^{\prime }\mathit{x}.\end{array}$$

To show ${\mathit{S}}_{p,q}^{\prime }$ is permutation invariant, first notice that for any statistical moment $q>0$, we have $|\mathrm{\Pi }\mathit{x}{|}^q=\mathrm{\Pi }|\mathit{x}{|}^q$ and further, as sums are commutative, ${\sum }_j(\mathrm{\Pi }\mathit{x}{)}_j={\sum }_j{\mathit{x}}_j$. We then have

$${\overline{\mathit{S}}}_{p,q}^{\prime }\mathrm{\Pi }\mathit{x}=\sum _{i=1}^n{\left|{\overline{\mathit{U}}}_p^{\prime }\mathrm{\Pi }\mathit{x}\left[v_i\right]\right|}^q=\sum _{i=1}^n{\left|\mathrm{\Pi }{\mathit{U}}_p^{\prime }\mathit{x}\left[v_i\right]\right|}^q={\mathit{S}}_{p,q}^{\prime }\mathit{x},$$

which completes the proof of the theorem.

We note that the results in Theorems 1–2 and their proofs closely follow the theoretical framework proposed by [20]. We carefully account here for the relaxed learned configuration, which replaces the original handcrafted one there. We also note that conclusions of Theorem 2 remain valid if $|\cdot |$ is replaced with any vertex-wise nonlinearity.

V. A Learnable Geom. Scattering Module

In this section, we show how the generalized geometric scattering construction presented in Sec. IV can be implemented in a data-driven way via a backpropagation-trainable module. Throughout this section, we consider an input graph signal $\mathit{x}\in {\mathbb{R}}^n$ or, equivalently, a collection of graph signals $\mathit{X}\in {\mathbb{R}}^{n\times N_{\ell -1}}$. For simplicity, our theory will focus on the case where there is a single signal $\mathbf{x}$. However, the numerical implementation proceeds in the exact same manner with multiple signals and our theoretical results may also be easily adapted to the multiple signal case. The forward propagation of these signals can be divided into three major submodules. First, a diffusion submodule implements the Markov process that forms the basis of the filter bank and transform. Then, a scattering submodule implements the filters and the corresponding cascade, while allowing the learning of the scales $t_1,\dots ,t_J$. Finally, the aggregation module collects the extracted features to provide a graph and produces the task-dependent output.

The diffusion submodule.

We build a set of $m\in \mathbb{N}$ subsequent diffusion steps of the signal $\mathit{x}$ by iteratively multiplying the diffusion matrix $\mathit{P}$ to the left of the signal, resulting in $\left[\mathit{P}\mathit{x},{\mathit{P}}^2\mathit{x},\dots ,{\mathit{P}}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}\mathit{x}\right]$. Since $\mathit{P}$ is often sparse, for efficiency reasons these filter responses are implemented via an RNN structure consisting of $t_{\text{max}}$ RNN modules. Each module propagates the incoming hidden state ${\mathit{h}}_{t-1},$ $t=1,\dots ,t_{\mathrm{m}\mathrm{a}\mathrm{x}}$ with $\mathit{P}$ with the readout ${\mathit{o}}_t$ equal to the produced hidden state ${\mathit{h}}_t:=\mathit{P}{\mathit{h}}_{t-1},$ ${\mathit{o}}_t:={\mathit{h}}_t$..

The scattering submodule.

Next, we consider the selection of $J\le m$ diffusion scales for the flexible filter bank construction with wavelets defined according to Eq. 5. We found this was the most influential part of the architecture. We experimented with methods of increasing flexibility: 1) Selection of ${\left\{t_j\right\}}_{j=1}^{J-1}$ as dyadic scales (as in Sec. III and Eq. 1), fixed for all datasets (LEGS-FIXED); and 2) Selection of each $t_j$ using softmax and sorting by $j$, learnable per model (LEGS-FCN and LEGS-RBF, depending on output layer explained below).

For the scale selection, we use a selection matrix $\mathit{F}\in {\mathbb{R}}^{J\times t_{\mathrm{m}\mathrm{a}\mathrm{x}}}$ where each row ${\mathit{F}}_{(j,\cdot )},$ $j=1,\dots ,J$ is dedicated to identifying the diffusion scale of the wavelet ${\mathit{P}}^{t_j}$ via a one-hot encoding. This is achieved by setting $\mathit{F}:=\sigma (\mathbf{\Theta })={\left[\sigma \left({\mathit{\theta }}_1\right),\sigma \left({\mathit{\theta }}_2\right),\dots ,\sigma \left({\mathit{\theta }}_J\right)\right]}^{\mathrm{\top }}$, where ${\mathit{\theta }}_j\in {\mathbb{R}}^{t_{\text{max}}}$ constitute the rows of the trainable weight matrix $\mathbf{\Theta }$, and $\sigma$ is the softmax function (see [28], section 6.2.2.3). For $\mathit{\theta }=\left[{\theta }_1,\dots ,{\theta }_{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}\right]\in {\mathbb{R}}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}$ and $\sigma (\mathit{\theta }{)}_i:=e^{{\theta }_i}/{\sum }_j{e}^{{\theta }_j}$, we have

$$\mathit{F}=\left[\begin{array}{ccc}\sigma {\left({\mathit{\theta }}_1\right)}_1& \dots & \sigma {\left({\mathit{\theta }}_1\right)}_{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}\\ \sigma {\left({\mathit{\theta }}_2\right)}_1& \dots & \sigma {\left({\mathit{\theta }}_2\right)}_{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}\\ ⋮& \ddots & ⋮\\ \sigma {\left({\mathit{\theta }}_J\right)}_1& \dots & \sigma {\left({\mathit{\theta }}_J\right)}_{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}\end{array}\right].$$

While this construction may not strictly guarantee an exact one-hot encoding, we assume that the softmax activations yield a sufficient approximation. Further, without loss of generality, we assume that the rows of $\mathit{F}$ are ordered according to the position of the leading “one” activated in every row. In practice, this can be easily enforced by reordering the rows. We now construct the filter bank ${\widetilde{𝒲}}_{\mathit{F}}:={\left\{{\tilde{\mathbf{\Psi }}}_j,{\tilde{\mathbf{\Phi }}}_J\right\}}_{j=0}^{J-1}$ with the filters

$$\begin{array}{l}{\tilde{\mathbf{\Psi }}}_0\mathit{x}=\mathit{x}-\sum _{t=1}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}{\mathit{F}}_{(1,t)}{\mathit{P}}^t\mathit{x},{\tilde{\mathbf{\Phi }}}_J\mathit{x}=\sum _{t=1}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}{\mathit{F}}_{(J,t)}{\mathit{P}}^t\mathit{x},\\ {\tilde{\mathbf{\Psi }}}_j\mathit{x}=\sum _{t=1}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}[{\mathit{F}}_{(j,t)}{\mathit{P}}^t\mathit{x}-{\mathit{F}}_{(j+1,t)}{\mathit{P}}^t\mathit{x}],1\le j\le J-1,\end{array}$$ (5)

matching and implementing the construction of ${𝒲}_J^{\prime }$ (Eq. 4). We further illustrate the relationship between $\mathit{F}$ and ${\widetilde{𝒲}}_{\mathit{F}}$ in Sec. X of the appendix. The following theorem shows that ${\widetilde{𝒲}}_{\mathit{F}}:={\left\{{\tilde{\mathbf{\Psi }}}_j,{\tilde{\mathbf{\Phi }}}_J\right\}}_{j=0}^{J-1}$ is a nonexpansive operator under the assumption that the rows of $\mathbf{F}$ have disjoint support. Since softmax only leads to approximately sparse rows, this condition will in general only hold approximately. However, it can be easily achieved via post-processing, e.g. via hard sampling using the Gumbel-softmax [29] in each row of $\mathbf{F}$ as long as $J<t_{max}$.

Theorem 3.

Suppose that the rows of $\mathbf{F}$ have disjoint support and that every element of $supp\left({\mathbf{F}}_{j,\cdot }\right)$ is less than every element of $supp\left({\mathbf{F}}_{j+1,\cdot }\right)$ for all $1\le j\le J-1$. Then ${\widetilde{𝒲}}_{\mathit{F}}$ is a nonexpansive operator, i.e.

$$\sum _{j=0}^J{\Vert {\tilde{\mathbf{\Psi }}}_j\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2+{\Vert {\tilde{\mathbf{\Phi }}}_J\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2\le {\Vert \mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2$$ (6)

The proof of Theorem 3 relies on applying Jensen’s inequality and then Theorem 1. We provide full details in the Section XI of the appendix. We note that since the proof of Theorem 3 relies upon Jensen’s inequality, we are not able to provide a lower bound in equation 6. However, we view the upper bound as more important for analyzing the theoretical properties of scattering transforms built using the filter bank ${\widetilde{𝒲}}_{\mathit{F}}$. Indeed, given Theorem 3, one may imitate the proof of Theorem 3.2 of [20] (and also analogous results in e.g., [11] and [14]) to easily derive upper bounds for scattering transforms constructed from ${\widetilde{𝒲}}_{\mathit{F}}$ using the fact that $\left|\right|a|-|b\left|\right|\le |a-b|$). However, even if one were to obtain a lower-bound in equation 6, one would not be able to transfer this lower-bound to the scattering transform since $|\cdot |$ is not injective.

The aggregation submodule.

While many approaches may be applied to aggregate node-level features into graph-level features such as max, mean, sum pooling, or the more powerful TopK [21] and attention pooling [30], we follow the statistical-moment aggregation explained in Secs. III–IV and leave exploration of other pooling methods to future work.These moments are a natural choice because they were shown to be effective in the (fixed) geometric scattering transform in [12]. The use of multiple moments together can be thought of as capturing the mean, standard deviation, and skew, and kurtosis of the ${\mathit{U}}_p\mathit{x}\left[v_i\right]$ over the graph (since these statistical quantities can all be computed from the first four moments of a random variable).

A. Incorporating LEGS into a larger neural network

As shown in [12] on graph classification, this aggregation works particularly well in conjunction with support vector machines (SVMs) based on the radial basis function (RBF) kernel. Here, we consider two configurations for the task-dependent output layer of the network, either using two fully connected layers after the learnable scattering layers, which we denote LEGS-FCN, or using a modified RBF network [31], which we denote LEGS-RBF, to produce the final classification.

The latter configuration more accurately processes scattering features as shown in Table II. Our RBF network works by first initializing a fixed number of movable anchor points. Then, for every point, new features are calculated based on the radial distances to these anchor points. In previous work on radial basis networks these anchor points were initialized independent of the data. We found that this led to training issues if the range of the data was not similar to the initialization of the centers. Instead, we first use a batch normalization layer to constrain the scale of the features and then pick anchors randomly from the initial features of the first pass through our data. This gives an RBF-kernel network with anchors that are always in the range of the data. Our RBF layer is then $\mathrm{R}\mathrm{B}\mathrm{F}(\mathit{x})=\varphi (\parallel \mathrm{B}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(\mathit{x})-\mathit{c}\parallel )$ with $\varphi (\mathit{x})=e^{-\parallel \mathit{x}{\parallel }^2}$. Where $\mathit{c}$ is the SVM margin, which is set to 1 by default.

TABLE II.

Accuracy (mean ± std.) over 10 test sets on biochemical (top) and social network (bottom) datasets.

	LEGS-RBF	LEGS-FCN	LEGS-ATTN-FCN	LEGS-FIXED	GCN	GraphSAGE	GAT	GIN	GS-SVM	Baseline
DD	72.58 ± 3.35	72.07 ± 2.37	70.93 ± 4.81	69.09 ± 4.82	67.82 ± 3.81	66.37 ± 4.45	68.50 ± 3.62	42.37 ± 4.32	72.66 ± 4.94	75.98 ± 2.81
ENZYMES	36.33 ± 4.50	38.50 ± 8.18	33.72 ± 6.45	32.33 ± 5.04	31.33 ± 6.89	15.83 ± 9.10	25.83 ± 4.73	36.83 ± 4.81	27.33 ± 5.10	20.50 ± 5.99
MUTAG	33.51 ± 4.34	82.98 ± 9.85	84.60 ± 6.13	81.84 ± 11.24	79.30 ± 9.66	81.43 ± 11.64	79.85 ± 9.44	83.57 ± 9.68	85.09 ± 7.44	79.80 ± 9.92
NCI1	74.26 ± 1.53	70.83 ± 2.65	68.62 ± 2.37	71.24 ± 1.63	60.80 ± 4.26	57.54 ± 3.33	62.19 ± 2.18	66.67 ± 2.90	69.68 ± 2.38	56.69 ± 3.07
NCI109	72.47 ± 2.11	70.17 ± 1.46	64.58 ± 5.26	69.25 ± 1.75	61.30 ± 2.99	55.15 ± 2.58	61.28 ± 2.24	65.23 ± 1.82	68.55 ± 2.06	57.38 ± 2.20
PROTEINS	70.89 ± 3.91	71.06 ± 3.17	67.69 ± 4.31	67.30 ± 2.94	74.03 ± 3.20	71.87 ± 3.50	73.22 ± 3.55	75.02 ± 4.55	70.98 ± 2.67	73.22 ± 3.76
PTC	57.26 ± 5.54	56.92 ± 9.36	49.85 ± 11.37	54.31 ± 6.92	56.34 ± 10.29	55.22 ± 9.13	55.50 ± 6.90	55.82 ± 8.07	56.96 ± 7.09	56.71 ± 5.54
COLLAB	75.78 ± 1.95	75.40 ± 1.80	64.29 ± 5.82	72.94 ± 1.70	73.80 ± 1.73	76.12 ± 1.58	72.88 ± 2.06	62.98 ± 3.92	74.54 ± 2.32	64.76 ± 2.63
IMDB-BINARY	64.90 ± 3.48	64.50 ± 3.50	53.17 ± 4.35	64.30 ± 3.68	47.40 ± 6.24	46.40 ± 4.03	45.50 ± 3.14	64.20 ± 5.77	66.70 ± 3.53	47.20 ± 5.67
IMDB-MULTI	41.93 ± 3.01	40.13 ± 2.77	32.60 ± 9.52	41.67 ± 3.19	39.33 ± 3.13	39.73 ± 3.45	39.73 ± 3.61	38.67 ± 3.93	42.13 ± 2.53	39.53 ± 3.63
REDDIT-BINARY	86.10 ± 2.92	78.15 ± 5.42	81.74 ± 6.62	85.00 ± 1.93	81.60 ± 2.32	73.40 ± 4.38	73.35 ± 2.27	71.40 ± 6.98	85.15 ± 2.78	69.30 ± 5.08
REDDIT-MULTI-12K	38.47 ± 1.07	38.46 ± 1.31	28.26 ± 3.15	39.74 ± 1.31	42.57 ± 0.90	32.17 ± 2.04	32.74 ± 0.75	24.45 ± 5.52	39.79 ± 1.11	22.07 ± 0.98
REDDIT-MULTI-5K	47.83 ± 2.61	46.97 ± 3.06	41.37 ± 5.34	47.17 ± 2.93	52.79 ± 2.11	45.71 ± 2.88	44.03 ± 2.57	35.73 ± 8.35	48.79 ± 2.95	36.41 ± 1.80

B. Backpropagation through the LEGS module

The LEGS module is fully suitable for incorporation in any neural network architecture and can be back-propagated through. To show this we write the partial derivatives with respect to the LEGS module parameters $\mathbf{\Theta }$ and $\mathit{W}$ here. The gradients of the filter ${\tilde{\mathbf{\Psi }}}_j,$ $j\ge 1$, with respect to scale weights ${\theta }_k,k=1,\dots J$, where $\sigma$ is the softmax function, can be written as

$$\frac{\partial {\tilde{\mathbf{\Psi }}}_j}{\partial {\mathbf{\Theta }}_{k,t}}=\left\{\begin{array}{ll}{\mathit{P}}^t\sigma {\left({\mathit{\theta }}_j\right)}_t{\left(1-\sigma \left({\mathit{\theta }}_j\right)\right)}_t& \text{if}k=j,\\ {\mathit{P}}^t\sigma {\left({\mathit{\theta }}_j\right)}_t\sigma {\left({\mathit{\theta }}_k\right)}_t& \text{if}k=j+1,\\ 0_{n\times n}& \text{else.}\end{array}\right.$$ (7)

Finally, we note that while in this paper we consider the setting where $\mathit{W}$ is fixed, one could also consider variations in which one attempted to learn $\mathit{W}$. In this case, the gradient matrix with respect to the adjacency matrix entry ${\mathit{W}}_{a,b}$ would be given by

$$\begin{array}{l}\frac{\partial {\tilde{\mathbf{\Psi }}}_j}{\partial {\mathit{W}}_{a,b}}=\sum _{t=1}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}[({\mathit{F}}_{j,t}-{\mathit{F}}_{j+1,t})\times \\ \frac{1}{2}\sum _{k=1}^t{\mathit{P}}^{k-1}({\mathit{J}}^{ab}{\mathit{D}}^{-1}+\mathit{W}\frac{\partial {\mathit{D}}^{-1}}{\partial {\mathit{W}}_{a,b}}){\mathit{P}}^{t-k}],\end{array}$$ (8)

where we denote ${\mathit{J}}^{ab}$ the matrix with ${\mathit{J}}_{a,b}^{ab}=1$ and all other entries zero, and $\frac{\partial {\mathit{D}}^{-1}}{\partial {\mathit{W}}_{a,b}}$ is defined in equation 17. In this setting, some tricks and heuristics may be needed to maintain desirable properties of W, (positivity, symmetry, etc.). Gradients of filters ${\tilde{\mathbf{\Phi }}}_J$ and ${\tilde{\mathbf{\Psi }}}_0$, which are simple modifications of the partial derivatives of ${\tilde{\mathbf{\Psi }}}_j$ and derivations of these gradients can be found in Sec. XIII of the appendix.

VI. Empirical Results

We investigate the LEGS module on whole graph classification and graph regression tasks that arise in a variety of contexts, with emphasis on the more complex biochemical datasets. Unlike other types of data, biochemical graphs do not exhibit the small-world structure of social graphs and may have large diameters relative to their size. Further, the connectivity patterns of biomolecules are very irregular due to 3D folding and long-range connections, and thus ordinary local node aggregation methods may miss such connectivity differences.

A. Whole Graph Classification

We perform whole graph classification by using eccentricity (max distance of a node to other nodes) and clustering coefficient (percentage of links between the neighbors of the node compared to a clique) as node features as are used in [12]. We compare against graph convolutional networks (GCN) [4], GraphSAGE [5], graph attention network (GAT) [30], graph isomorphism network (GIN) [6], Snowball network [26], and fixed geometric scattering with a support vector machine classifier (GS-SVM) as in [12], and a baseline which is a 2-layer neural network on the features averaged across nodes (disregarding graph structure).

These comparisons are meant to inform when including learnable graph scattering features are helpful in extracting whole graph features. Specifically, we are interested in the types of graph datasets where existing graph neural network performance can be improved upon with scattering features. We evaluate these methods across 7 biochemical datasets and 6 social network datasets where the goal is to classify between two or more classes of compounds with hundreds to thousands of graphs and tens to hundreds of nodes (See Table I). For more specific information on individual datasets see Appendix XIV. We use 10-fold cross validation on all models which is elaborated on in Section XV of the supplement.

TABLE I.

Dataset statistics, diameter, nodes, edges, clustering coefficient (CC) averaged over all graphs. Split into bio-chemical and social network types.

	# Graphs	# Classes	Diameter	Nodes	Edges	CC
DD	1178	2	19.81	284.32	715.66	0.48
ENZYMES	600	6	10.92	32.63	62.14	0.45
MUTAG	188	2	8.22	17.93	19.79	0.00
NCI1	4110	2	13.33	29.87	32.30	0.00
NCI109	4127	2	13.14	29.68	32.13	0.00
PROTEINS	1113	2	11.62	39.06	72.82	0.51
PTC	344	2	7.52	14.29	14.69	0.01
COLLAB	5000	3	1.86	74.49	2457.22	0.89
IMDB-B	1000	2	1.86	19.77	96.53	0.95
IMDB-M	1500	3	1.47	13.00	65.94	0.97
REDDIT-B	2000	2	8.59	429.63	497.75	0.05
REDDIT-12K	11929	11	9.53	391.41	456.89	0.03
REDDIT-5K	4999	5	10.57	508.52	594.87	0.03

LEGS outperforms on biochemical datasets.

Most work on graph neural networks has focused on social networks which have a well-studied structure. However, biochemical graphs that represent molecules and tend to be overall smaller and less connected than social networks (see Table I). In particular, we find that LEGS outperforms other methods by a significant margin on biochemical datasets with relatively small but high diameter graphs (NCI1, NCI109, ENZYMES, PTC), as shown in Table II. On extremely small graphs we find that GS-SVM performs best, which is likely because other methods with more parameters can more easily overfit the data. We reason that the performance increase exhibited by LEGS module networks, and to a lesser extent GS-SVM, on these biochemical graphs is due the ability of geometric scattering to compute complex connectivity features via its multiscale diffusion wavelets. Thus, methods that rely on a scattering construction would in general perform better, with the flexibility and trainability of the LEGS module giving it an edge on most tasks.

LEGS performs well on social network datasets and considerably improves performance in ensemble models.

In Table II, we see that on the social network datasets LEGS performs well. We note that one of the scattering style networks (either LEGS or GS-SVM) is either the best or second best on each dataset. On each of these datasets, LEGS-RBF (which uses a SVM with a radial basis function similar to GS-SVM) and GS-SVM are well within one standard deviation of each other. If we also consider combining LEGS module features with GCN features the LEGS module performs the best on five out of six of the social network datasets. Across all datasets, an ensemble model considerably increases accuracy over GCN (see Table VIII). This underlines the capabilities of the LEGS module, not only as an isolated model, but also as a tool for powerful hybrid GNN architectures. Similar to [27], this supports the claim that the LEGS module (due to geometric scattering) is sensitive to complementary regularity over graphs, compared to many traditional GNNs. That is, many traditional GNNs focus on low-frequency information, whereas the bandpass wavelet filters are also able to capture high-frequency information. Since the two network types capture different information, using them together achieves better performance than either network by itself.

TABLE VIII.

Mean ± std. test set accuracy on biochemical and social network datasets.

(μ ± σ)	GCN	GCN-LEGS-FIXED	GCN-LEGS-FCN
DD	67.82 ± 3.81	74.02 ± 2.79	73.34 ± 3.57
ENZYMES	31.33 ± 6.89	31.83 ± 6.78	35.83 ± 5.57
MUTAG	79.30 ± 9.66	82.46 ± 7.88	83.54 ± 9.39
NCI1	60.80 ± 4.26	70.80 ± 2.27	72.21 ± 2.32
NCI109	61.30 ± 2.99	68.82 ± 1.80	69.52 ± 1.99
PROTEINS	74.03 ± 3.20	73.94 ± 3.88	74.30 ± 3.41
PTC	56.34 ± 10.29	58.11 ± 6.06	56.64 ± 7.34
COLLAB	73.80 ± 1.73	76.60 ± 1.75	75.76 ± 1.83
IMDB-BINARY	47.40 ± 6.24	65.10 ± 3.75	65.90 ± 4.33
IMDB-MULTI	39.33 ± 3.13	39.93 ± 2.69	39.87 ± 2.24
REDDIT-BINARY	81.60 ± 2.32	86.90 ± 1.90	87.00 ± 2.36
REDDIT-MULTI-12K	42.57 ± 0.90	45.41 ± 1.24	45.55 ± 1.00
REDDIT-MULTI-5K	52.79 ± 2.11	53.87 ± 2.75	53.41 ± 3.07

LEGS preserves enzyme exchange preferences while increasing performance.

One advantage of geometric scattering over other graph embedding techniques lies in the rich information present within the scattering feature space. This was demonstrated in [12] where it was shown that the embeddings created through fixed geometric scattering can be used to accurately infer inter-graph relationships. Scattering features of enzyme graphs within the ENZYMES dataset [33] possessed sufficient global information to recreate the enzyme class exchange preferences observed empirically in [32]. We demonstrate here that LEGS features retain similar descriptive capabilities, as shown in Figure 2. Here we show chord diagrams where the chord size represents the exchange preference between enzyme classes, which is estimated as suggested in [12]. Our results here (and in Table X, which provides complementary quantitative comparison) show that, with relaxations on the scattering parameters, LEGS-FCN achieves better classification accuracy than both LEGS-FIXED and GCN (see Table II) while also retaining a more descriptive embedding that maintains the global structure of relations between enzyme classes. We ran LEGS-FIXED and LEGS-FCN on the ENZYMES dataset. For comparison, we also ran a standard GCN whose graph embeddings were obtained via mean pooling. To infer enzyme exchange preferences from their embeddings, we followed [12] in defining the distance from an enzyme $e$ to the enzyme class ${\mathrm{E}\mathrm{C}}_j$ as $\mathrm{dist}\left(e,{\mathrm{E}\mathrm{C}}_j\right):=∥v_e-{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}_{C_j}\left(v_e\right)∥$, where $v_e$ is the embedding of $e$, and $C_j$ is the PCA subspace of the enzyme feature vectors within ${\mathrm{E}\mathrm{C}}_j$. The distance between the enzyme classes ${\mathrm{E}\mathrm{C}}_i$ and ${\mathrm{E}\mathrm{C}}_j$ is the average of the individual distances, mean $\left\{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\left(e,{\mathrm{E}\mathrm{C}}_j\right):e\in {\mathrm{E}\mathrm{C}}_i\right\}$. From here, the affinity between two enzyme classes is computed as $\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{f}\left({\mathrm{E}\mathrm{C}}_i,{\mathrm{E}\mathrm{C}}_j\right)=w_i/\mathrm{m}\mathrm{i}\mathrm{n}\left(\frac{D_{i,i}}{D_{i,j}},\frac{D_{j,j}}{D_{j,i}}\right)$, where $w_i$ is the percentage of enzymes in class $i$ which are closer to another class than their own, and $D_{i,j}$ is the distance between ${\mathrm{E}\mathrm{C}}_i$ and ${\mathrm{E}\mathrm{C}}_j$.

Fig. 2.

Enzyme class exchange preferences empirically observed in [32], and estimated from LEGS and GCN embeddings.

Robustness to reduced training set size.

We remark that similar to the robustness shown in [12] for handcrafted scattering, LEGS-based networks are able to maintain accuracy even when the training set size is shrunk to as low as 20% of the dataset, with a median decrease of 4.7% accuracy as when 80% of the data is used for training, as discussed in the supplement (see Table III).

TABLE III.

Accuracy (mean ± std.) over test set selection on cross-validated LEGS-RBF Net with reduced training set size.

Train, Val, Test %	80%, 10%, 10%	70%, 10%, 20%	40%, 10%, 50%	20%, 10%, 70%
COLLAB	75.78 ± 1.95	75.00 ± 1.83	74.00 ± 0.51	72.73 ± 0.59
DD	72.58 ± 3.35	70.88 ± 2.83	69.95 ± 1.85	69.43 ± 1.24
ENZYMES	36.33 ± 4.50	34.17 ± 3.77	29.83 ± 3.54	23.98 ± 3.32
IMDB-BINARY	64.90 ± 3.48	63.00 ± 2.03	63.30 ± 1.27	57.67 ± 6.04
IMDB-MULTI	41.93 ± 3.01	40.80 ± 1.79	41.80 ± 1.23	36.83 ± 3.31
MUTAG	33.51 ± 4.34	33.51 ± 1.14	33.52 ± 1.26	33.51 ± 0.77
NCI1	74.26 ± 1.53	74.38 ± 1.38	72.07 ± 0.28	70.30 ± 0.72
NCI109	72.47 ± 2.11	72.21 ± 0.92	70.44 ± 0.78	68.46 ± 0.96
PROTIENS	70.89 ± 3.91	69.27 ± 1.95	69.72 ± 0.27	68.96 ± 1.63
PTC	57.26 ± 5.54	57.83 ± 4.39	54.62 ± 3.21	55.45 ± 2.35
REDDIT-BINARY	86.10 ± 2.92	86.05 ± 2.51	85.15 ± 1.77	83.71 ± 0.97
REDDIT-MULTI-12K	38.47 ± 1.07	38.60 ± 0.52	37.55 ± 0.05	36.65 ± 0.50
REDDIT-MULTI-5K	47.83 ± 2.61	47.81 ± 1.32	46.73 ± 1.46	44.59 ± 1.02

Ensembling LEGS with GCN features improves classification.

Recent work by [27] combines the features from a fixed scattering transform with a GCN network, showing that this has empirical advantages in semi-supervised node classification, and theoretical representation advantages over a standard [4] style GCN. We ensemble the learned features from a learnable scattering network (LEGS-FCN) with those of GCN and compare this to ensembling fixed scattering features with GCN as in [27], as well as the solo features. Our setting is slightly different in that we use the GCN features from pretrained networks, only training a small 2-layer ensembling network on the combined graph level features. This network consists of a batch norm layer, a 128 width fully connected layer, a leakyReLU activation, and a final classification layer down to the number of classes. In Table VIII we see that combining GCN features with fixed scattering features in LEGS-FIXED or learned scattering features in LEGS-FCN always helps classification. Learnable scattering features help more than fixed scattering features overall and particularly in the biochemical domain.

B. Graph Regression

We next evaluate learnable scattering on two graph regression tasks, the QM9 [34] graph regression dataset, and a new task from the critical assessment of structure prediction (CASP) challenge [35]. In the CASP task, the main objective is to score protein structure prediction/simulation models in terms of the discrepancy between their predicted structure and the actual structure of the protein (which is known a priori). The accuracy of such 3D structure predictions are evaluated using a variety of metrics, but we focus on the global distance test (GDT) score [36]. The GDT score measures the similarity between tertiary structures of two proteins with amino acid correspondence. A higher score means two structures are more similar. For a set of predicted 3D structures for a protein, we would like to quantify their quality by the GDT score.

For this task we use the CASP12 dataset [35] and preprocess it similarly to [37], creating a KNN graph between proteins based on 3D coordinates of each amino acid. From this KNN graph we regress against the GDT score. We evaluate on 12 proteins from the CASP12 dataset and choose random (but consistent) splits with 80% train, 10% validation, and 10% test data out of 4000 total structures. We are interested in structure similarity and use no non-structural node features.

LEGS outperforms on all CASP targets.

Across all CASP targets we find that LEGS-based architectures significantly outperform GNN and baseline models. This performance improvement is particularly stark on the easiest structures (measured by average GDT) but is consistent across all structures. In Fig. 3 we show the relationship between percent improvement of LEGS over the GCN model and the average GDT score across the target structures.

Fig. 3.

CASP dataset LEGS-FCN % decrease over GCN in MSE of GDT prediction vs. Average GDT score.

LEGS outperforms on the QM9 dataset.

We evaluate the performance of LEGS-based networks on the quantum chemistry dataset QM9 [34], which consists of 130,000 molecules with ~18 nodes per molecule. We use the node features from [34], with the addition of eccentricity and clustering coefficient features, and ignore the edge features. We whiten all targets to have zero mean and unit standard deviation. We train each network against all 19 targets as provided in the PyTorch geometric package [38], which includes the targets from [34] and [39]. and evaluate the mean squared error on the test set with mean and std. over four runs finding that LEGS improves over existing models (Table V). On more difficult targets (i.e., those with large test error) LEGS-FCN is able to perform better, where on easy targets GIN is the best. We suspect this is due to difficult proteins having more long range connections. We also investigate performance on individual regression targets in Table XI.

TABLE V.

Mean ± std. over four runs of mean squared error over 19 targets for the QM9 dataset, lower is better.

(μ ± σ)	Test MSE
LEGS-FCN	0.216 ± 0.009
LEGS-ATTN-FCN	0.348 ± 0.106
LEGS-FIXED	0.228 ± 0.019
GraphSAGE	0.524 ± 0.224
GCN	0.417 ± 0.061
GIN	0.247 ± 0.037
Baseline	0.533 ± 0.041

Overall, scattering features offer a robust signal over many targets, and while perhaps less flexible (by construction), they achieve good average performance with significantly fewer parameters.

LEGS outperforms on the ZINC15 dataset.

We compared the performance of LEGS-based networks against various architectures using 3 tranches of the ZINC15 [40] dataset in a multi-property prediction task (Tab. VI). The BBAB, FBAB and JBCD tranches contain molecules with molecular weight in the range of 200–250, 350–375 and 450–500 Daltons respectively. All networks were trained using the PyTorch geometric package [38], using a 1-hot encoding of atom type as the node signal. We predicted 10 chemical, physical and structural properties for each molecule, including measures of size, shape, lipophilicity, hydrogen bonding, and polarity. To illustrate the flexibility of the LEGS module, we also include a variant, LEGS-ATTN-FCN, which includes an attention layer between geometric scattering and the fully connected regression network and MP-LEGS-FCN, which applies LEGS to the output of a 3-layer graph messaging passing network. We observe that LEGS-ATTN-FCN and LEGS-FCN are the best performing methods and achieve better performance compared to LEGS-FIXED, MP-LEGS-FCN, and other graph neural networks.

LEGS outperforms on the BindingDB dataset.

We predicted the inhibition coefficient for ligands binding to 2 different targets in the BindingDB dataset [41], namely D(2) dopamine receptor (UniProtKB ID: P14416) and Carbonic anhydrase 2 (UniProtKB ID: P00918). The molecular weight and structural diversity of these ligands was significantly higher compared to the ZINC15 tranches. Learnable scattering networks (LEGS-FCN and LEGS-ATTN-FCN) outperformed GNN and baseline models (Tab. VII). Applying attention to the scattering output did not result in a significant performance improvement over the LEGS-FCN model, perhaps due to the FCN being capable of identifying scattering features pertaining to parts of ligand responsible for binding.

VII. Conclusion

Here we introduced a flexible geometric scattering module, that serves as an alternative to standard graph neural network architectures and is capable of learning rich multi-scale features. Our learnable geometric scattering module allows a task-dependent network to choose the appropriate scales of the multiscale graph diffusion wavelets that are part of the geometric scattering transform. We show that incorporation of this module yields improved performance on graph classification and regression tasks, particularly on biochemical datasets, while keeping strong guarantees on extracted features. This also opens the possibility to provide additional flexibility to the module to enable node-specific or graph-specific tuning via attention mechanisms, which are an exciting future direction, but out of scope for the current work.

Fig. 1.

LEGS module learns to select the appropriate scattering scales from the data.

TABLE IV.

CASP GDT regression error over three seeds

(μ ± σ)	Train MSE	Test MSE
LEGS-FCN	134.34 ± 8.62	144.14 ± 15.48
LEGS-RBF	140.46 ± 9.76	152.59 ± 14.56
LEGS-FIXED	136.84 ± 15.57	160.03 ± 1.81
GCN	289.33 ± 15.75	303.52 ± 18.90
GraphSAGE	221.14 ± 42.56	219.44 ± 34.84
Baseline	393.78 ± 4.02	402.21 ± 21.45

IX. Biography Section

Alexander Tong is a postdoctoral fellow in the Department of Computer Science and Operations Research (DIRO) at the Université de Montréal and is also affiliated with Mila (the Quebec AI institute). He holds an M.Phil. and Ph.D. in computer science from Yale University. His research interests are in machine learning and algorithms. His research focuses on optimal transport and neural network methods for geometric single-cell data using a combination of graph and deep learning methods.

Frederik Wenkel received the B.Sc. degree in Mathematics and the M.Sc. degree in Mathematics at Technical University of Munich, in 2019. He is currently a Ph.D. candidate in Applied Mathematics at Universitéde Montréal (UdeM) and Mila (the Quebec AI institute), working on geometric deep learning. In particular, he is interested in graph neural networks and their applications in domains such as social networks and bio-chemistry.

Dhananjay Bhaskar is a Postdoctoral Research Associate in the Department of Genetics at Yale University and a Yale-Boehringer Ingelheim Biomedical Data Science Fellow. He earned his Ph.D. in Biomedical Engineering from Brown University in 2021. His research combines mathematical modeling, topological data analysis and machine learning for the analysis of multimodal and multiscale data in a variety of biological systems.

Kincaid MacDonald is an undergraduate at Yale studying mathematics, computer science, philosophy, and exploring their practical intersection in artificial intelligence. He is particularly interested in developing mathematical tools to shine light into the “black box” of modern deep neural networks.

Jackson Grady is an undergraduate at Yale College currently pursuing a B.S. in Computer Science. He joined the Krishnaswamy Lab in Summer 2021 to explore his research interests in machine learning. His research focuses on informative representations of graph-structured data and its applications to tackling problems such as drug discovery and analysis of protein folding.

Michael Perlmutter is an Assistant Professor in the the Department of Mathematics at Boise State University. Previously, he has held postdoctoral positions in the Department of Mathematics at UCLA, the Department of Computational Mathematics, Science and Engineering at Michigan State University, and the Department of Statistics and Operations Research at the University of North Carolina at Chapel Hill. He earned his Ph.D. in Mathematics from Purdue University in 2016. His research uses the methods of applied probability and computational harmonic analysis to develop an analyze new methods for data sets with geometric structure.

Smita Krishnaswamy is an Associate Professor in the Department of Genetics at the Yale School of Medicine and in the Department of Computer Science in the Yale School of Applied Science and Engineering, and a core member of the Program in Applied Mathematics. She is also affiliated with the Yale Center for Biomedical Data Science, Yale Cancer Center, and Program in Interdisciplinary Neuroscience. Her research focuses on developing unsupervised machine learning methods (especially graph signal processing and deep-learning) to denoise, impute, visualize and extract structure, patterns and relationships from big, high throughput, high dimensional biomedical data. Her methods have been applied variety of datasets from many systems including embryoid body differentiation, zebrafish development, the epithelial-to-mesenchymal transition in breast cancer, lung cancer immunotherapy, infectious disease data, gut microbiome data and patient data. She was trained as a computer scientist with a Ph.D. from the University of Michigan’s EECS department where her research focused on algorithms for automated synthesis and probabilistic verification of nanoscale logic circuits. Following her time in Michigan, she spent 2 years at IBM’s TJ Watson Research Center as a researcher in the systems division where she worked on automated bug finding and error correction in logic.

Guy Wolf is an associate professor in the Department of Mathematics and Statistics (DMS) at the Université de Montréal (UdeM), a core academic member of Mila (the Quebec AI institute), and holds a Canada CIFAR AI Chair. He is also affiliated with the CRM center of mathematical sciences and the IVADO institute of data valorization. He holds an M.Sc. and a Ph.D. in computer science from Tel Aviv University. Prior to joining UdeM in 2018, he was a postdoctoral researcher (2013–2015) in the Department of Computer Science at École Normale Supérieure in Paris (France), and a Gibbs Assistant Professor (2015–2018) in the Applied Mathematics Program at Yale University. His research focuses on manifold learning and geometric deep learning for exploratory data analysis, including methods for dimensionality reduction, visualization, denoising, data augmentation, and coarse graining. Further, he is particularly interested in biomedical data exploration applications of such methods, e.g., in single cell genomics/proteomics and neuroscience.

X. Illustration of Scale Selection Matrix

A traditional filter bank (Sec. III) ${𝒲}_J:={\left\{{\mathbf{\Psi }}_j,{\mathbf{\Phi }}_J\right\}}_{0\le j\le J}$, for the case $J=4$, would be the result of a scale selection matrix

$$\mathit{F}=\left[\begin{array}{llllllll}1& 0& 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right]\in {\mathbb{R}}^{J\times t_{\mathrm{m}\mathrm{a}\mathrm{x}}}.$$

However, as we derive $\mathit{F}$ from a learned matrix $\mathbf{\Theta }\in {\mathbb{R}}^{J\times t_{\mathrm{m}\mathrm{a}\mathrm{x}}}$, we do not obtain an exact one-hot encoding per row. For example,

$$\mathbf{\Theta }\approx \left[\begin{array}{llllllll}4.1& 0.1& 0.2& 0.1& 0.1& 0.3& 0.1& 0.2\\ 0.2& 0.1& 5.2& 0.1& 0.1& 0.3& 0.1& 0.2\\ 0.1& 0.1& 0.3& 0.1& 4.0& 0.2& 0.2& 0.2\\ 0.3& 0.3& 0.2& 0.1& 0.1& 0.1& 0.1& 5.3\end{array}\right].$$

would yield $\mathit{F}\approx \left[\begin{array}{cccccccc}0.88& 0.02& 0.02& 0.02& 0.02& 0.02& 0.02& 0.02\\ 0.01& 0.01& 0.96& 0.01& 0.01& 0.01& 0.01& 0.01\\ 0.02& 0.02& 0.02& 0.02& 0.87& 0.02& 0.02& 0.02\\ 0.0& 0.011& 0.01& 0.01& 0.01& 0.01& 0.01& 0.96\end{array}\right].$

Then, the output wavelet ${\tilde{\mathbf{\Psi }}}_2$ of a learned filter bank (Sec. V) ${\widetilde{𝒲}}_{\mathit{F}}:={\left\{{\tilde{\mathbf{\Psi }}}_j,{\tilde{\mathbf{\Phi }}}_J\right\}}_{j=0}^{J-1}$ for $J=4$ would have the form

$$\begin{array}{l}{\tilde{\mathbf{\Psi }}}_2\approx -0.01\cdot {\mathit{P}}^1-0.01\cdot {\mathit{P}}^2+0.94\cdot {\mathit{P}}^3-0.01\cdot {\mathit{P}}^4\\ & -0.86\cdot {\mathit{P}}^5-0.01\cdot {\mathit{P}}^6-0.01\cdot {\mathit{P}}^7-0.01\cdot {\mathit{P}}^8\\ \approx {\mathit{P}}^3-{\mathit{P}}^5.\end{array}$$

XI. The Proof of Theorem 3

Proof.

Since each of the rows of $\mathbf{F}$ sums to one, we may define ${\tau }_j,$ $j=0,\dots ,J$ to be an independent random variables with probability distribution given by ${\mathbf{F}}_{j,\cdot }$. Then, by definition

$${\tilde{\mathbf{\Psi }}}_j\mathbf{x}=\sum _{t=1}^{t_{\mathrm{m}\mathrm{a}\mathrm{x}}}{\mathbf{F}}_{(j,t)}{\mathbf{P}}^t\mathbf{x}-{\mathbf{F}}_{(j+1,t)}{\mathbf{P}}^t\mathbf{x}=\mathbb{E}\left[{\mathbf{P}}^{{\tau }_j}\mathbf{x}-{\mathbf{P}}^{{\tau }_{j+1}}\mathbf{x}\right]$$

for for all $1\le j\le J-1$. Similarly, we have

$${\tilde{\mathbf{\Psi }}}_0\mathbf{x}=\mathbb{E}\left[\mathbf{x}-{\mathbf{P}}^{{\tau }_1}\mathbf{x}\right]\text{and}{\tilde{\mathbf{\Phi }}}_J\mathbf{x}=\mathbb{E}{\mathbf{P}}^{{\tau }_J}\mathbf{x}$$

Therefore, by Jensen’s inequality we have

$$\sum _{j=0}^J{\Vert {\tilde{\mathbf{\Psi }}}_j\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2+{\Vert {\tilde{\mathbf{\Phi }}}_J\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2$$ (9)

$$\begin{gathered} ={\Vert \mathbb{E}[\mathbf{x}-{\mathbf{P}}^{{\tau }_1}\mathbf{x}]\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2 \\ +\sum _{j=1}^{J-1}{\Vert \mathbb{E}[{\mathbf{P}}^{{\tau }_j}\mathbf{x}-{\mathbf{P}}^{{\tau }_{j+1}}\mathbf{x}]\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2+{\Vert \mathbb{E}{\mathbf{P}}^{{\tau }_J}\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2 \end{gathered}$$ (10)

$$\le \mathbb{E}[{\Vert \mathbf{x}-{\mathbf{P}}^{{\tau }_1}\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2+\sum _{j=0}^J{\Vert {\mathbf{P}}^{{\tau }_j}\mathbf{x}-{\mathbf{P}}^{{\tau }_{j+1}}\mathbf{x}\Vert }_{{\mathit{D}}^{-\frac{1}{2}}}^2$$ (11)

$$\left.+{∥{\mathbf{P}}^{{\tau }_J}\mathbf{x}∥}_{{\mathit{D}}^{-\frac{1}{2}}}^2\right].$$ (12)

Fig. 4.

Two graphs with similar structure consisting of an 8-cycle. Each cycle has two nodes that are connected to another node, which (with respected to shortest-path distance) are 3 steps (a) and 4 steps (b) apart, respectively.

TABLE IX.

Table indicating of simple handcrafted feature extractors can distinguish the two graphs from Figure 4 using GCN and Scattering aggregations, respectively. Only sufficiently large receptive fields allow to distinguish the graphs

Recept. field	Aggregation matrices M
Radius R	AR (GCN)	P1 – PR	P2 – PR	P3 – PR	P4 – PR
1	✘
2	✘	✘
3	✘	✓	✓
4	✔	✓	✓	✓
5	✔	✓	✓	✓	✓

By our assumptions on the support of the rows of $\mathbf{F}$, we have ${\tau }_j<{\tau }_{j+1}$ for all $j$. Therefore, the conditions of Theorem 1 are satisfied with probability one and so we have

$$\begin{gathered} {∥\mathbf{x}-{\mathbf{P}}^{{\tau }_1}\mathbf{x}∥}_{{\mathit{D}}^{-\frac{1}{2}}}^2+\sum _{j=0}^J{∥{\mathbf{P}}^{{\tau }_j}\mathbf{x}-{\mathbf{P}}^{{\tau }_{j+1}}\mathbf{x}∥}_{{\mathit{D}}^{-\frac{1}{2}}}^2+{∥{\mathbf{P}}^{{\tau }_J}\mathbf{x}∥}_{{\mathit{D}}^{-\frac{1}{2}}}^2 \\ \le \parallel \mathbf{x}{\parallel }_{{\mathit{D}}^{-\frac{1}{2}}}^2 \end{gathered}$$

with probability one. Combining this with completes the proof.

XII. Case study of long-range dependencies

Here, we study the effect of the receptive field of different aggregations on the capacity to capture long-range interactions by the example of two graphs, shown in Figure 4. We aim to distinguish two structurally similar graphs that both contain $n=12$ nodes and consist of an 8-cycle with each containing two nodes which are connected to an additional node. The major difference between the graphs is the shortest-path distance between those two nodes in terms of shortest-path distance, being 4 steps and 5 steps, respectively.

To compare the capacity of different models to differentiate between the graphs, we create a simple feature extractor that leverages the aggregation scheme of a GCN model and of wavelets that resemble those learned by the LEGS module. In particular, we use the node degree $\mathbf{d}\in {\mathbb{R}}^n$ as input signal ($d_i$ is the degree of node $v_i\in V$) and aggregate the signal according to the different approaches. Finally, we calculate the average of the output signal across the graph nodes to obtain a single value for each graph, i.e., $y=\frac{1}{n}{\sum }_{i=1}^n(\mathbf{M}\mathbf{d}{)}_i$, with $\mathbf{M}\in {\mathbb{R}}^{n\times n}$ representing the different aggregation schemes.

For the GCN model, we use the matrix

$$\mathbf{A}:=(\mathbf{D}+\mathbf{I}{)}^{-1/2}(\mathbf{W}+\mathbf{I})(\mathbf{D}+\mathbf{I}{)}^{-1/2}$$

as proposed in [4] and vary the radius radius $R>0$ of the receptive field by setting $\mathbf{M}:={\mathbf{A}}^R$ for $R\in \mathbb{N}$. Similarly we experiment with scattering wavelets of different receptive fields that resemble the wavelets learned by the LEGS module by setting $\mathbf{M}:=:={\mathit{P}}^i-{\mathit{P}}^R$ with $1\le i<R$.¹

Table IX shows that the radius of the receptive field of the aggregation plays an important role in distinguishing the two graphs and that only methods with a receptive field of at least four are able to tell the graphs apart. Notably, GCN-style networks with large receptive fields tend not to perform well because of the oversmoothing problem, but wavelet-based methods do not suffer from this issue [27].

XIII. Gradients of the LEGS module

Here we analyze the gradients of the LEGS module with respect to its outputs. As depicted in Fig. IV, the LEGS module has 3 inputs, $W,$ $\mathrm{\Theta }$, and $\alpha$, and $J$ permutation equivariant output matrices ${\tilde{W}}_{\mathit{F}}={\left\{{\tilde{\mathbf{\Psi }}}_j,{\tilde{\mathbf{\Phi }}}_J\right\}}_{j=0}^{J-1}$. Here we compute partial derivatives of ${\tilde{\mathbf{\Psi }}}_j$ for $1\le j\le J-1$. The other related gradients of ${\tilde{\mathbf{\Psi }}}_0$ and ${\tilde{\mathbf{\Phi }}}_J$ are easily deducible from these. The following partial derivatives and an application of the chain rule yield the gradients presented in the main paper:

$$\frac{\partial {\tilde{\mathbf{\Psi }}}_j}{\partial {\mathit{F}}_{k,t}}=\left\{\begin{array}{ll}{\mathit{P}}^t& \text{if}k=j,\\ -{\mathit{P}}^t& \text{if}k=j+1,\\ 0_{n\times n}& \text{else,}\end{array}\right.$$ (13)

$$\frac{\partial {\tilde{\mathbf{\Psi }}}_j}{\partial {\mathit{P}}^t}=\left({\mathit{F}}_{j,t}-{\mathit{F}}_{j+1,t}\right),$$ (14)

$$\frac{\partial {\mathit{F}}_{k,t}}{{\partial \mathbf{\Theta }}_{k,t}}=\left\{\begin{array}{ll}\sigma \left({\mathit{\theta }}_j\right)\left(1-\sigma \left({\mathit{\theta }}_j\right)\right)& \text{if}k=j,\\ -\sigma \left({\mathit{\theta }}_j\right)\sigma \left({\mathit{\theta }}_k\right)& \text{else,}\end{array}\right.$$ (15)

$$\frac{\partial {\mathit{P}}^t}{\partial {\mathit{W}}_{ab}}=\frac{1}{2}\sum _{k=1}^t{\mathit{P}}^{k-1}({\mathit{J}}^{ab}{\mathit{D}}^{-1}+\mathit{W}\frac{\partial {\mathit{D}}^{-1}}{\partial {\mathit{W}}_{ab}}){\mathit{P}}^{t-k}.$$ (16)

We note that in equation 16, we use the fact that $\mathit{P}=\frac{1}{2}(\mathit{I}+\mathit{W}{\mathit{D}}^{-1}$). The first term in the sum is constant, and to compute the derivative of the second term, we apply the product rule. The Jacobian of $\mathit{W}$ with respect to coordinate $ab$ is given by ${\mathit{J}}^{ab}$ and the Jacobian of ${\mathit{D}}^{-1}$ is given by

$${\left(\frac{\partial {\mathit{D}}^{-1}}{\partial {\mathit{W}}_{ab}}\right)}_{ij}=\left\{\begin{array}{ll}-{\left(\sum _j{\mathit{W}}_{aj}\right)}^{-2}& \text{if}i=j=a\\ 0& \text{else}\end{array}\right.$$ (17)

XIV. Datasets

In this section we provide further information and analysis on the individual datasets which relates the composition of the dataset as shown in Table I to the relative performance of our models as shown in Table II. Datasets used in Tables VI and VII are also described here.

TABLE VI.

Mean ± std. over four runs of mean squared error over 10 properties in ZINC, lower is better.

(μ ± σ)	BBAB	FBAB	JBCD
LEGS-FCN	0.591 ± 0.026	0.472 ± 0.018	0.603 ± 0.022
LEGS-ATTN-FCN	0.551 ± 0.033	0.510 ± 0.029	0.598 ± 0.072
LEGS-FIXED	0.548 ± 0.017	0.496 ± 0.015	0.685 ± 0.017
MP-LEGS-FCN	1.033 ± 0.081	0.795 ± 0.024	0.802 ± 0.093
GraphSAGE	1.004 ± 0.037	0.994 ± 0.026	0.987 ± 0.011
GCN	0.841 ± 0.025	0.923 ± 0.037	0.892 ± 0.039
GIN	0.670 ± 0.018	0.673 ± 0.024	0.689 ± 0.022
Baseline	1.232 ± 0.146	1.399 ± 0.163	1.357 ± 0.154

TABLE VII.

Mean ± std. over four runs of mean squared error in binding affinity prediction of targets in BindingDB, lower is better.

(μ ± σ)	P00918	P14416
LEGS-FCN	0.0318 ± 0.0009	0.0441 ± 0.0013
LEGS-ATTN-FCN	0.0332 ± 0.0016	0.0424 ± 0.0012
LEGS-FIXED	0.0597 ± 0.0017	0.0514 ± 0.0020
MP-LEGS-FCN	0.1991 ± 0.0014	0.1429 ± 0.0034
GraphSAGE	0.1083 ± 0.0074	0.1106 ± 0.0087
GCN	0.1072 ± 0.0053	0.0994 ± 0.0039
GIN	0.0615 ± 0.0022	0.0583 ± 0.0036
Baseline	0.1137 ± 0.0076	0.1201 ± 0.0068

DD [42] is a dataset extracted from the protein data bank (PDB) of 1178 high resolution proteins where each node represents an amino acid and nodes are connected if they are proximal in 3D space or along the linear protein backbone. The task is to distinguish between enzymes and non-enzymes. Since these are high resolution structures, these graphs are significantly larger than those found in our other biochemical datasets with a mean graph size of 284 nodes with the next largest biochemical dataset with a mean size of 39 nodes.

ENZYMES [33] is a dataset of 600 enzymes divided into 6 balanced classes of 100 enzymes each. Each node represents an amino acid and edges are present either in case of linkage along the protein backbone or if two amino acids are within close proximity. As we analyzed in the main text, scattering features are better able to preserve the structure between classes. LEGS-FCN slightly relaxes this structure but improves accuracy from 32 to 39% over LEGS-FIXED.

NCI1, NCI109 [43] contain slight variants of 4100 chemical compounds encoded as graphs where each node represents an atom and edges represent bonds. Each compound is separated into one of two classes based on its activity against non-small cell lung cancer and ovarian cancer cell lines. Graphs in this dataset have 30 nodes with a similar number of edges. This makes for long graphs with high diameter.

PROTEINS [33] contains 1178 protein structures with the goal of classifying enzymes vs. non enzymes. Nodes represent secondary structure elements and two nodes are connected if they are adjacent on the backbone or less than six Angstroms apart. GCN outperforms all other models on this dataset, however the Baseline model, where no structure is used also performs very similarly. This suggests that the graph structure within this dataset does not add much information over the structure encoded in the eccentricity and clustering coefficient.

PTC [44] contains 344 chemical compound graphs divided into two classes based on whether or not they cause cancer in rats. Here each node is an atom and each edge is an atomic bond. This dataset is very difficult to classify without features however LEGS-RBF and LEGS-FCN are able to capture the long range connections slightly better than other methods.

COLLAB [45] contains 5000 ego-networks of different researchers from high energy physics, condensed matter physics or astrophysics. Here each node is a researcher and edges represent co-authorship. The goal is to determine which field the research belongs to. The GraphSAGE model performs best on this dataset although the LEGS-RBF network performs nearly as well. Ego graphs have a very small average diameter. Thus, shallow networks can perform quite well on them as is the case here.

IMDB [45] contains graphs with nodes representing actresses/actors and edges between them if they are in the same move. These graphs are also ego graphs around specific actors. IMDB-BINARY classifies between action and romance genres. IMDB-MULTI classifies between 3 classes. Somewhat surprisingly GS-SVM performs the best with other LEGS networks close behind. This could be due to oversmoothing on the part of GCN and GraphSAGE when the graphs are so small.

REDDIT [45] consists of three independent datasets. In REDDIT-BINARY/MULTI-5K/MULTI-12K, each graph represents a discussion thread where nodes correspond to users and there is an edge between two nodes if one replied to the other’s comment. The task is to identify which subreddit a given graph came from. On these datasets GCN outperforms other models.

ZINC15 [40] is a database of small drug-like molecules for virtual screening. Here each node represents an atom and each edge represents a bond between two atoms. The data is organized into 2D tranches consisting of approximately 997 million molecules categorized by molecular weight, solubility (LogP), reactivity and availability for purchase.

BindingDB [41] is a publicly available database of binding affinities, focusing on interactions between small, drug-like ligands and proteins considered to be candidate drug-targets. BindingDB contains approximately 2.5 million interactions between more than 8,000 proteins and 1 million drug-like molecules.

QM9 [34], [39] is a dataset of stable organic molecules with up to 9 heavy atoms. Each node is a heavy atom with edges representing bonds between heavy atoms.

CASP We use the CASP-12 dataset where each graph represents a protein with nodes connected if they are close in 3D space or connected along the backbone. We include the amino acid type as a node feature.

XV. Training Details

We train all models for a maximum of 1000 epochs with an initial learning rate of 10⁻⁴ using the ADAM optimizer [46]. We terminate training if validation loss does not improve for 100 epochs testing every 10 epochs. Our models are implemented with Pytorch [47] and Pytorch geometric [38]. Models were run on a variety of hardware resources. For all models we use $q=4$ normalized statistical moments for the node to graph level feature extraction and m = 16 diffusion scales in line with choices in [12]. Most experiments were run on a 2×18 core Intel(R) Xeon(R) CPU E5–2697 v4 @ 2.30GHz server with 512GB of RAM equipped with two Nvidia TITAN RTX gpus.

A. Cross Validation Procedure

For all datasets we use 10-fold cross validation with 80% training data 10% validation data and 10% test data for each model. We first split the data into 10 (roughly) equal partitions. For each model we take exactly one of the partitions to be the test set and one of the remaining nine to be the validation set. We then train the model on the remaining eight partitions using the cross-entropy loss on the validation for early stopping checking every ten epochs. For each test set, we use majority voting of the nine models trained with that test set. We then take the mean and standard deviation across these test set scores to average out any variability in the particular split chosen. This results in 900 models trained on every dataset. With mean and standard deviation over 10 ensembled models each with a separate test set.

TABLE X.

Quantified distance between the empirically observed enzyme class exchange preferences of [32]

LEGS-FIXED	LEGS-FCN	GCN
0.132	0.146	0.155

TABLE XI.

Mean ± std. over four runs of mean squared error over 19 targets for the QM9 dataset, lower is better.

	LEGS-FCN	LEGS-FIXED	GCN	GraphSAGE	GIN	Baseline
μ	0.749 ± 0.025	0.761 ± 0.026	0.776 ± 0.021	0.876 ± 0.083	0.786 ± 0.032	0.985 ± 0.020
α	0.158 ± 0.014	0.164 ± 0.024	0.448 ± 0.007	0.555 ± 0.295	0.191 ± 0.060	0.593 ± 0.013
ϵ HOMO	0.830 ± 0.016	0.856 ± 0.026	0.899 ± 0.051	0.961 ± 0.057	0.903 ± 0.033	0.982 ± 0.027
ϵ LUMO	0.511 ± 0.012	0.508 ± 0.005	0.549 ± 0.010	0.688 ± 0.216	0.555 ± 0.006	0.805 ± 0.025
Δϵ	0.587 ± 0.007	0.587 ± 0.006	0.609 ± 0.009	0.755 ± 0.177	0.613 ± 0.013	0.792 ± 0.010
〈R2〉	0.646 ± 0.013	0.674 ± 0.047	0.889 ± 0.014	0.882 ± 0.118	0.699 ± 0.033	0.833 ± 0.026
ZPVE	0.018 ± 0.012	0.020 ± 0.011	0.099 ± 0.011	0.321 ± 0.454	0.012 ± 0.006	0.468 ± 0.005
U 0	0.017 ± 0.005	0.024 ± 0.008	0.368 ± 0.015	0.532 ± 0.405	0.015 ± 0.005	0.379 ± 0.013
U	0.017 ± 0.005	0.024 ± 0.008	0.368 ± 0.015	0.532 ± 0.404	0.015 ± 0.005	0.378 ± 0.013
H	0.017 ± 0.005	0.024 ± 0.008	0.368 ± 0.015	0.532 ± 0.404	0.015 ± 0.005	0.378 ± 0.013
G	0.017 ± 0.005	0.024 ± 0.008	0.368 ± 0.015	0.533 ± 0.404	0.015 ± 0.005	0.380 ± 0.014
c v	0.254 ± 0.013	0.279 ± 0.023	0.548 ± 0.023	0.617 ± 0.282	0.294 ± 0.003	0.631 ± 0.013
$U_0^{\mathrm{A}\mathrm{T}\mathrm{O}\mathrm{M}}$	0.034 ± 0.014	0.033 ± 0.010	0.215 ± 0.009	0.356 ± 0.437	0.020 ± 0.002	0.478 ± 0.014
U ATOM	0.033 ± 0.014	0.033 ± 0.010	0.214 ± 0.009	0.356 ± 0.438	0.020 ± 0.002	0.478 ± 0.014
H ATOM	0.033 ± 0.014	0.033 ± 0.010	0.213 ± 0.009	0.355 ± 0.438	0.020 ± 0.002	0.478 ± 0.014
G ATOM	0.036 ± 0.014	0.036 ± 0.011	0.219 ± 0.009	0.359 ± 0.436	0.023 ± 0.002	0.479 ± 0.014
A	0.002 ± 0.002	0.001 ± 0.001	0.017 ± 0.034	0.012 ± 0.022	0.000 ± 0.000	0.033 ± 0.013
B	0.083 ± 0.047	0.079 ± 0.033	0.280 ± 0.354	0.264 ± 0.347	0.169 ± 0.206	0.205 ± 0.220
C	0.062 ± 0.005	0.176 ± 0.231	0.482 ± 0.753	0.470 ± 0.740	0.321 ± 0.507	0.368 ± 0.525

XVI. Additional Experiments

Quantification of the enzyme class exchange preferences

We quantify the empirically observed enzyme class exchange preferences of [32] and the class exchange preferences inferred from LEGS-FIXED, LEGS-FCN, and a GCN in Table X. We measure the cosine distance between the graphs represented by the chord diagrams in Figure 2. As before, the self-affinities were discarded. We observe that LEGS-Fixed is best able to reproduces the exchange preferences. However, LEGS-FCN still reproduces the observed exchange preferences well and has significantly better classification accuracy than LEGS-Fixed.

QM9 Target Breakdown

[34], [39] contains graphs that each represent chemicals with $\tilde{1}8\text{atoms}$. Regression targets represent chemical properties of the molecules. These targets are respectively, the dipole moment $\mu$, the isotropic polarizability $\alpha$, the highest occupied molecular orbital energy ${ϵ}_{\mathrm{H}\mathrm{O}\mathrm{M}\mathrm{O}}$, the lowest unoccupied molecular orbital energy ${ϵ}_{\text{LUMO}}$, the difference $\mathrm{\Delta }ϵ={ϵ}_{\text{HOMO}}-{ϵ}_{\text{LUMO}}$, the electronic spatial extent, $⟨R^2⟩$, the zero point vibrational energy ZPVE, the internal energy at 0K $U_0$, the internal energy $U$, the enthalpy $H$, the free energy $G$, and the heat capacity $c_{\mathrm{v}}$ at 25C, the atomization energy at $0\mathrm{K}{U}_0^{\mathrm{A}\mathrm{T}\mathrm{O}\mathrm{M}}$ and $25\mathrm{C}{U}^{\mathrm{A}\mathrm{T}\mathrm{O}\mathrm{M}}$, the atomization enthalpy $H^{\mathrm{A}\mathrm{T}\mathrm{O}\mathrm{M}}$ and free energy $G^{\text{ATOM}}$ at 25C, and three rotational constants $A,$ $B,$ $C$ measured in gigahertz. For more information see references [34], [38], [39]. In Table XI we split out performance by target. GIN performs slightly better on the molecule energy targets both overall and atomization targets $U_0,$ $U,$ $H$, and $G$ where both LEGS and GIN significantly outperform the other models GCN, GraphSAGE and the structure invariant baseline. On all other targets, and especially the more difficult targets (measured by baseline MSE) the LEGS module performs the best or near to the best.