Inferring networks from time series: A neural approach

Abstract

Network structures underlie the dynamics of many complex phenomena, from gene regulation and foodwebs to power grids and social media. Yet, as they often cannot be observed directly, their connectivities must be inferred from observations of the dynamics to which they give rise. In this work, we present a powerful computational method to infer large network adjacency matrices from time series data using a neural network, in order to provide uncertainty quantification on the prediction in a manner that reflects both the degree to which the inference problem is underdetermined as well as the noise on the data. This is a feature that other approaches have hitherto been lacking. We demonstrate our method’s capabilities by inferring line failure locations in the British power grid from its response to a power cut, providing probability densities on each edge and allowing the use of hypothesis testing to make meaningful probabilistic statements about the location of the cut. Our method is significantly more accurate than both Markov-chain Monte Carlo sampling and least squares regression on noisy data and when the problem is underdetermined, while naturally extending to the case of nonlinear dynamics, which we demonstrate by learning an entire cost matrix for a nonlinear model of economic activity in Greater London. Not having been specifically engineered for network inference, this method in fact represents a general parameter estimation scheme that is applicable to any high-dimensional parameter space.

Significance Statement.

In this work, we learn static network structures from time series, an important problem across the quantitative sciences, where networks can appear either as physical links or abstract connections, but in many cases will only be indirectly observable through their dynamics. We show our method to be more accurate, computationally efficient, and versatile both than Markov-chain Monte Carlo sampling and regression. Additionally, it provides meaningful uncertainty quantification on the network prediction, which is both novel and key, as it allows estimating the range of networks compatible with the observation data, which may not fully determine the inference problem. Our method thus lets researchers make probabilistic statements about the connectivity matrices underlying the dynamics they observe.

Introduction

Networks are important objects of study across the scientific disciplines. They materialize as physical connections in the natural world, for instance as the mycorrhizal connections between fungi and root networks that transport nutrients and warning signals between plants (1, 2), human traffic networks (3, 4), or electricity grids (5, 6). However, they also appear as abstract, nonphysical entities, such as when describing biological interaction networks and food webs (7–9), gene or protein networks (10–13), economic cost relations (14, 15), or social links between people along which information (and misinformation) can flow (16–18). In all examples, though the links constituting the network may not be tangible, the mathematical description is the same. In this work, we are concerned with inferring the structure of a static network from observations of dynamics on it. The problem is of great scientific bearing: for instance, one may wish to understand the topology of an online social network from observing how information is passed through it, and some work has been done on this question (19–21). Another important application is inferring the connectivity of neurons in the brain by observing their responses to external stimuli (22, 23). In an entirely different setting, networks crop up in statistics in the form of conditional independence graphs, describing dependencies between different variables, which again are to be inferred from data (24, 25).

Our approach allows inferring network connectivities from time series data with uncertainty quantification. Uncertainty quantification for network inference is important for two reasons: first, the observations will often be noisy, and one would like the uncertainty on the data to translate to the uncertainty on the predicted network. Second, however, completely inferring large networks requires equally large amounts of data—typically at least $N-1$ equations per node, N being the number of nodes—and these observations must furthermore be linearly independent. Data of such quality and quantity will often not be available, leading to an underdetermined inference problem. The uncertainty on the predicted network should thus also reflect (at least to a certain degree) the “nonconvexity” of the loss function, i.e. how many networks are compatible with the observed data. To the best of our knowledge, no current network inference method is able to provide this information.

Network inference can be performed using ordinary least squares (OLS) regression (6, 26), but this is confined to the case where the dynamics are linear in the adjacency matrix. An alternative are sampling-based methods that generalize to the nonlinear case (27–29), but these tend to struggle in very high-dimensional settings and can be computationally expensive. Efficient inference methods for large networks have been developed for cascading dynamics (19–21), but these are highly specialized to a particular type of observation data and give no uncertainty quantification on the network prediction. Our method avoids these limitations. Its use of neural networks is motivated by their recent and successful application to low-dimensional parameter calibration problems (30, 31), both on synthetic and real data, as well as by their conceptual proximity to Bayesian inference, e.g. through neural network Gaussian processes or Bayesian neural networks (32–37). Our method’s underlying approach ties into this connection, and in fact, since it has not been specifically engineered to fit the network case, constitutes a general and versatile parameter estimation method.

Method description

We apply the method proposed in Ref. (31) to the network case. The approach consists of training a neural network to find a graph adjacency matrix $\hat{\mathbf{A}}\in {\mathbb{R}}^{N\times N}$ that, when inserted into the model equations, reproduces the observed time series $\mathbf{T}=({\mathbf{x}}_1,\dots ,{\mathbf{x}}_L)$. A neural network is a function $u_{\theta }:{\mathbb{R}}^{N\times q}\to {\mathbb{R}}^p$, where $q\ge 1$ represents the number of time series steps that are passed as input. Its output is the (vectorized) estimated adjacency matrix $\hat{\mathbf{A}}$, which is used to run a numerical solver for $B$ iterations ($B$ is the batch size) to produce an estimated time series $\hat{\mathbf{T}}(\hat{\mathbf{A}})=({\hat{\mathbf{x}}}_i,\dots ,{\hat{\mathbf{x}}}_{i+B})$. This in turn is used to train the internal parameters $\mathit{\theta }$ of the neural net (the weights and biases) via a loss function $J(\hat{\mathbf{A}}|\mathbf{T})$. The likelihood of any sampled estimate is simply proportional to

$$p(\hat{\mathbf{A}}|\mathbf{T})\propto e^{-J}$$ (1)

and by Bayes’ rule, the posterior density is then

$$\pi (\hat{\mathbf{A}}|\mathbf{T})=p(\hat{\mathbf{A}}|\mathbf{T})\times {\pi }^0(\hat{\mathbf{A}}),$$ (2)

with ${\pi }^0$ the prior density (38). As $\hat{\mathbf{A}}=\hat{\mathbf{A}}(\mathit{\theta })$, we may calculate the gradient ${\mathrm{\nabla }}_{\mathit{\theta }}J$ and use it to optimize the internal parameters of the neural net using a backpropagation method of choice; popular choices include stochastic gradient descent, Nesterov schemes, or the Adam optimizer (39). Calculating ${\mathrm{\nabla }}_{\mathit{\theta }}J$ thus requires differentiating the predicted time series $\hat{\mathbf{T}}$, and thereby the system equations, with respect to $\hat{\mathbf{A}}$. In other words, the loss function contains knowledge of the dynamics of the model. Finally, the true data are once again input to the neural net to produce a new parameter estimate $\hat{\mathbf{A}}$, and the cycle starts afresh. A single pass over the entire dataset is called an epoch.

Using a neural net allows us to exploit the fact that, as the net trains, it traverses the parameter space, calculating a loss at each point. Unlike Monte Carlo sampling, the posterior density is not constructed from the frequency with which each point is sampled, but rather calculated directly from the loss value at each sample point. This entirely eliminates the need for rejection sampling or a burn-in: at each point, the true value of the likelihood is obtained, and sampling a single point multiple times provides no additional information, leading to a significant improvement in computational speed. Since the stochastic sampling process is entirely gradient-driven, the regions of high probability are typically found much more rapidly than with a random sampler, leading to a high sample density around the modes of the target distribution. We thus track the neural network’s path through the parameter space and gather the loss values along the way. Multiple training runs can be performed in parallel, and each chain terminated once it reaches a stable minimum, increasing the sampling density on the domain, and ensuring convergence to the posterior distribution in the limit of infinitely many chains.

We begin this article with two application studies: first, we infer locations of a line failure in the British power grid from observations of the network response to the cut; and second, we infer economic cost relations between retail centers in Greater London. Thereafter, we conduct a comparative analysis of our method’s performance, before finally demonstrating the connection between the uncertainty on the neural net prediction and the uncertainty of the inference problem.

Inferring line failures in the British power grid

Power grids can be modeled as networks of coupled oscillators using the Kuramoto model of synchronized oscillation (40–44). Each node $i$ in the network either produces or consumes electrical power $P_i$ while oscillating at the grid reference frequency $\mathit{\Omega }$. The nodes are connected through a weighted undirected network $\mathbf{A}=(a_{ij})$, where the link weights $a_{ij}\sim Y_{ij}{U}_{ij}^2$ are obtained from the electrical admittances $Y_{ij}$ and the voltages $U_{ij}$ of the lines. The network coupling allows the phases ${\phi }_i(t)$ of the nodes to synchronize according to the differential equation (43)

$$\alpha \frac{{\mathrm{d}}^2{\phi }_i}{\mathrm{d}{t}^2}+\beta \frac{\mathrm{d}{\phi }_i}{\mathrm{d}t}=P_i+\kappa \sum _j{a}_{ij}\sin ({\phi }_j-{\phi }_i),$$ (3)

where $\alpha$, $\beta$, and $\kappa$ are the inertia, friction, and coupling coefficients, respectively. A requirement for dynamical stability of the grid is that $\sum _i{P}_i=0$, i.e. that as much power is put into the grid as is taken out through consumption and energy dissipation (42).

A power line failure causes the network to redistribute the power loads, causing an adjustment cascade to ripple through the network until equilibrium is restored (5). In this work, we recover the location of a line failure in the British power grid from observing these response dynamics. Figure 1a shows the high-voltage transmission grid of Great Britain as of January 2023, totalling 630 nodes (representing power stations, substations, and transformers) and 763 edges with their operating voltages. Of the roughly 1,300 power stations dotted around the island, we include those 38 with installed capacities of at least 400 MW that are directly connected to the national grid (45); following Refs. (5, 42), we give all other nodes a random value $P_i\sim \mathcal{U}[-200,+200]$ such that $\sum _i{P}_i=0$.

Fig. 1.

a) Approximate high-voltage electricity transmission grid of Great Britain. Six hundred and thirty accurately placed nodes, representing power stations, substations, and transmission line intersections, and their connectivity as of January 2023 are shown (46–48). Colors indicate the operating voltage of the lines. The size of the nodes indicate their power generation or consumption capacity (absolute values shown). White ringed nodes indicate the 38 nodes that are real power stations with capacities over 400 MW (45), with all other nodes assigned a random capacity in $[-200,+200]$. The two dotted edges in the northeast of England are the edges affected by a simulated power cut, labeled by the indices of their start and end vertices. b) The network response to the simulated power line failure, measured at four different nodes in the network (marked A–D). The equation parameters were tuned to ensure phase-locking of the oscillators ( $\alpha =1$, $\beta =0.2$, $\kappa =60$). Nodes closer to the location of the line cut (A and B) show a stronger and more immediate response than nodes further away (C and D). The shaded area indicates the 4-second window we use to infer the line location. Background image: (49).

We simulate a power cut in the northeast of England by iterating the Kuramoto dynamics until the system reaches a steady state of equilibrium (defined as $|{\dot{\phi }}_i|/{\phi }_i\le 0.01\mathrm{\forall }i$) and then removing two links and recording the network response (Fig. 1b). From the response, we can infer the adjacency matrix of the perturbed network $\tilde{\mathbf{A}}$ (with missing links) and, by comparing with the unperturbed network ${\mathbf{A}}^0$ (without missing links), the line failure locations.

We let a neural network output a (vectorized) adjacency matrix $\hat{\mathbf{A}}$ and use this estimated adjacency matrix to run the differential equation [3], which will produce an estimate $\hat{\mathbf{T}}$ of the observed time series of phases $\mathbf{T}$. A hyperparameter sweep on synthetic data showed that using a deep neural network with 5 layers, 20 nodes per layer, and no bias yields optimal results (see Figs. S2–S4). We use the hyperbolic tangent as an activation function on each layer except the last, where we use the “hard sigmoid” (50, 51)

$$\sigma (x)=\{\begin{array}{cc}0,& x\le -3,\\ 1,& x\ge +3,\\ x/6+1/2,& \mathrm{else},\end{array}$$

which allows neural net output components to actually become zero, and not just asymptotically close, thereby ensuring sparsity of the adjacency matrix—a reasonable assumption given that the power grid is far from fully connected. We use the Adam optimizer (39) with a learning rate of $0.002$ for the gradient descent step. Since the neural network outputs are in $[0,1]$, we scale the network weights $a_{ij}\to \lambda a_{ij}$ such that $a_{ij}\in [0,1]$, and absorb the scaling constant $\lambda$ into the coupling constant $\kappa$; see Supplementary material for details on the calculations.

We use the following loss function to train the internal weights $\mathit{\theta }$ of the neural network such that it will output an adjacency matrix that reproduces the observed data:

$$\begin{array}{rl}J(\hat{\mathbf{A}}|\mathbf{T})& =\Vert \hat{\mathbf{T}}(\hat{\mathbf{A}})-\mathbf{T}{\Vert }_2^2+\Vert \hat{\mathbf{A}}-{\hat{\mathbf{A}}}^{\mathrm{\top }}{\Vert }_2^2+\mathrm{tr}(\hat{\mathbf{A}})\\ & +\nu \Vert \hat{\mathbf{A}}-{\mathbf{A}}^0{\Vert }_2^2.\end{array}$$

The first summand is the data-model mismatch, the second penalizes asymmetry to enforce undirectedness of the network, and the third sets the diagonal to zero (which cannot be inferred from the data, since all terms $\sin ({\theta }_j-{\theta }_i)=0$ for $i=j$). $\nu =\nu (s)$ is a function of the iteration count $s$ designed to let the neural network search for $\tilde{\mathbf{A}}$ in the vicinity of ${\mathbf{A}}^0$, since we can assume a priori that the two will be similar in most entries. To this end, we set $\nu =10$ while the loss function has not yet reached a stable minimum, quantified by $|⟨{\mathrm{\partial }}_{s}J⟩|>{10}^{-10}$ and $|⟨{\mathrm{\partial }}_{ss}J⟩|>{10}^{-10}$, and $\nu =0$ thereafter. Here, $⟨\cdot ⟩$ is a rolling average over a window of 20 iterations, see Fig. 2. In other words, we push the neural network toward a stable minimum in the neighborhood of ${\mathbf{A}}^0$ and, once the loss stabilizes, permanently set $\nu =0$.

Fig. 2.

The total loss $J$ and its derivatives with respect to the iteration count ${\mathrm{\partial }}_{s}J$ and ${\mathrm{\partial }}_{ss}J$, averaged over a window of 20 iterations (absolute values shown). The dotted line indicates the value at which $\nu$ is set to 0.

In theory $L=N-1$, observations are needed to completely infer the network, though symmetries in the data usually mean $L>N$ is required in practice (52). In this experiment, we purposefully underdetermine the problem by only using $L<N-1$ steps; additionally, we train the network on data recorded 1 simulated second after the power cut, where many nodes will still be close to equilibrium. Although the neural network may be unable to completely infer the network, it can nevertheless produce a joint distribution on the network edge weights $p(\hat{\mathbf{A}}|\mathbf{T})$, recorded during the training, that allows us to perform hypothesis testing on the line failure location. The marginal likelihood on each network edge ${\hat{a}}_{ij}$ is given by

$$\rho ({\hat{a}}_{ij}|\mathbf{T})=\int p(\hat{\mathbf{A}}|\mathbf{T})\mathrm{d}{\hat{\mathbf{A}}}_{-ij}\times {\pi }^0({\hat{a}}_{ij}),$$ (4)

where the $-ij$ subscript indicates we are omitting the $ij$th component of $\hat{\mathbf{A}}$ in the integration. We assume uniform priors ${\pi }^0$ on each edge. In high dimensions, calculating the joint of all network edge weights can become computationally infeasible, but we can circumvent this by instead considering the two-dimensional joint density of the edge weight under consideration and the likelihood, $p({\hat{a}}_{ij},e^{-J})$, and then integrating over the likelihood,

$$\rho ({\hat{a}}_{ij}|\mathbf{T})=\int p({\hat{a}}_{ij},e^{-J})\mathrm{d}(e^{-J}).$$ (5)

We show the results in Fig. 3. Given the marginal distributions $\rho ({\hat{a}}_{ij}|\mathbf{T})$ with modes ${\tilde{a}}_{ij}$, we plot the densities on the four network edges with the highest relative prediction error $|{\tilde{a}}_{ij}-a_{ij}^0|/a_{ij}^0$. The advantage of obtaining uncertainty quantification on the network is now immediately clear: even in the underdetermined case, we are able to make meaningful statistical statements about the line failure location. We see that the missing edges consistently have the highest relative prediction errors, and that the p-values for measuring the unperturbed value $a_{ij}^0$ under the null ${\hat{a}}_{ij}$ are 0.2% and 0.04%, respectively, while being statistically insignificant for all other edges. It is interesting to note that the other candidate locations are also within the vicinity of the line failure, though their predicted values are much closer to the unperturbed value. In Fig. 3b, we see that the predicted network reproduces the response dynamics for the range covered by the training data when inserted into Eq. 3, but, since the problem was purposefully underdetermined, the errors in the prediction $\hat{\mathbf{A}}$ cause the predicted and true time series to diverge for larger $t$. Densities on all 200,000 potential edges were obtained in about 20 min on a regular laptop CPU.

Fig. 3.

Estimating the line failure location. a) The densities on the four edges with the highest relative prediction error $|{\tilde{a}}_{ij}-a_{ij}^0|/a_{ij}^0$ and their respective p-values for measuring the unperturbed value $a_{ij}^0$ (${\tilde{a}}_{ij}$ is the prediction mode). Red dotted lines indicate the values of the unperturbed network, green lines the expectation values of the distributions. The marginals are smoothed using a Gaussian kernel. We use a training set of length $L=400$ steps, and the batch size is $B=2$. CPU runtime: 24 min. b) True (black) and predicted network responses at three different locations in the network. The responses are each normalized to the value at $t=0$. The shaded area represents the 400 time steps used to train the model. While the model is able to perfectly fit the response within the training range, it is not able to learn the full network from insufficient data, causing the time series to diverge for larger $t$.

Inferring economic cost networks from noisy data

In the previous example, the underlying network was a physical entity, but in many cases networks model abstract connections. We therefore now consider a commonly used economic model of the coupling of supply and demand (14, 15, 56) and a dataset of economic activity across Greater London. The goal is to learn the entire coupling network, not just to infer the (non)existence of individual edges. In the model, $N$ origin zones of sizes $O_i$, representing economic demand, are coupled to $M$ destination zones of sizes $W_j$, modeling the supply side, through a network whose weights quantify the convenience with which demand from zone $i$ can be supplied from zone $j$: the higher the weight, the more demand flows through that edge (see Fig. 4a). Such a model is applicable e.g. to an urban setting (14), the origin zones representing residential areas, the destination zones e.g. commercial centers, and the weights quantifying the connectivity between the two (transport times, distances, etc.). The resulting cumulative demand at destination zone $j$ depends both on the current size $W_j(t)$ of the destination zone and the network weights $c_{ij}$:

Fig. 4.

Inferring economic cost networks. a) In the model, $N$ origin zones (red) are connected to $M$ destination zones (blue) through a weighted directed network. Economic demand flows from the origin zones to the destination zones, which supply the demand. We model the origin zones $O_i(t)$ as a Wiener process with diffusion coefficient ${\sigma }_O=0.1$. The resulting cumulative demand at destination zone $j$ is given by $W_j$. Note that the origin zone sizes fluctuate more rapidly than the destination zones, since there is a delay in the destination zones’ response to changing consumer patterns, controlled by the parameter $ϵ$. We use the parameters as estimated in Ref. (31), $\alpha =0.92$, $\beta =0.54$, $\kappa =8.3$, and set $ϵ=2$. b) The initial origin and destination zone sizes, given by the total household income of the $N=629$ wards in London (blue nodes) and the retail floor space of $M=49$ major centers (red nodes) (53, 54). The network is given by travel times as detailed in the text. Background map: Ref. (55). c) Predicted degree distribution (sold line) of the inferred network, for a high noise level of $\sigma =0.14$, and 1 SD (shaded area), and the true distribution (dotted line). CPU runtime: 3 min 41 s.

$$D_j=\sum _{i=1}^N\frac{W_j{(t)}^{\alpha }{c}_{ij}^{\beta }}{\sum _{k=1}^M{W}_k{(t)}^{\alpha }{c}_{ik}^{\beta }}{O}_i(t).$$ (6)

The sizes $W_j$ are governed by a system of $M$ coupled logistic Stratonovich stochastic differential equations

$$\mathrm{d}{W}_j=ϵW_j(D_j-\kappa W_j)\mathrm{d}t+\sigma W_j\circ \mathrm{d}{\xi }_j,$$ (7)

with given initial conditions $W_j(0)$, see Fig. 4a. $\alpha$, $\beta$, $\kappa$, and $ϵ$ are scalar parameters. Our goal is to infer the cost matrix $\mathbf{C}=(c_{ij})$ from observations of the time series $\mathbf{O}(t)$ and $\mathbf{W}(t)$. The model includes multiplicative noise with strength $\sigma \ge 0$, where the ${\xi }_j$ are independent white noise processes and ○ signifies Stratonovich integration (57). Crucially, the model depends nonlinearly on $\mathbf{C}$.

We apply this model to a previously studied dataset of economic activity in Greater London (15, 31). We use the ward-level household income from $N=625$ wards for 2015 (54) and the retail floor space of the $M=49$ largest commercial centers in London (53) as the initial origin zone and destination zone sizes, respectively, i.e. $\mathbf{O}(0)$ and $\mathbf{W}(0)$, and from this generate a synthetic time series using the parameters estimated in Ref. (31) for a high noise level of $\sigma =0.14$. For the network $\mathbf{C}$, we use the Google Distance Matrix API^a to extract the shortest travel time $d_{ij}$ between nodes, using either public transport or driving. The network weights are derived in Ref. (58) as

$$c_{ij}=e^{-d_{ij}/\tau },$$

where the scale factor $\tau =\underset{i,j}{max}{d}_{ij}$ ensures a unitless exponent.

We generate a synthetic time series of 10,000 time steps, from which we subsample 2,500 2-step windows, giving a total training set size of $L=\mathrm{5,000}$ time steps. This is to ensure we sample a sufficiently broad spectrum of the system’s dynamics, thereby fully determining the inference problem and isolating the effect of the training noise. A hyperparameter sweep on synthetic data showed that using a neural network with 2 layers, 20 nodes per layer, and no bias yields optimal results. We use the hyperbolic tangent as the activation function on all layers except the last, where we use the standard sigmoid function (since the network is complete, there is no need to use the hard sigmoid as all edge weights are nonzero). To train the neural network, we use the simple loss function

$$J=\Vert \hat{\mathbf{T}}(\hat{\mathbf{A}})-\mathbf{T}{\Vert }_2^2,$$

where $\hat{\mathbf{T}}$ and $\mathbf{T}$ are the predicted and true time series of destination zone sizes. Since the dynamics are invariant under scaling of the cost matrix $\mathbf{C}\to \lambda \mathbf{C}$, we normalize the row sums of the predicted and true networks, $\sum _j{c}_{ij}=1$.

Figure 4c shows the inferred distribution $\rho (k)$ of the (weighted) origin zone node degrees $k_j=\sum _i{c}_{ij}$. The solid line is the maximum likelihood prediction, and the dotted line the true distribution. Even with a high level of noise, the model manages to accurately predict the underlying connectivity matrix, comprising over 30,000 weights, in under 5 min on a regular laptop CPU. Uncertainty on $P(k)$ is given by the standard deviation,

$${\mathbb{E}}_{\hat{\mathbf{T}}}{[P(k|\hat{\mathbf{T}})-\hat{P}(k)]}^2,$$ (8)

where $\hat{P}$ is the maximum likelihood estimator. As we will discuss in the last section, this method meaningfully captures the uncertainty due to the noise in the data and the degree to which the problem is underdetermined.

Comparative performance analysis

We now analyze our method’s performance, both in terms of prediction quality and computational speed, by comparing it to a Markov-chain Monte Carlo (MCMC) approach as well as a classical regression method, presented e.g. in Refs. (6, 59). As mentioned in the Introduction section, computationally efficient network learning methods have been developed for specific data structures; however, we compare our approach with MCMC and OLS since both are general in the types of data to which they are applicable.

Consider noisy Kuramoto dynamics,

$$\alpha \frac{{\mathrm{d}}^2{\phi }_i}{\mathrm{d}{t}^2}+\frac{\mathrm{d}{\phi }_i}{\mathrm{d}t}-{\omega }_i=\sum _j{a}_{ij}\sin ({\phi }_j-{\phi }_i)+{\xi }_i,$$ (9)

with ${\xi }_i$ independent white noise processes with strength $\sigma$ and ${\omega }_i$ the eigenfrequencies of the nodes. Given $L$ observations of each node’s dynamics, we can gather the left-hand side into a single vector ${\mathbf{X}}_i\in {\mathbb{R}}^{1\times L}$ for each node, and obtain $N$ equations

$${\mathbf{X}}_i={\mathbf{A}}_i\times {\mathbf{G}}_i+{\mathit{\xi }}_i,i=1,\dots ,N,$$ (10)

with ${\mathbf{A}}_i\in {\mathbb{R}}^{1\times N}$ the $i$th row of the adjacency matrix $\mathbf{A}$ and ${\mathbf{G}}_i\in {\mathbb{R}}^{N\times L}$ the $L$ observations of the interaction terms $\sin ({\phi }_j-{\phi }_i),$  $j=1,\dots ,N$. From this, we can then naturally estimate the $i$th row of $\mathbf{A}$ using OLS:

$${\hat{\mathbf{A}}}_i=\underset{\mathit{\gamma }\in {\mathbb{R}}^{1\times N}}{\mathrm{argmin}}\Vert {\mathbf{X}}_i-\mathit{\gamma }{\mathbf{G}}_i{\Vert }_2^2={\mathbf{X}}_i{\mathbf{G}}_i^{\mathrm{\top }}{\left({\mathbf{G}}_i{\mathbf{G}}_i^{\mathrm{\top }}\right)}^{-1}.$$ (11)

Given sufficiently many linearly independent observations, the Gram matrices ${\mathbf{G}}_i{\mathbf{G}}_i^{\mathrm{\top }}$ will all be invertible; in the underdetermined case, a pseudoinverse can be used to approximate their inverses. As before, the diagonal of $\hat{\mathbf{A}}$ is manually set to 0.

In addition to regression, we also compare our method to a preconditioned Metropolis-adjusted Langevin (MALA) sampling scheme (27–29, 60), which constructs a Markov chain of sampled adjacency matrices $\hat{\mathbf{A}}$ by drawing proposals from the normal distribution

$${\hat{\mathbf{A}}}^{i+1}\sim \mathcal{N}({\hat{\mathbf{A}}}^i+\frac{\tau }{2}{\lambda }^{-1}\mathbf{P}\mathrm{\nabla }J({\hat{\mathbf{A}}}^i|\mathbf{T}),\tau {\lambda }^{-1}\mathbf{P}).$$ (12)

Here, $\tau >0$ is the integration step size, $\mathbf{P}\in {\mathbb{R}}^{N^2\times N^2}$ is a preconditioner (note that we are reshaping $\hat{\mathbf{A}}$ into an $N^2$-dimensional vector), and $\lambda =\mathrm{tr}(\mathbf{P})/N^2$ is its average eigenvalue. Each proposal is accepted with probability

$$\eta =min[1,\frac{\exp (-J\left({\hat{\mathbf{A}}}^{i+1}\right))q\left({\hat{\mathbf{A}}}^{i+1}\right|{\hat{\mathbf{A}}}^i)}{\exp (-J\left({\hat{\mathbf{A}}}^i\right))q\left({\hat{\mathbf{A}}}^i\right|{\hat{\mathbf{A}}}^{i+1})}],$$ (13)

with the transition probability

$$q\left({\hat{\mathbf{A}}}^{i+1}\right|{\hat{\mathbf{A}}}^i)\propto \exp (-\frac{1}{4\tau }\Vert {\hat{\mathbf{A}}}^{i+1}-{\hat{\mathbf{A}}}^i-\tau \mathrm{\nabla }\log \pi ({\hat{\mathbf{A}}}^I){\Vert }_2^2).$$ (14)

We tune $\tau$ so that the acceptance ratio $\eta$ converges to the optimum value of 0.57 (61).

We set the preconditioner $\mathbf{P}$ to be the inverse Fisher information covariance matrix

$${\mathbf{P}}^{-1}={\mathbb{E}}_{\hat{\mathbf{A}}}[\mathrm{\nabla }J({\hat{\mathbf{A}}}^i)\mathrm{\nabla }J({\hat{\mathbf{A}}}^i{)}^{\mathrm{\top }}],$$ (15)

which has been shown to optimize the expected squared jump distance (29). The expectation value is calculated empirically over all samples drawn using the efficient algorithm given in Ref. (29). In all experiments, we employ a “warm start” by initializing the sampler close to the minimum of the problem. We found this to be necessary in such high dimensions (between 256 and 490,000) to produce decent results. Unlike the MCMC sampler, the neural network is initialized randomly.

Figure 5a and b shows our method’s prediction accuracy alongside that of OLS regression and preconditioned MALA on synthetic Kuramoto data; the accuracy here is defined as the $L^1$ error

Fig. 5.

Computational performance analysis. a) $L^1$ prediction error (Eq. 16) of the neural scheme, the preconditioned Metropolis-adjusted Langevin sampler, and OLS regression as a function of the noise variance $\sigma$ on the training data. For very high noise levels, the training data are essentially pure noise, and the prediction errors begin to plateau. First-order Kuramoto dynamics are used ($\alpha =0$), though these results also hold for second-order dynamics (cf. Fig. S6). Enough data are used to ensure full invertibility of the Gram matrix ($\mathfrak{c}=1$). b) The $L^1$ accuracy as a function of the convexity $\mathfrak{c}$ of the loss function (Eq. 17). c) Compute times for 10 epochs, or 100 samples, of the neural scheme and the preconditioned Metropolis-adjusted Langevin sampler, averaged over 10 runs. The shaded areas show one standard deviation. On the right axis, the average $L^1$ prediction error of the neural scheme $\frac{1}{N}\Vert \hat{\mathbf{A}}-\mathbf{A}{\Vert }_1$ after 10 epochs is shown, which remains fairly constant as a function of $N$, showing that the number of gradient descent steps required to achieve a given average prediction error does not depend on $N$. d) Predicted degree distribution and e) triangle distribution of an inferred network with $N=\mathrm{1,000}$ nodes, trained on first-order noisy Kuramoto data ($\sigma =0.001$). The blue shaded areas indicate one standard deviation, and the dotted lines are the true distributions. CPU runtime: 1 h 3 min.

$$\Vert \hat{\mathbf{A}}-\mathbf{A}{\Vert }_1=\sum _{i,j}|{\hat{a}}_{ij}-a_{ij}|,$$ (16)

where $\hat{\mathbf{A}}$ is the mode of the posterior. In Fig. 5a, the accuracy is shown as a function of the noise $\sigma$ on the training data. We generate enough data to ensure the likelihood function is unimodal. For the practically noiseless case of $\sigma <{10}^{-5}$, the regression scheme on average outperforms the neural approach; however, even for very low noise levels $\sigma \ge {10}^{-5}$ and above, the neural approach proves far more robust, outperforming OLS by up to one order of magnitude and maintaining its prediction performance up to low noise levels of $\sigma \le {10}^{-3}$. Meanwhile, we find that in the low- to mid-level noise regime, the neural scheme approximates the mode of the distribution by between 1 and 2 orders of magnitude more accurately than the Langevin sampler. For high levels of noise ($\sigma >{10}^{-2}$), the performances of the neural and MALA schemes converge. These results hold both for first-order ($\alpha =0$) and second-order Kuramoto dynamics [3]; in the second-order case, the neural method begins outperforming OLS at even lower levels of $\sigma$ than in the first-order case, though the improvement is not as significant (cf. Fig. S6).

In Fig. 5b, we show the accuracy as a function of the convexity of the loss function. In general, it is hard to quantify the convexity of $J$, since we do not know how many networks fit the equation at hand. However, when the dynamics are linear in the adjacency matrix $\mathbf{A}$, we can do so using the Gram matrices of the observations of each node $i$, ${\mathbf{G}}_i{\mathbf{G}}_i^{\mathrm{\top }}$, where we quantify the (non)convexity of the problem by the minimum rank of all the Gram matrices,

$$\mathfrak{c}:=\underset{i}{min}\mathrm{rk}\left({\mathbf{G}}_i{\mathbf{G}}_i^{\mathrm{\top }}\right).$$ (17)

The problem is fully determined if $\mathfrak{c}=N-1$ and all Gram matrices are invertible. As shown, regression is again more accurate when the problem is close to fully determined; however, as $\mathfrak{c}$ decreases, the accuracy quickly drops, with the neural scheme proving up to an order of magnitude more accurate. Meanwhile, the MCMC scheme is consistently outperformed by the neural scheme, though it too eclipses regression for $\mathfrak{c}<0.75$. In summary, regression is only viable for the virtually noiseless and fully determined case, while the neural scheme maintains good prediction performance even in the noisy and highly underdetermined case (see also Fig. 5d and e).

In Fig. 5c, we show compute times to obtain 100 samples for both the neural and MALA schemes. The complexity of the neural scheme is $\mathcal{O}(n_{\mathrm{E}}\times L{N}^2)$, with $n_{\mathrm{E}}$ the number of training epochs. This is because each epoch of the model equation requires $\mathcal{O}(L{N}^2)$ operations for the vector–matrix multiplication in Eq. 11, and $\mathcal{O}(L{N}^2/B)$ for the stochastic gradient descent update, where we are holding $L/B$ constant to ensure comparability. As is visible, the average $L^1$ error per edge weight remains constant over $N$, showing that the number of epochs required to achieve a given node-averaged prediction accuracy is independent of $N$. The preconditioned MALA scheme is considerably slower, due to the computational cost of calculating the preconditioner and the Metropolis–Hastings rejection step.

Lastly, Figs. 5d and e show the estimated weighted degree and triangle distributions of a large graph with 1,000 nodes, or 1 million edge weights to be estimated, for noisy training data. The number of weighted, undirected triangles on each node $i$ is given by $\frac{1}{2}\sum _{jk}{a}_{ij}{a}_{jk}{a}_{ki}$. The model robustly finds the true adjacency matrix, and we again quantify uncertainty on the prediction using the standard deviation (Eq. 8). Estimating a network with 1,000 nodes on a standard laptop CPU took about 1 h, which reduces to 6 min when using a GPU. Most high-performance network inference techniques demonstrate their viability on graphs with at most this number of nodes, e.g. ConNIe (19) and NetINF (21). In Ref. (19), the authors state that graphs with 1,000 nodes can typically be inferred from cascade data in under 10 min on a standard laptop. Similarly, the authors of NetINF (21) state that it can infer a network with 1,000 nodes in a matter of minutes, though this algorithm does not infer edge weights, only the existence of edges, and neither technique provides uncertainty quantification.

Quantifying uncertainty

There are two sources of uncertainty when inferring adjacency matrices: the nonconvexity of the loss function $J$ and the noise $\sigma$ on the data. In Fig. 6a, we show the expected Hellinger error

Fig. 6.

Quantifying the two types of uncertainty: a) Hellinger error (Eq. 18) on the degree distribution $P(k)$ as a function of $\mathfrak{c}$ (Eq. 17) in the noiseless case. The error is normalized to the value at $\mathfrak{c}=0.21(N-1)$. As $\mathfrak{c}$ increases, the error on the prediction decreases almost linearly. We run the model from 10 different initializations and average over each (shaded area: SD). b and c) Prediction uncertainty due to noise in the data. The expected Hellinger error (Eq. 18) and expected relative entropy (Eq. 19) to the maximum likelihood estimate, as well as the total SD $s$ (Eq. 20) for the degree distribution $P(k)$ and triangle distribution $P(t)$ as a function of the noise $\sigma$ on the data are shown. Each line is an average over 10 different initializations. In all cases, training was conducted on synthetic, first-order Kuramoto data (Eq. 9, with $\alpha =0$).

$$\frac{1}{2}{\mathbb{E}}_{\hat{\mathbf{T}}}\int {[\sqrt{P(x|\hat{\mathbf{T}})}-\sqrt{\hat{P}(x)}]}^2\mathrm{d}x$$ (18)

on the predicted degree distribution as a function of $\mathfrak{c}$. As is visible, the error on the distribution decreases as $\mathfrak{c}$ tends to its maximum value of $N-1$. For $\mathfrak{c}=N-1$, some residual uncertainty remains due to the uncertainty on the neural network parameters $\mathit{\theta }$.

In Fig. 6b and c, we show the expected Hellinger error (Eq. 18) on the maximum likelihood estimator $\hat{P}$ as a function of σ, for both the degree and triangle distributions, i.e. $x\in \{k,t\}$. In addition, we also show the behavior of the expected relative entropy

$${\mathbb{E}}_{\hat{\mathbf{T}}}\int P(x|\hat{\mathbf{T}})\log \left(\frac{P(x|\hat{\mathbf{T}})}{\hat{P}(x)}\right)\mathrm{d}x$$ (19)

and the total SD

$$s^2=\int {\mathbb{E}}_{\hat{\mathbf{T}}}{[P(x|\hat{\mathbf{T}})-\hat{P}(x)]}^2\mathrm{d}x.$$ (20)

All three metrics reflect the noise on the training data, providing similarly behaved, meaningful uncertainty quantification. As the noise tends to 0, some residual uncertainty again remains, while for very high noise levels, the uncertainty begins to plateau. Our method thus manages to capture the uncertainty arising from both sources: the nonconvexity of $J$ and the noise $\sigma$ on the data.

Discussion

In this work, we have demonstrated a performative method to estimate network adjacency matrices from time series data. We showed its effectiveness at correctly and reliably inferring networks in a variety of scenarios: convex and nonconvex cases, low- to high-noise regimes, and equations that are both linear and nonlinear in $\mathbf{A}$. We were able to reliably infer power line failures in the national power grid of Great Britain, and the connectivity matrix of an economic system covering all of Greater London. We showed that our method is well able to handle inference of hundreds of thousands to a million edge weights, while simultaneously giving uncertainty quantification that meaningfully reflects both the nonconvexity of the loss function as well as the noise on the training data. Our method is significantly more accurate than MCMC sampling and outperforms OLS regression in all except the virtually noiseless and fully determined cases. This is an important improvement since large amounts of data are typically required to ensure the network inference problem is fully determined, which may often not be available, as suggested in the power grid study. Unlike regression, our method also naturally extends to the case of nonlinear dynamics. In conjunction with our previous work (31), we have now also demonstrated the viability of using neural networks for parameter calibration in both the low- and high-dimensional case. Our method is simple to implement as well as highly versatile, giving excellent results across a variety of problems. All experiments in this work were purposefully conducted on a standard laptop CPU, typically taking on the order of minutes to run.

Many lines for future research open up from this work. First, a thorough theoretical investigation of the method is warranted, establishing rigorous convergence guarantees and bounds on the error of the posterior estimate. Another direction is further reducing the amount of data required to accurately learn parameters, and in future research the authors aim to address the question of learning system properties from observations of a single particle trajectory at the mean-field limit (62, 63). In this work, we have also not considered the impact of the network topology on the prediction performance, rather focusing on the physical dynamics of the problem. An interesting question is to what degree different network structures themselves are amenable to or hinder the learning process.

Over the past decade much work has been conducted into graph neural architectures (64, 65), the use of which may further expand the capabilities of our method. More specialized architectures may prove advantageous for different (and possibly more difficult) inference tasks, though we conducted a limited number of experiments with alternatives (e.g. autoencoders, cf. Fig. S4) and were unable to find great performance improvements. Finally, one drawback of our proposed method in its current form is it that it requires differentiability of the model equations in the parameters to be learned; future research might aim to develop a variational approach to expand our method to weakly differentiable settings.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

T.G. was funded by the University of Cambridge School of Physical Sciences VC Award via DAMTP and the Department of Engineering, and supported by EPSRC grants EP/P020720/2 and EP/R018413/2. The work of G.A.P. was partially funded by EPSRC grant EP/P031587/1. M.G. was supported by EPSRC grants EP/T000414/1, EP/R018413/2, EP/P020720/2, EP/R034710/1, EP/R004889/1, and a Royal Academy of Engineering Research Chair.

Author Contributions

T.G., G.A.P., and M.G. designed and performed the research and wrote the paper. T.G. wrote the code and performed the numerical experiments.

Preprints

This manuscript was posted on a preprint: arxiv.org/abs/2303.18059.

Data Availability

Code and synthetic data can be found under https://github.com/ThGaskin/NeuralABM. It is easily adaptable to new models and ideas. The code uses the utopya package (utopia-project.org, utopya.readthedocs.io/en/latest) (66, 67) to handle simulation configuration and efficiently read, write, analyze, and evaluate data. This means that the model can be run by modifying simple and intuitive configuration files, without touching code. Multiple training runs and parameter sweeps are automatically parallelized. The neural core is implemented using pytorch (pytorch.org). All datasets have been made available, together with the configuration files needed to reproduce the plots. Detailed instructions are provided in Supplementary material and the repository.