Learning mean-field equations from particle data using WSINDy

Abstract

We develop a weak-form sparse identification method for interacting particle systems (IPS) with the primary goals of reducing computational complexity for large particle number $N$ and offering robustness to either intrinsic or extrinsic noise. In particular, we use concepts from mean-field theory of IPS in combination with the weak-form sparse identification of nonlinear dynamics algorithm (WSINDy) to provide a fast and reliable system identification scheme for recovering the governing stochastic differential equations for an IPS when the number of particles per experiment $N$ is on the order of several thousands and the number of experiments $M$ is less than 100. This is in contrast to existing work showing that system identification for $N$ less than 100 and $M$ on the order of several thousand is feasible using strong-form methods. We prove that under some standard regularity assumptions the scheme converges with rate $\mathcal{O}(N^{-1∕2})$ in the ordinary least squares setting and we demonstrate the convergence rate numerically on several systems in one and two spatial dimensions. Our examples include a canonical problem from homogenization theory (as a first step towards learning coarse-grained models), the dynamics of an attractive–repulsive swarm, and the IPS description of the parabolic–elliptic Keller–Segel model for chemotaxis. Code is available at https://github.com/MathBioCU/WSINDy_IPS.

1. Problem statement

Recently there has been considerable interest in the methodology of data-driven discovery for governing equations. Building on the Sparse Identification of Nonlinear Dynamics (SINDy) [1], we developed a weak form version (WSINDy) for ODEs [2] and for PDEs [3]. In this work, we develop a formulation for discovering governing stochastic differential equations (SDEs) for interacting particle systems (IPS). To promote clarity and for reference later in the article, we first state the problem of interest. Subsequently, we will provide a discussion of background concepts and current results in the literature.

Consider a particle system ${\mathbf{X}}_t=(X_t^{(1)},\dots ,X_t^{(N)})\in {\mathbb{R}}^{Nd}$ where on some fixed time window $t\in [0,T]$, each particle $X_t^{(i)}\in {\mathbb{R}}^d$ evolves according to the overdamped dynamics

$$d{X}_t^{(i)}=\left(-\nabla K\ast {\mu }_t^N\left(X_t^{(i)}\right)-\nabla V\left(X_t^{(i)}\right)\right)dt+\sigma (X_t^{(i)})d{B}_t^{(i)}$$ (1.1)

with initial data $X_0^{(i)}$ each drawn independently from some probability measure ${\mu }_0\in {\mathcal{P}}_p({\mathbb{R}}^d)$, where ${\mathcal{P}}_p({\mathbb{R}}^d)$ is the space probability measures on ${\mathbb{R}}^d$ with finite pth moment.¹ Here, $K$ is the interaction potential defining the pairwise forces between particles, $V$ is the local potential containing all exogenous forces, $\sigma$ is the diffusivity, and ${\left(B_t^{(i)}\right)}_{i=1,\dots ,N}$ are independent Brownian motions each adapted to the same filtered probability space ($\Omega$, $\mathcal{B}$, $\mathbb{P}$, ($({\mathcal{F}}_t{)}_{t\ge 0}$). The empirical measure is defined

$${\mu }_t^N≔\frac{1}{N}\sum _{i=1}^N{\delta }_{X_t^{(i)}},$$

and the convolution $\nabla K\ast {\mu }_t^N$ is defined

$$\nabla K\ast {\mu }_t^N(x)=\nabla {\int }_{{\mathbb{R}}^d}K(x-y)d{\mu }_t^N(y)=\frac{1}{N}\sum _{i=1}^N\nabla K\left(x-X_t^{(i)}\right)$$

where we set $\nabla K(0)=0$ whenever $\nabla K(0)$ is undefined. The recovery problem we wish to solve is the following.

$(\mathbf{P})$ Let $\mathbb{X}=({\mathbf{X}}_{\mathbf{t}}^{(1)},\dots ,{\mathbf{X}}_{\mathbf{t}}^{(M)})$ be discrete-time data at $L$ timepoints $\mathbf{t}≔(t_1,\dots ,t_L$ for $M$ i.i.d. trials of the process (1.1) with $K=K^{\star }$, $V=V^{\star }$, and $\sigma ={\sigma }^{\star }$ and let $\mathbb{Y}=\mathbb{X}+\epsilon$ be a corrupted dataset. For some fixed compact domain $\mathcal{D}\subset {\mathbb{R}}^d$ containing supp ($\mathbb{Y}$), and finite-dimensional hypothesis spaces² ${\mathcal{H}}_K\subset L^2(\mathcal{D}-\mathcal{D})$, ${\mathcal{H}}_V\subset L^2(\mathcal{D})$, and ${\mathcal{H}}_{\sigma }\subset L^2(\mathcal{D})$, solve

$$\begin{gathered} (\widehat{K},\widehat{V},\widehat{\sigma })=\underset{K\in {\mathcal{H}}_K,V\in {\mathcal{H}}_V,\sigma \in {\mathcal{H}}_{\sigma }}{\text{argmin}}{\Vert \nabla K-\nabla K^{\star }\Vert }_{L^2(\mathcal{D}-\mathcal{D})} \\ +{\Vert \nabla V-\nabla V^{\star }\Vert }_{L^2(\mathcal{D})}+\Vert \sigma -{\sigma }^{\star }{\Vert }_{L^2(\mathcal{D})}. \end{gathered}$$

The problem $(\mathbf{P})$ is clearly intractable because we do not have access to $K^{\star }$, $V^{\star }$, or ${\sigma }^{\star }$, and moreover the interactions between these terms render simultaneous identification of them ill-posed. We consider two cases: (i) $\epsilon \ne 0$ and ${\sigma }^{\star }=0$, corresponding to purely extrinsic noise, and (ii) $\epsilon =0$ and ${\sigma }^{\star }\ne 0$, corresponding to purely intrinsic noise. The extrinsic noise case is important for many applications, such as cell tracking, where uncertainty is present in the position measurements. In this case we examine $\epsilon$ representing i.i.d. Gaussian noise with mean zero and variance³ ${ϵ}^2{\mathbf{I}}_d$ added to each particle position in $\mathbb{X}$. In the case of purely intrinsic noise, identification of the diffusivity ${\sigma }^{\star }$ is required as well as the deterministic forces on each particle as defined by $K^{\star }$ and $V^{\star }$. A natural next step is to consider the case with both extrinsic and intrinsic noise. However, the combined noise case is sufficiently nuanced as to render it beyond the scope of the article, and we leave it for future work.

2. Background

Interacting particle systems (IPS) such as (1.1) are used to describe physical and artificial phenomena in a range of fields including astrophysics [4,5], molecular dynamics [6], cellular biology [7-9], and opinion dynamics [10]. In many cases the number of particles $N$ is large, with cell migration experiments often tracking 10³ – 10⁶ cells and simulations in physics (molecular dynamics, particle-in-cell, etc.) requiring $N$ in the range 10⁶ – 10¹². Inference of such systems from particle data thus requires efficient means of computing pairwise forces from $\mathcal{O}(N^2)$ interactions at each timestep for multiple candidate interaction potentials $K$. Frequently, so-called mean-field equations at the continuum level are sufficient to describe the evolution of the system, however in many cases (e.g. chemotaxis in biology [11]) only phenomenological mean-field equations are available. Moreover, it is often unclear how many particles $N$ are needed for a mean-field description to suffice. Many disciplines are now developing machine learning techniques to extract coarse-grained dynamics from high-fidelity simulations (see [12] for a recent review in molecular dynamics). In this work we provide a means for inferring governing mean-field equations from particle data assumed to follow the dynamics (1.1) that is highly efficient for large $N$, and is effective in learning mean-field equations when $N$ is in range 10³ – 10⁵.

Inference of the drift and diffusion terms for stochastic differential equations (SDEs) is by now a mature field, with the primary method being maximum-likelihood estimation, which uses Girsanov’s theorem together with the Radon–Nikodym derivative to arrive at a log-likelihood function for regression. See [13, 14] for some early works and [15] for a textbook on this approach. More recently, sparse regression approaches using the Kramers–Moyal expansion have been developed [16-18] and the authors of [19] use sparse regression to learn population level ODEs from agent-based modeling simulations. The authors of [20] also derived a bias-correcting regression framework for inferring the drift and diffusion in underdamped Langevin dynamics, and in [21] a neural network-based algorithm for inferring SDEs was developed.

Only in the last few years have significant strides been made towards parameter inference of interacting particle systems such as (1.1) from data. Apart from some exceptions, such as a Gaussian process regression algorithm recently developed in [22], applications of maximum likelihood theory are by far the most frequently studied. An early but often overlooked work by Kasonga [23] extends the maximum-likelihood approach to inference of the interaction potential $K$, assuming full availability of the continuous particle trajectories and the diffusivity $\sigma$. Two decades later, Bishwal [24] further extended this approach to discrete particle observations in the specific context of linear particle interactions. In both cases, a sequence of finite-dimensional subspaces is used to approximate the interaction function, and convergence is shown as the dimension of the subspace $J$ and number of particles $N$ both approach infinity. More recently, the maximum likelihood approach has been carried out in [25,26] in the case of radial interactions and in [27] in the case of linear particle interactions and single-trajectory data (i.e. one instance of the particle system). The authors of [28] recently developed an online maximum likelihood method for inference of IPS, and in [29] maximum likelihood is applied to parameter estimation in an IPS for pedestrian flow. It should also be noted that parameter estimation for IPS is common in biological sciences, with the most frequently used technique being nonlinear least squares with a cost function comprised of summary statistics [7,30].

Problem $(\mathbf{P})$ is made challenging by the coupled effects of $K$, $V$, and $\sigma$. In each of the previously mentioned algorithms, the assumption is made that $\sigma$ is known and/or that $K$ takes a specific form (radial or linear). In addition, the maximum likelihood-based approach approximates the differential $d{X}_t^{(i)}$ of particle $i$ using a 1st-order finite difference: $d{X}_t^{(i)}\approx X_{t+\Delta t}^{(i)}-X_t^{(i)}$, which is especially ill-suited to problems involving extrinsic noise in the particle positions. Our primary goal is to show that the weak-form sparse regression framework allows for identification of the full model ($K$, $V$, $\sigma$), with significantly reduced computational complexity, when $N$ is on the order of several thousands or more. We use a two-step process: the density of particles is approximated using a density kernel G and then the WSINDy algorithm (weak-form sparse identification of nonlinear dynamics) is applied in the PDE setting [2,3]. WSINDy is a modified version of the original SINDy algorithm [1,31] where the weak formulation of the dynamics is enforced using a family of test functions that offers reduced computational complexity, high-accuracy recovery in low-noise regimes, and increased robustness to high-noise scenarios. The feasibility of this approach for IPS is grounded in the convergence of IPS to associated mean-field equations. The reduction in computational complexity follows from the reduction in evaluation of candidate potentials (as discussed in Section 4.2), as well as the convolutional nature of the weak-form algorithm.

To the best of our knowledge, we present here the first weak-form sparse regression approach for inference of interacting particle systems, however we now review several related approaches that have recently been developed. In [32], the authors learn local hydrodynamic equations from active matter particle systems using the SINDy algorithm in the strong-form PDE setting. In contrast to [32], our approach learns nonlocal equations using the weak-form, however similarly to [32] we perform model selection and inference of parameters using sparse regression at the continuum level. The weak form provides an advantage because no smoothness is required on the particle density (for requisite smoothness the authors of [32] use a Gaussian kernel, which is more expensive to compute than simple particle binning as done here). The authors of [33] developed an integral formulation for inference of plasma physics models from PIC data using SINDy, however their method involves first computing strong-form derivatives and then averaging, rather than integration by parts against test functions as done here, and as in [32], the learned models are local. In [34], the authors apply the maximum likelihood approach in the continuum setting on the underlying nonlocal Fokker–Planck equation and learn directly the nonlocal PDE using strong-form discretizations of the dynamics. While we similarly use the continuum setting for inference (albeit in weak form), our approach differs from [34] in that it is designed for the more realistic setting of discrete-time particle data, rather than pointwise data on the particle density (assumed to be smooth in [34]).

2.1. Contributions

The purpose of the present article is to show that the weak form provides an advantage in speed and accuracy compared with existing inference methods for particle systems when the number of particles is sufficiently large (on the order of several thousand or more). The key points of this article include:

1. Formulation of a weak-form sparse recovery algorithm for simultaneous identification of the particle interaction force $K$, local potential $V$, and diffusivity $\sigma$ from discrete-time particle data.

2. Convergence with rate $\mathcal{O}(N^{-1∕2})$ of the resulting full-rank least-squares solution as the number of particles $N\to \infty$ and timestep $\Delta t\to 0$.

3. Numerical illustration of (II) along with robustness to either intrinsic randomness (e.g. Brownian motion) or extrinsic randomness (e.g. additive measurement noise).

2.2. Paper organization

In Section 3 we review results from mean-field theory used to show convergence of the weak-form method. In Section 4 we introduce the WSINDy algorithm applied to interacting particles, including hyperparameter selection, computational complexity, and convergence of the method under suitable assumptions in the limit of large $N$. Section 5 contains numerical examples exhibiting the convergence rates of the previous section and examining the robustness of the algorithm to various sources of corruption, and Section 6 contains a discussion of extensions and future directions. In the Appendix we provide information on the hyperparameters used A.1, derivation of the homogenized equation (5.3) A.2, results and discussion for the case of small $N$ and large $M$ (in comparison with [26]) A.3, and proofs to technical lemmas A.4. Table 1 includes a list of notations used throughout.

Table 1.

Notations used throughout.

Variable	Definition	Domain
$K$	Pairwise interaction potential	$L_{\mathit{loc}}^1({\mathbb{R}}^d,\mathbb{R})$
$V$	Local potential	$C({\mathbb{R}}^d,\mathbb{R})$
$\sigma$	Diffusivity	$C({\mathbb{R}}^d,{\mathbb{R}}^{d\times d})$
$N$	Number of particles per experiment	{2, 3,…}
$d$	Dimension of latent space	$\mathbb{N}$
$T$	Final time	(0, ∞)
($\Omega ,\mathcal{B}$, $\mathbb{P}$, ${(\mathcal{F})}_{t\ge 0}$, ${(B_t^{(i)})}_{i=1}^N$)	Filtered probability space
${\mathbb{R}}^d$	Independent $\Omega ,\mathcal{B}$ Brownian motions on ($\mathbb{P}$, ${(\mathcal{F})}_{t\ge 0}$, ${(B_t^{(i)})}_{i=1}^N$, $X_t^{(i)}$)
$i\mathrm{th}$	$t$ particle in the particle system (1.1) at time ${\mathbb{R}}^d$	${\mathbf{X}}_t$
$N$	$t$-particle system (1.1) at time ${\mathbb{R}}^{Nd}$	${\mu }_t^N$
	Empirical measure of ${\mathbf{X}}_t$	$\mathcal{P}({\mathbb{R}}^d)$
$F_t^N$	Distribution of ${\mathbf{X}}_t$	$\mathcal{P}({\mathbb{R}}^{Nd})$
$X_t$	Mean-field process (3.2) at time $t$	${\mathbb{R}}^d$
${\mu }_t$	Distribution of $X_t$	$\mathcal{P}({\mathbb{R}}^d)$
$\mathbf{t}$	$L$ discrete timepoints	[0, $[0,T]$]
${\mathbb{X}}_{\mathbf{t}}$	Collection of $M$ independent samples of ${\mathbf{X}}_t$ at $\mathbf{t}$	${\mathbb{R}}^{MLNd}$
${\mathbf{Y}}_t$	Sample of ${\mathbf{X}}_{\mathbf{t}}$ corrupted with i.i.d. additive noise	${\mathbb{R}}^{MLNd}$
$U_t$	Approximate density from particle positions	$\mathcal{P}({\mathbb{R}}^d)$
$G$	Density kernel mapping ${\mu }_t^N$ to $U_t$	$L^1({\mathbb{R}}^d\times {\mathbb{R}}^d,\mathbb{R})$
$\mathcal{D}$	Spatial support of $U_t$, $t\in [0,T]$	Compact subset of ${\mathbb{R}}^d$
$\mathbf{C}$	Discretization of $\mathcal{D}$
${\mathbf{U}}_t$	Discrete approximate density $U_t(\mathbf{C})$
${\langle \cdot ,\cdot \rangle }_h$	semi-discrete inner product, trapezoidal rule over $\mathbf{C}$
${\langle \cdot ,\cdot \rangle }_{h,\Delta t}$	Fully-discrete inner product, trapezoidal rule over $\mathbf{C}\times \mathbf{t}$
${\mathbb{L}}_K$	Library of candidate interaction forces
${\mathbb{L}}_V$	Library of candidate local forces
${\mathbb{L}}_{\sigma }$	Library of candidate diffusivities
$\mathbb{L}$	(${\mathbb{L}}_K$, ${\mathbb{L}}_V$, ${\mathbb{L}}_{\sigma }$)
$\Psi$	Set of $n$ test functions ${({\psi }_k)}_{k=1}^n$	$C^2({\mathbb{R}}^d\times (0,T))$
${\varphi }_{m,p}(v;\Delta )$	Test functions used in this work (Eq. (4.4)
$\lambda$	Set of sparsity thresholds
$\mathcal{L}$	Loss function for sparsity thresholds (Eq. (4.6)

3. Review of mean-field theory

Our weak-form approach utilizes that under fairly general assumptions the empirical measure ${\mu }_t^N$ of the process ${\mathbf{X}}_t$ defined in (1.1) converges weakly to ${\mu }_t$, the distribution of the associated mean-field process $X_t$ defined in (3.2). Specifically, under suitable assumptions on $V$, $K$, $\sigma$, and ${\mu }_0$, there exists $T>0$ such that for all $t\in [0,T]$, the mean-field limit⁴

$$\underset{N\to \infty }{\lim }{\mu }_t^N={\mu }_t$$

holds in the weak topology of measures,⁵ where ${\mu }_t$ is a weak-measure solution to the mean-field dynamics

$$\begin{gathered} {\partial }_t{\mu }_t=\nabla \cdot ({\mu }_t\nabla K\ast {\mu }_t)+\nabla \cdot ({\mu }_t\nabla V) \\ +\frac{1}{2}\sum _{i,j=1}^d\frac{{\partial }^2}{\partial x_i\partial x_j}(\sigma {\sigma }^T{\mu }_t),{\mu }_0\in {\mathcal{P}}_2({\mathbb{R}}^d). \end{gathered}$$ (3.1)

Eq. (3.1) describes the evolution of the distribution of the McKean–Vlasov process

$$d{X}_t=-\nabla K\ast {\mu }_t(X_t)dt-\nabla V(X_t)dt+\sigma (X_t)d{B}_t.$$ (3.2)

This implies that as $N\to \infty$, an initially correlated particle system driven by pairwise interaction becomes uncorrelated and only interacts with its mean-field distribution ${\mu }_t$. In particular, the following theorem summarizes several mean-field results taken from the review article [35] with proofs in [36,37].⁶

Theorem ([35-37]). Assume that $\Delta K$ is globally Lipschitz, $V=0$, and $\sigma (x)=\sigma =const$. In addition assume that ${\mu }_0\in {\mathcal{P}}_2({\mathbb{R}}^d)$. Then for any $T>0$, for all $t\le T$ it holds that

1. There exists a unique solution ($X_t$, ${\mu }_t$) where $X_t$ is a strong solution to (3.2) and ${\mu }_t$ is a weak-measure solution to (3.1).

2. For any $\varphi \in C_b^1({\mathbb{R}}^d)$,

   $$\mathbb{E}\left[{\left(\frac{1}{N}\sum _{i=1}^N\varphi (X_t^{(i)})-{\int }_{{\mathbb{R}}^d}\varphi (x)d{\mu }_t(x)\right)}^2\right]\le C\frac{\Vert \varphi {\Vert }_{C^1}^2}{N}$$ (3.3)

   with $C$ depending on $Lip(\nabla K)$ and $T$.

3. For any $k\in \mathbb{N}$, a.e. $-t<T$, the k-particle marginal

   $$\begin{gathered} {\rho }_t^{(k),N}(x_1,\dots ,x_k)≔{\int }_{{\mathbb{R}}^{d(N-k)}}{F}_t^N(x_1,\dots ,x_k,x_{k+1},\dots ,x_N) \\ \times d{x}_{k+1}\cdots d{x}_N \end{gathered}$$

   converges weakly to ${\mu }_t^{\otimes k}$ as $N\to \infty$, where $F_t^N\in \mathcal{P}({\mathbb{R}}^{Nd})$ is the distribution of ${\mathbf{X}}_t$.

The previous result immediately extends to the case of $\nabla V$ and $\sigma$ both globally Lipschitz and has been extended to $\nabla K$ only locally-Lipschitz in [38], $\nabla K$ with Coulomb-type singularity at the origin in [39], and domains with boundaries in [40,41]. Analysis of the model (3.1) continues to evolve in various contexts, including analysis of equilibria [42-44] and connections to deep learning [45]. For our convergence result below we simply assume that $K^{\star }$, $V^{\star }$, ${\sigma }^{\star }$ and ${\mu }_0$ are such that (i) and (ii) from the above theorem hold.

3.1. Weak form

Despite the $\mathcal{O}(N^{-1∕2})$ convergence of the empirical measure in previous theorem, it is unclear at what particle number $N$ the mean-field equations become a suitable framework for inference using particle data, due to the complex variance structure at any finite $N$. A key piece of the present work is to show that the weak form of the mean-field equations does indeed provide a suitable setting when $N$ is at least several thousands. Moreover, since in many cases (3.1) can only be understood in a weak sense, the weak form is the natural framework for identification. We say that ${\mu }_t$ is a weak solution to (3.1) if for any $\psi \in C^2({\mathbb{R}}^d\times (0,T))$ compactly supported it holds that

$$\begin{gathered} {\int }_0^T{\int }_{{\mathbb{R}}^d}{\partial }_t\psi (x,t)d{\mu }_t(x)dt \\ ={\int }_0^T{\int }_{{\mathbb{R}}^d}\left(\nabla \psi (x,t)\cdot \nabla K\ast {\mu }_t(x)+\nabla \psi (x,t)\cdot \nabla V(x) \\ -\frac{1}{2}\text{Tr}\left({\nabla }^2\psi (x,t)\sigma (x){\sigma }^T(x)\right)\right)d{\mu }_t(x)dt, \end{gathered}$$ (3.4)

where ${\nabla }^2\psi$ denotes the Hessian of $\psi$ and $\text{Tr}(\mathbf{A})$ is the trace of the matrix $\mathbf{A}$. Our method requires discretizing (3.4) for all $\psi \in \Psi$ where $\Psi =({\psi }_1,\dots ,{\psi }_n)$ is a suitable test function basis, and approximating the mean-field distribution ${\mu }_t$ with a density $U_t$ constructed from discrete particle data at time $t$. We then find $K$, $V$, and $\sigma$ within specified finite-dimensional function spaces.

4. Algorithm

We propose the general Algorithm 4.1 for discovery of mean-field equations from particle data. The inputs are a discrete-time sample $\mathbb{Y}$ containing $M$ experiments each with $N$ particle positions over $L$ timepoints $\mathbf{t}=(t_1,\dots ,t_L)$. The following hyperparameters are defined by the user: (i) a kernel $G$ used to map the empirical measure ${\mu }_t^N$ to an approximate density $U_t$, (ii) a spatial grid $\mathbf{C}$ over which to evaluate the approximate density ${\mathbf{U}}_t=U_t(\mathbf{C})$, (iii) a library of trial functions $\mathbb{L}=\{{\mathbb{L}}_K,{\mathbb{L}}_V,{\mathbb{L}}_{\sigma }\}=\{(K_j{)}_{j=1}^{J_K},(V_j{)}_{j=1}^{J_V},({\sigma }_j{)}_{j=1}^{J_{\sigma }}\}$, (iv) a basis of test functions $\Psi =({\psi }_k{)}_{k=1}^n$, (v) a quadrature rule over the spatiotemporal grid ($\mathbf{C}$, $\mathbf{t}$) denoted by an inner product ⟨·, ·⟩, and (vi) sparsity factors $\lambda$ for the modified sequential thresholding least-squares Algorithm 4.2 (MSTLS) reviewed below. We discuss choices of these hyperparameters in Section 4.1, computational complexity of the algorithm in Section 4.2, convergence of the algorithm in Section 4.3. In Section 4.4 we briefly discuss gaps between theory and practice. Table 1 includes a list of notations used throughout.

Algorithm 4.1 WSINDy for identifying mean-field Eq. (3.1) from particle data $\mathbb{Y}$ $(\widehat{\mathbf{w}},\widehat{\lambda })=\mathbf{WSINDy}(\mathbb{Y},\mathbf{t};G,\mathbf{C},\mathbb{L},\Psi ,\langle \cdot ,\cdot \rangle ,\lambda )$

4.1. Hyperparameter selection

4.1.1. Quadrature

We assume that the set of gridpoints $\mathbf{C}$ in Algorithm 4.1 is chosen from some compact domain $\mathcal{D}\subset {\mathbb{R}}^d$ containing supp ($\mathbb{Y}$). The choice of $\mathbf{C}$ (and $\mathcal{D}$) must be chosen in conjunction with the quadrature scheme, which includes integration in time using the given timepoints $\mathbf{t}$ as well as space. For completeness, the inner products in lines 10, 16, 22, and 27 of Algorithm 4.1 are defined in the continuous setting by

$$\langle f,g\rangle ={\int }_0^T{\int }_{\mathcal{D}}f(x,t)g(x,t)dxdt,$$

and the convolution in line 10 is defined by

$$\nabla K_j\ast U_t(x)={\int }_{\mathcal{D}}\nabla K_j(x-y)U_t(y)dy.$$

In the present work we adopt the scheme used in the application of WSINDy for local PDEs [3], which includes the trapezoidal rule in space and time with test functions $\psi$ compactly supported in $\mathcal{D}\times (0,T)$. We take $\mathcal{D}$ to be a rectangular domain enclosing supp ($\mathbb{Y}$) and $\mathbf{C}\subset \mathcal{D}$ to be equally-spaced in order to efficiently evaluate convolution terms. In what follows we denote by ⟨·, ·⟨ the continuous inner product, $\langle \cdot ,\cdot \rangle$ the inner product over $\langle \cdot ,\cdot {\rangle }_h$ evaluated using the composite trapezoidal rule in space with meshwidth $\mathcal{D}\times [0,T]$ and Lebesgue integration in time, and by $h$ the trapezoidal rule in both space and time, with meshwidth $\langle \cdot ,\cdot {\rangle }_{h,\Delta t}$ in space and $h$ in time. With some abuse of notation, $\Delta t$ will denote the convolution of $f\ast g$ and $f$, understood to be discrete or continuous by the context. Note also that we denote by $g$, ${\mu }^N$, and $\mu$ the measures over $U$ defined by ${\mathbb{R}}^d\times [0,T]$, ${\mu }_t^N{\Lambda }_{[0,T]}$ and ${\mu }_t{\Lambda }_{[0,T]}$, respectively, where $U_t{\Lambda }_{[0,T]}$ is the Lebesgue measure on [0, ${\Lambda }_{[0,T]}$].

4.1.2. Density kernel

Having chosen the domain $[0,t]$ containing the particle data $\mathcal{D}\subset {\mathbb{R}}^d$, let $\mathbb{Y}$ be a partition of $P^h=\{B_k{\}}_k$ with $\mathcal{D}({\cup }_k{B}_k=\mathcal{D})$ indicating the size of the atoms $h$. For the remainder of the paper we take $B_k$ to be hypercubes of equal side length $B_k$ in order to minimize computation time for integration, although this is by no means necessary. For particle positions $h$, we define the histogram⁷

$$U_t=\sum _k\frac{1}{\mid B_k\mid }{1}_{B_k}(x)\left(\frac{1}{N}\sum _i{1}_{B_k}(X_t^{(i)})\right)={\int }_{\mathcal{D}}G(x,y)d{\mu }_t^N(y).$$ (4.1)

Here the density kernel is defined

$$G(x,y)=\sum _k\frac{1}{\mid B_k\mid }{1}_{B_k}(x)1_{B_k}(y),$$

and in this setting the corresponding spatial grid ${\mathbf{X}}_t$ is the set of center-points of the bins $\mathbf{C}=({\mathbf{c}}_k{)}_k$, from which we define the discrete histogram data $B_k$. The discrete histogram ${\mathbf{U}}_t=U_t(\mathbf{C})$ then serves as an approximation to the mean-field distribution ${\mathbf{U}}_t$.

Pointwise estimation of densities from samples of particles usually requires large numbers of particles to achieve reasonably low variance, and in general the variance grows inversely proportional to the bin width ${\mu }_t$. One benefit of the weak form is that integrating against a histogram $h$ does not suffer from the same increase in variance with small $U$. In particular,

Lemma 1. Let $h$ be a sequence of $(Y^{(1)},Y^{(2)},\dots )$-valued random variables such that the empirical measure ${\mathbb{R}}^d$ of ${\mu }^N$ converges weakly to $\mathbf{Y}≔(Y^{(1)},\dots ,Y^{(N)})$ according to

$$\mathbb{E}\left[(\langle \psi ,{\mu }^N\rangle -\langle \psi ,\mu \rangle {)}^2\right]\le C\Vert \psi {\Vert }_{C^1}^2{N}^{-1}$$ (4.2)

for all $\mu \in \mathcal{p}({\mathbb{R}}^d)$ and $\psi \in C^1({\mathbb{R}}^d)$ a universal constant. Let $C$ be the histogram computed with kernel $U$ using (4.1) with $G$ bins and equal sidelength $n$. Then for any $h$ in $\psi$ compactly supported in $C^1({\mathbb{R}}^d)$, we have the mean-squared error (for $\mathcal{D}$ depending on $\tilde{C}$ and $C$)

$$\mathbb{E}\left[{\left(\langle \psi ,U{\rangle }_h-\langle \psi ,\mu \rangle \right)}^2\right]\le \tilde{C}\Vert \psi {\Vert }_{C^1}^2(h^2+N^{-1}).$$

Remark 1. We note that (4.2) follows immediately for $d$ i.i.d.,⁸ and also for $Y^{(i)}\sim \mu$ a solution to (1.1) at time $\mathbf{Y}={\mathbf{X}}_t$ with mean-field distribution $t$ according to (3.3) (for suitable $\mu ={\mu }_t$, $K$, and $V$), which is the setting of the current article.

Proof of Lemma 1. First we note that by compact support of $\sigma$, the trapezoidal rule can be written

$$\langle \psi ,U{\rangle }_h={\langle \psi ,{\int }_{{\mathbb{R}}^d}G(\cdot ,y)d{\mu }^N(y)\rangle }_h=\langle {\psi }^{\mathbf{C}},{\mu }^N\rangle =\frac{1}{N}\sum _{i=1}^N{\psi }^{\mathbf{C}}(Y^{(i)})$$

where the midpoint approximation $\psi$ of ${\psi }^{\mathbf{C}}$ is given by

$${\psi }^{\mathbf{C}}(x)=\sum _{k=1}^K\psi (c_k)1_{B_k}(x).$$ (4.3)

Hence we simply split the error and use (4.2):

$$\begin{gathered} \mathbb{E}\left[(\langle \psi ,U{\rangle }_h-\langle \psi ,\mu \rangle {)}^2\right]\le 2\mathbb{E}\left[\langle {\psi }^{\mathbf{C}}-\psi ,{\mu }^N{\rangle }^2\right] \\ +2\mathbb{E}\left[(\langle \psi ,{\mu }^N\rangle -\langle \psi ,\mu \rangle {)}^2\right] \\ \le \Vert \psi {\Vert }_{C^1}^2\left(\frac{d}{2}{h}^2+2C{N}^{-1}\right). \end{gathered}$$

The previous lemma in particular shows that small bin width $\psi$ does not negatively impact $h$ as an estimator of $\langle \psi ,U{\rangle }_h$, which is in contrast to $\langle \psi ,\mu \rangle$ as a pointwise estimator of $U(x)$. For example, if we assume that $\mu (x)$ is sampled from a $\mathbf{Y}$ density $C^1$, it is well known that the mean-square optimal bin width is $\mu$ [46]. Summarizing this result, elementary computation reveals the pointwise bias for $h=\mathcal{O}(N^{-1∕3})$,

$$\text{bias}(U(x))=\mathbb{E}[U(x)]-\mu (x)=\frac{\mu (B_k)}{\mid B_k\mid }-\mu (x)≔\mu (\xi )-\mu (x)$$

for some $x\in B_k$. Letting $\xi \in B_k$, we have

$$\text{bias}(U(x){)}^2\le L_k^2{2}^{d-1}{h}^2.$$

For the variance we get

$$\text{Var}(U(x))=\frac{1}{N}\frac{\mu (B_k)(1-\mu (B_k))}{\mid B_k{\mid }^2}=\frac{\mu (\xi )}{N}(1-\mu (B_k))\frac{1}{{\sqrt{2}}^{d-1}h},$$

and hence a bound for the mean-squared error

$$\mathbb{E}\left[(U(x)-\mu (x){)}^2\right]\le L_k^2{2}^{d-1}{h}^2+\frac{\mu (\xi )}{N{\sqrt{2}}^{d-1}}{h}^{-1}.$$

Minimizing the bound over $L_k={\max }_{x\in B_k}\mid \nabla \mu (x)\mid$ we find an approximately optimal bin width

$$h^{\ast }={\left(\frac{\rho (\xi )}{2^{\frac{3d-1}{2}}{L}_k^2}\right)}^{1∕3}{N}^{-1∕3}=\mathcal{O}(N^{-1∕3}),$$

which provides an overall pointwise root-mean-squared error of $h$. Hence, not only does the weak form remove the inverse $\mathcal{O}(N^{-1∕3})$ dependence in the variance, but fewer particles are needed to accurately approximate integrals of the density $h$.

4.1.3. Test function basis

For the test functions $\mu$ we use the same approach as the PDE setting [3], namely we fix a reference test function $({\psi }_k{)}_{1\le k\le n}$ and set

$${\psi }_k(x,t)=\psi ({\mathbf{x}}_k-x,t_k-t)$$

where $\psi$ is a fixed set of query points. This, together with a separable representation

$$\psi (x,t)={\varphi }_1(x_1)\cdots {\varphi }_d(x_d){\varphi }_{d+1}(t),$$

enables construction of the linear system ($\mathcal{Q}≔\{({\mathbf{x}}_k,t_k){\}}_{1\le k\le n}$, $\mathbf{G}$) using the FFT. We choose $\mathbf{b}$, ${\varphi }_j$, of the form

$${\varphi }_{m,p}(v;\Delta )≔\max {\left(1-{\left(\frac{v}{m\Delta }\right)}^2,0\right)}^p$$ (4.4)

where $1\le j\le d+1$ is the integer support parameter such that $m$ is supported on ${\varphi }_{m,p}$ points of spacing $2m+1$ and $\Delta \in \{h,\Delta t\}$ is the degree of $p\ge 1$. For simplicity we set ${\varphi }_{m,p}$ for ${\varphi }_j={\varphi }_{m_x,p_x}$ and $1\le j\le d$, so that only the numbers ${\varphi }_{d+1}={\varphi }_{m_t,p_t}$, $m_x$, $p_x$, $m_t$ need to be specified.

Since $p_t$ has exactly ${\varphi }_{m,p}$ weak derivatives, $p$ and $p_x$ must be at least as large as the maximum spatial and temporal derivatives appearing in the library $p_t$, or $\mathbb{L}$, $p_x\ge 2$. Larger $p_t\ge 1$ results in higher-accuracy enforcement of the weak form (3.4) in low-noise situations (see Lemma 2 of [2] for details), however the convergence analysis below indicates that smaller $p$, $\text{Lip}({\partial }^{\alpha }\psi )$, may reduce variance. The support parameter m determines the length and time scales of interest and must be chosen small enough to extract relevant scales yet large enough to sufficiently reduce variance.

In [3, Appendix A] the authors developed a changepoint algorithm to choose $\mid \alpha \mid \le 2$, $m_x$, $m_t$, $p_x$ automatically from the Fourier spectrum of the data $p_t$. Here, for each of the three examples in Section 5, we fix $\mathbf{U}$ across all particle numbers $\psi$, extrinsic noises $N$, and intrinsic noises $\epsilon$, in order to instead focus on convergence in $\sigma$. To strike a balance between accuracy and small $N$ we choose $\text{Lip}(\psi )$ and $p_t=3$ throughout. We used a combination of the changepoint algorithm and manual tuning to arrive at $p_x=5$ and $m_x$ which work well across all noise levels and numbers of particles examined. Query points $m_t$ are taken to be an equally-spaced subgrid of $\mathcal{Q}$ with spacing $\mathbf{C}$ and $s_x$ for spatial and temporal coordinates. The resulting values $s_t$, $p_x$, $p_t$, $m_x$, $m_t$, and $s_x$ determine the weak discretization scheme and can be found in Appendix A.1 for each example below.

The results in Section 5 appear robust to $s_t$, $3\le p_x$. In addition, choosing $p_t\le 9$ and $m_x$ specific to each dataset $m_t$ using the changepoint method often improves results. Although automated in the changepoint algorithm, we recommend visualizing the overlap between the Fourier spectra of $\mathbb{Y}$ and $\psi$ when choosing $\mathbf{U}$, $p_x$, $p_t$, $m_x$ in order to directly observe which the modes in the data will experience filtering under convolution with $m_t$. In general, there is much flexibility in the choice of $\psi$. Optimizing $\psi$ continues to be an active area of research.

4.1.4. Trial function library

The general Algorithm 4.1 does not impose a radial structure for the interaction potential $\psi$, nor does it assume any prior knowledge that the particle system is in fact interacting. In the examples below,⁹ the libraries $K$, ${\mathbb{L}}_K$, ${\mathbb{L}}_V$ are composed of monomial and/or trigonometric terms to demonstrate that sparse regression is effective in selecting the correct combination of nonlocal drift, local drift, and diffusion terms. Rank deficiency can result, however, from naive choices of nonlocal and local bases. Consider the kernel ${\mathbb{L}}_{\sigma }$, which satisfies

$$\nabla K\ast {\mu }_t=x-M_1({\mu }_t)=\nabla V(x)$$

where $K(x)=\frac{1}{2}\mid x{\mid }^2$ and $V(x)=\frac{1}{2}\mid x-M_1({\mu }_t){\mid }^2$ is the first moment of $M_1({\mu }_t)$. Since ${\mu }_t$ is conserved in the model (3.2) posed in free-space,¹⁰ including the same power-law terms in both libraries $M_1({\mu }_t)$ and ${\mathbb{L}}_K$ will lead to rank deficiency. This is easily avoided by incorporating known symmetries of the model (3.2), however in general we recommend that the user builds the library ${\mathbb{L}}_V$ incrementally and monitors the condition number of $\mathbb{L}$ while selecting terms.

4.1.5. Sparse regression

As in [3], we enforce sparsity using a modified sequential thresholding least-squares algorithm (MSTLS), included as Algorithm 4.2, where the “modifications” are two-fold. First, we incorporate into the thresholding step the magnitude of the overall term $\mathbf{G}$ as well as the coefficient magnitude ${\Vert {\mathbf{w}}_j{\mathbf{G}}_j\Vert }_2$, by defining non-uniform lower and upper thresholds

$$\{\begin{array}{c}{L}_j^{\lambda }=\lambda \max \left\{1,\frac{\Vert \mathbf{b}\Vert }{\Vert {\mathbf{G}}_j\Vert }\right\}\hfill \\ U_j^{\lambda }=\frac{1}{\lambda }\text{min}\left\{1,\frac{\Vert \mathbf{b}\Vert }{\Vert {\mathbf{G}}_j\Vert }\right\}\hfill \end{array}\phantom{\}},1\le j\le \mathfrak{J},$$ (4.5)

where $\mid {\mathbf{w}}_j\mid$ is the number of columns in $\mathfrak{J}=J_K+J_V+J_{\sigma }$. Second, we perform a grid search¹¹ over candidate sparsity parameters $\mathbf{G}$ and choose the parameter $\lambda$ that is the smallest minimizer over $\widehat{\lambda }$ of the cost function

$$\mathcal{L}(\lambda )=\frac{\Vert \mathbf{G}({\mathbf{w}}^{\lambda }-{\mathbf{w}}^0){\Vert }_2}{\Vert {\mathbf{Gw}}^0{\Vert }_2}+\frac{\Vert {\mathbf{w}}^{\lambda }{\Vert }_0}{\mathfrak{J}}$$ (4.6)

where $\lambda$ is the output of the sequential thresholding algorithm with non-uniform thresholds (4.5) and ${\mathbf{w}}^{\lambda }$ is the least-squares solution.¹² The final coefficient vector is then set to ${\mathbf{w}}^0={\mathbf{G}}^{†}\mathbf{b}$.

We now review some aspects of Algorithm 4.2. Results from [47] on the convergence of STLS carry over for the inner loop of Algorithm 4.2, namely if $\widehat{\mathbf{w}}={\mathbf{w}}^{\widehat{\lambda }}$ is full-rank, the inner loop terminates in at most $\mathbf{G}$ iterations with a resulting coefficient vector $\mathfrak{J}$ that is a local minimizer of the cost function ${\mathbf{w}}^{\lambda }$. This implies that the full algorithm terminates in atmost $F(\mathbf{w})=\Vert \mathbf{Gw}-\mathbf{b}{\Vert }_2^2+{\lambda }^2\Vert \mathbf{w}{\Vert }_0$ least-squares solves (each on a subset of columns of $m\mathfrak{J}$).

When considering recovery of the true weight vector $\mathbf{G}$, Theorem 1 implies convergence in particle number ${\mathbf{w}}^{\star }$ of $N$ to $\widehat{\mathbf{w}}$ when ${\mathbf{w}}^{\star }$ is full-rank. The rate of convergence depends implicitly on the condition number of $\mathbf{G}$, hence it is recommended that one builds the library $\mathbf{G}$ incrementally, stopping before the conditional number $\mathbb{L}$ grows too large. If $\kappa (\mathbf{G})$ is rank deficient, classical recovery guarantees from compressive sensing do not necessarily apply, due to high correlations between the columns of $\mathbf{G}$ (recall each column is constructed from the same dataset $\mathbf{G}$).¹³ One may employ additional regularization (e.g. Tikhonov regularization as in [31]); however, in general, improvements to existing sparse regression algorithms for rank-deficient, noisy, and highly-correlated matrices is an active area of research.

Algorithm 4.2 Modified sequential thresholding with automatic threshold selection $\mathbf{U}$

The bounds (4.5) enforce a quasi-dominant balance rule, such that ${\Vert {\mathbf{w}}_j{\mathbf{G}}_j\Vert }_2$ is within ${\log }_{10}(\lambda )$ orders of magnitude from $\Vert \mathbf{b}{\Vert }_2$ and $\mid {\mathbf{w}}_j\mid$ is within ${\log }_{10}(\lambda )$ orders of magnitude from 1 (the coefficient of time derivative ${\partial }_t{\mu }_t$. This is specifically designed to handle poorly-scaled data (see the Burgers and Korteweg–de Vries examples in [3]), however we leave a more thorough examination of the thresholding requirements necessary for models with multiple scales to future work.

As the sum of two relative errors, minimizers of the cost function $\mathcal{L}$ equally weight the accuracy and sparsity of ${\mathbf{w}}^{\widehat{\lambda }}$. By choosing $\widehat{\lambda }$ to be the smallest minimizer of $\mathcal{L}$ over $\lambda$, we identify the thresholds $\lambda \in \lambda$ such that $\lambda <\widehat{\lambda }$ as those resulting in an overfit model. We commonly choose $\lambda$ to be log-equally spaced (e.g. 50 points from 10⁻⁴ to 1), and starting from a coarse grid, refine $\lambda$ until the minimum of $\mathcal{L}$ is stationary.

4.2. Computational complexity

To compute convolutions against $\nabla K$ for each $K\in {\mathbb{L}}_K$, we first evaluate $({\partial }_{x_i}K{)}_{1\le i\le d}$ at the grid $\mathbf{C}-\mathbf{C}$ defined by

$$\mathbf{C}-\mathbf{C}≔\{x\in {\mathbb{R}}^d:x=(i_{1}h,\dots ,i_{d}h),-n_{\ell }\le i_{\ell }\le n_{\ell }\},$$

where $h$ is the spacing of $\mathbf{C}$ and $n_{\ell }$, $1\le \ell \le d$, is the number of points in $\mathbf{C}$ along the $\ell \text{th}$ coordinate. Computing¹⁴ ${\partial }_{x_i}\mathbf{K}≔{\partial }_{x_i}(\mathbf{C}-\mathbf{C})$ requires $2^d\mid \mathbf{C}\mid$ evaluations of $K$, where $\mid \mathbf{C}\mid ={\prod }_{\ell =1}^d{n}_{\ell }$ is the number of points in $\mathbf{C}$. We then use the $d$-dimensional FFT to compute the convolutions

$${\partial }_{x_i}\mathbf{K}\ast {\mathbf{U}}_t\approx {\partial }_{x_i}K\ast U_t(\mathbf{C}),t\in \mathbf{t}$$

where only entries corresponding to particle interactions within $\mathbf{C}$ are retained. For $d=1$ this amounts to $\mathcal{O}(\mid \mathbf{C}\mid \log \mid \mathbf{C}\mid )$ flops per timestep. For $d=2$ and higher dimensions, the $d$-dimensional FFT is considerably slower unless one of the arrays is separable. To enforce separability, trial interaction potentials in ${\mathbb{L}}_K$ can be chosen to be a sum of separable functions,

$$K(x)=\sum _{q=1}^Q{k}_{1,q}(x_1)\cdots k_{d,q}(x_d),$$ (4.7)

in which case only a series of one-dimensional FFTs are needed to compute ${\partial }_{x_i}\mathbf{K}\ast {\mathbf{U}}_t$, and again the cost is $\mathcal{O}(\mid \mathbf{C}\mid \log \mid \mathbf{C}\mid )$ per timestep. When $K$ is not separable, a low-rank approximation can be computed from ${\partial }_{x_i}\mathbf{K}$,

$${\partial }_{x_i}\mathbf{K}\approx \sum _{q=1}^Q{\sigma }_q{\mathbf{k}}_{1,q}\otimes \cdots \otimes {\mathbf{k}}_{d,q}$$ (4.8)

which again reduces convolutions to a series of one-dimensional FFTs. For $d=2$, this is accomplished using the truncated SVD, while for higher dimensions there does not exist a unique best rank-Q tensor approximation, although several efficient algorithms are available to compute a sufficiently accurate decomposition [49-51] (and the field of fast tensor decompositions is advancing rapidly).

We propose to compute convolutions by first computing a low-rank decomposition of ${\partial }_{x_i}\mathbf{K}$ using the randomized truncated SVD [52] or a suitable randomized tensor decomposition and then applying the $d$-dimensional FFT as a series of one-dimensional FFTs. In the examples below we consider only $d=1$ and $d=2$, and leave extension to higher dimensions to future work.

Using low-rank approximations, the mean-field approach provides a significant reduction in computational complexity compared to direct evaluations of particle trajectories when $N$ is sufficiently large. A particle-level computation of the nonlocal force in weak-form requires evaluating terms of the form

$$\begin{gathered} \sum _{\ell =1}^L\left(\frac{1}{N^2}\sum _{i=1}^N\sum _{j=1}^N{\partial }_x\psi (X_{t_{\ell }}^{(i)},t_{\ell }){\partial }_{x}K(X_{t_{\ell }}^{(i)}-X_{t_{\ell }}^{(j)})\right)\Delta t \\ ={\langle {\partial }_x\psi ,{\mu }^N({\partial }_{x}K\ast {\mu }^N)\rangle }_{h,\Delta t}. \end{gathered}$$

For a single candidate interaction potential $K$, a collection of $J$ test functions $\psi$, and $M$ experiments, this amounts to $ML{N}^2+MLNJ$ function evaluations in ${\mathbb{R}}^d$ and $\mathcal{O}(ML{N}^{2}J)$ flops. If we use the proposed method, employing the convolutional weak form with a separable reference test function $\psi$ (as in WSINDy for PDEs [3]) and exploiting a rank $Q$ approximation of ${\partial }_x\mathbf{K}$ when computing convolutions against interaction potential, we instead evaluate

$${\partial }_x\psi \ast (U({\partial }_{x}K\ast U))$$

using $\mathcal{O}(LQ\mid \mathbf{C}\mid \log (\mid \mathbf{C}\mid ))$ flops and only $2^d\mid \mathbf{C}\mid$ evaluations of ${\partial }_{x}K$, reused at each of the $L$ timepoints.¹⁵ Fig. 1 provides a visualization of the reduction in function evaluations for $L=100$ timepoints and $M=10$ experiments over a range of $N$ and $\mid \mathbf{C}{\mid }^{1∕d}$ (points along each spatial dimension when $\mid \mathbf{C}\mid$ is a hypercube) in $d=2$ and $d=3$ spatial dimensions. Table 5 in Appendix A.1 lists walltimes for the examples below, showing that with $N=64,000$ particles the full algorithm implemented in MATLAB runs in under 10 s with all computations in serial on a laptop with an AMD Ryzen 7 pro 4750u processor, and requiring less than 8 Gb of RAM. The dependence on $N$ is only through the $\mathcal{O}(N)$ computation of the histogram, hence this approach may find applications in physical coarse-graining (e.g. of molecular dynamics or plasma simulations).

Fig. 1.

Factor by which the mean-field evaluation of interaction forces using histograms reduces total function evaluations as a function of particle number $N$ and average gridpoints per dimension ${\mid \mathbf{C}\mid }^{1∕d}$ for data with $M=10$ experiments each with $L=100$ timepoints. For example, with $d=2$ spatial dimensions (left) and $N>2000$ particles, the number of function evaluations is reduced by at least a factor of 10⁴.

4.3. Convergence

We now show that the estimators $\widehat{K}$, $\widehat{V}$, and $\widehat{\sigma }$ of the weak-form method converge with a rate $\mathcal{O}(h+N^{-1∕2}+\Delta t^{\eta })$ when ordinary least squares are used (i.e. $\lambda =0$) and only $M=1$ experiment is available. Here $\eta >0$ is the Hölder exponent of the sample paths of the process ${\mathbf{X}}_t$. We assume that $\mathcal{D}$, $\mathbf{C}$, $G$, $P^h$ and the resulting histogram $\mathbf{U}=({\mathbf{U}}_t{)}_{t\le T}$ are as in Section 4.1.2. We make the following assumptions on the true model and resulting linear system throughout this section.

Assumption H. Let $p\ge 1$ be fixed.

• (H.1) For each $N\ge 2$, ${\mathbf{X}}_t=(X_t^{(1)},\dots ,X_t^{(N)})$ is a strong solution to (1.1) for $t\in [0,T]$, and for some $\eta >0$ the sample paths $t\to X_t^{(i)}(\omega )$ are almost-surely $\eta$-Hölder continuous, i.e. for some $C_{\eta }>0$,

  $$\begin{gathered} \mid X_t^{(i)}(\omega )-X_s^{(i)}(\omega )\mid \le C_{\eta }\mid t-s{\mid }^{\eta },\forall 0\le s\le t\le T, \\ \forall 1\le i\le N,\text{for}\text{a.e.}\omega \in \Omega . \end{gathered}$$
• (H.2) The initial particle distribution ${\mu }_0$ satisfies the moment bound

  $${\int }_{{\mathbb{R}}^d}\mid x{\mid }^{p}d{\mu }_0(x)≔M_p<\infty .$$
• (H.3) $\nabla K^{\star }$ and $\nabla V^{\star }$ satisfy for some $C_p>0$ the growth bound:

  $$\begin{gathered} \mid \nabla V^{\star }(x)-\nabla V^{\star }(y)\mid +\mid \nabla K^{\star }(x)-\nabla K^{\star }(y)\mid \\ \le C_p\mid x-y\mid (1+\max \{\mid x\mid ,\mid y\mid {\}}^{p-1}),x,y\in {\mathbb{R}}^d. \end{gathered}$$
• (H.4) For the same constant $C_p>0$, it holds that¹⁶

  $$\begin{gathered} \Vert {\sigma }^{\star }(x)-{\sigma }^{\star }(y){\Vert }_F\le C_p\mid x-y{\mid }^{1∕2} \\ \times (1+\max \{\mid x\mid ,\mid y\mid {\}}^{p∕2-1∕2}),x,y\in {\mathbb{R}}^d \end{gathered}$$

• (H.5) The test functions $({\psi }_k{)}_{1\le k\le \eta }\subset C^2({\mathbb{R}}^d\times (0,T))$ are compactly supported and together with the library $\mathbb{L}$ are such that $\mathbf{G}$ has full column rank with¹⁷ ${\Vert {\mathbf{G}}^{†}\Vert }_1\le C_{\mathbf{G}}$ almost surely for some constant $C_{\mathbf{G}}>0$.

• (H.6) The true functions $K^{\star }$, $V^{\star }$, and ${\sigma }^{\star }$ are in the span of $\mathbb{L}$.

We will now define some notation and state some technical lemmas with proofs found in Appendix A.4. Define the weak-form operator

$$\begin{gathered} \mathcal{L}(\rho ,\psi ,\langle \cdot ,\cdot \rangle ) \\ ≔\langle {\partial }_t\psi -\nabla \psi \cdot \nabla K^{\star }\ast \rho -\nabla \psi \cdot \nabla V^{\star }+\frac{1}{2}\text{Tr}({\nabla }^2\psi {\sigma }^{\star }({\sigma }^{\star }{)}^T),\rho \rangle , \end{gathered}$$ (4.9)

where $\rho =({\rho }_t{)}_{t\le T}$ is a curve in ${\mathcal{P}}_p({\mathbb{R}}^d)$, $\psi$ is a $C^2$ function compactly supported over ${\mathbb{R}}^d\times (0,T)$, and $\langle \cdot ,\cdot \rangle$ is an inner product over ${\mathbb{R}}^d\times (0,T)$. If $\rho =({\mu }_t{)}_{t\le T}$ is a weak solution to (3.1) and $\langle \cdot ,\cdot \rangle$ is the $L^2({\mathbb{R}}^d)$ inner product then $\mathcal{L}(\rho ,\psi ,\langle \cdot ,\cdot \rangle )=0$. If instead $\rho =({\mu }_t^N{)}_{t\le T}$, then by Itô’s formula $\mathcal{L}(\rho ,\psi ,\langle \cdot ,\cdot \rangle )$ takes the form of an Itô integral, and we have the following:

Lemma 2. Under Assumptions (H.1)-(H.5), there exists a constant $C>0$ independent of $N$ such that

$$\mathbb{E}\left[\mid \mathcal{L}({\mu }^N,\psi ,\langle \cdot ,\cdot \rangle )\mid \right]\le \frac{C}{\sqrt{N}}.$$

Proof. See Appendix A.4.

With the following lemma, we can relate the histogram $U$ to the empirical measure ${\mu }^N$ through $\mathcal{L}$ using the inner product $\langle \cdot ,\cdot {\rangle }_h$ defined by trapezoidal-rule integration in space and continuous integration in time.

Lemma 3. Under Assumptions (H.1)-(H.5), for $C$ independent of $N$ and $h$, it holds that

$$\mathbb{E}\left[\mid \mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_h)-\mathcal{L}({\mu }^N,\psi ,\langle \cdot ,\cdot \rangle )\mid \right]\le Ch.$$

Proof. See Appendix A.4.

To incorporate discrete-time effects, we consider the difference between $\mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_h)$ and $\mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_{h,\Delta t})$, where recall that $\langle \cdot ,\cdot {\rangle }_{h,\Delta t}$ denotes trapezoidal rule integration in space with meshwidth $h$ and in time with sampling rate $\Delta t$.

Lemma 4. Under Assumptions (H.1)-(H.5), for $C$ independent of $N$, $h$, and $\Delta t$, it holds that

$$\mathbb{E}\left[\mid \mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_h)-\mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_{h,\Delta t})\mid \right]\le C(h+\Delta t^{\eta }).$$

Proof. See Appendix A.4.

The previous estimates directly lead to the following bound on the model coefficients $\widehat{\mathbf{w}}$:

Theorem 1. Assume that Assumption H holds. Let $\widehat{\mathbf{w}}$ be the learned model coefficients and ${\mathbf{w}}^{\star }$ the true model coefficients. For $C$ independent of $N$, $h$, and $\Delta t$ it holds that

$$\mathbb{E}\left[{\Vert \widehat{\mathbf{w}}-{\mathbf{w}}^{\ast }\Vert }_1\right]\le C\left(h+N^{-1∕2}+\Delta t^{\eta }\right).$$

Proof. Using that $K^{\star }$, $V^{\star }$, and ${\sigma }^{\star }$ are in the span of $\mathbb{L}$ (H.6), we have that

$${\mathbf{b}}_k=\langle {\partial }_t{\psi }_k,\mathbf{U}{\rangle }_{h,\Delta t}=\mathcal{L}(U,{\psi }_k,\langle \cdot ,\cdot {\rangle }_{h,\Delta t})+{\mathbf{G}}_k^T{\mathbf{w}}^{\star }≔{\mathbf{L}}_k+{\mathbf{G}}_k^T{\mathbf{w}}^{\star },$$

where ${\mathbf{G}}_k^T$ is the $k\text{th}$ row of $\mathbf{G}$. From Lemmas 2-4 we have

$$\begin{gathered} \mathbb{E}[\mid {\mathbf{L}}_k\mid ]\le \mathbb{E}\left[\mid \mathcal{L}(U,{\psi }_k,\langle \cdot ,\cdot {\rangle }_{h,\Delta t})-\mathcal{L}(U,{\psi }_k,\langle \cdot ,\cdot {\rangle }_h)\mid \right] \\ +\mathbb{E}\left[\mid \mathcal{L}(U,{\psi }_k,\langle \cdot ,\cdot {\rangle }_h)-\mathcal{L}({\mu }^N,{\psi }_k,\langle \cdot ,\cdot \rangle )\mid \right] \\ +\mathbb{E}\left[\mathcal{L}({\mu }^N,{\psi }_k,\langle \cdot ,\cdot \rangle )\mid \right] \\ \le C^{\prime }(h+N^{-1∕2}+\Delta t^{\eta }). \end{gathered}$$

Using that $\mathbf{G}$ is full rank, it holds that $\widehat{\mathbf{w}}={\mathbf{G}}^{†}\mathbf{b}={\mathbf{G}}^{†}\mathbf{L}+{\mathbf{w}}^{\star }$, hence the result follows from the uniform bound on ${\Vert {\mathbf{G}}^{†}\Vert }_1$ (H.5):

$$\mathbb{E}[\Vert \widehat{\mathbf{w}}-{\mathbf{w}}^{\star }{\Vert }_1]\le \mathbb{E}[\Vert {\mathbf{G}}^{†}{\Vert }_1\Vert \mathbf{L}{\Vert }_1]\le C^{\prime }{C}_{\mathbf{G}}(h+N^{-1∕2}+\Delta t^n).$$

□

Under the assumption (H.6), an immediate corollary is

$$\begin{gathered} \mathbb{E}[\Vert K^{\star }-\widehat{K}{\Vert }_{L^2(\mathcal{D}-\mathcal{D})}+\Vert V^{\ast }-\widehat{V}{\Vert }_{L^2(\mathcal{D})}\phantom{]} \\ \phantom{[}+\Vert \Vert {\sigma }^{\star }({\sigma }^{\star }{)}^T-\widehat{\sigma }(\widehat{\sigma }{)}^T{\Vert }_F{\Vert }_{L^2(\mathcal{D})}] \\ \le C(h+N^{-1∕2}+\Delta t^{\eta }), \end{gathered}$$ (4.10)

This follows from

$$\begin{gathered} \Vert K^{\star }-\widehat{K}{\Vert }_{L^2(\mathcal{D}-\mathcal{D})}\le \sum _{j=1}^{\mathfrak{J}}\mid {\mathbf{w}}_j^{\star }-{\widehat{\mathbf{w}}}_j\mid \Vert K_j{\Vert }_{L^2(\mathcal{D}-\mathcal{D})} \\ \le \left(\underset{j}{\text{sup}}\Vert K_j{\Vert }_{L^2(\mathcal{D}-\mathcal{D})}\right)\Vert {\mathbf{w}}^{\star }-\widehat{\mathbf{w}}{\Vert }_1, \end{gathered}$$

and similarly for $\widehat{V}$ and $\widehat{\sigma }$. Finally, setting $h=N^{-\alpha }$ for $\alpha >0$ will ensure convergence as $N\to \infty$ and $\Delta t\to 0$.

4.4. Theory vs. Practice

We now make several remarks about the practical performance of Algorithm 4.1 with respect to the theoretical convergence of Theorem 1.

Remark 2. An important case of Theorem 1 is ${\sigma }^{\star }=0$, in which case ${\mu }_t^N$ itself is a weak-measure solution to the mean-field Eq. (3.1) and the algorithm returns, for $\eta \ge 2$, $\Vert \widehat{\mathbf{w}}-{\mathbf{w}}^{\star }{\Vert }_1\le C(h+\Delta t^{\eta })$. This partially explains the accuracy observed for purely-extrinsic noise examples in Figs. 5 and 9. We note further that in the absence of noise ($\epsilon =0$ and ${\sigma }^{\star }=0$, not included in this work) Algorithm 4.1 recovers systems to high accuracy similarly to WSINDy applied to local dynamical systems [2,3].

Fig. 5.

Recovery of (3.1) in one spatial dimension for $K^{\star }=K_{\mathrm{QANR}}$ and ${\sigma }^{\star }=0$ under different levels of observational noise $ϵ$. Left: relative error in learned interaction kernel $\widehat{K}$. Middle: true positivity ratio for full model (3.1). Right: true positivity ratio for drift term.

Fig. 9.

Recovery of (3.1) in two spatial dimensions with $K^{\star }$ given by (5.5) from deterministic particles (${\sigma }^{\star }=0$) with extrinsic noise $ϵ$.

Remark 3. Algorithm 4.1 in general implements sparse regression, yet Theorem 1 deals with ordinary least squares. Since least squares is a common subroutine of many sparse regression algorithms (including the MSTLS algorithm used here), the result is still relevant to sparse regression. Lastly, the full-rank assumption on $\mathbf{G}$ implies that as $N\to \infty$ sequential thresholding reduces to least squares.

Remark 4. Theorem 1 assumes data from a single experiment ($M=1$), while the examples below show that $M>1$ experiments improve results. For any fixed $M>1$, the $N\to \infty$ limit results in convergence, however, the $N$-fixed and $M\to \infty$ limit does not result in convergence, as this does not lead to the mean-field equations.¹⁸ The examples below show that using $M>1$ has a practical advantage, and in Appendix A.3 we demonstrate that even for small particle systems ($N=10$) the large $M$ regime yields satisfactory results.

Remark 5. Many interesting examples have non-Lipschitz $\nabla K$, in particular a lack of smoothness at $x=0$. If ${\mu }_t^N$ does not converge to a singular measure as $N\to \infty$, then the bound (A.4) holds for $\nabla K$ with a jump discontinuity at $x=0$, where an additional $\mathcal{O}(h)$ term arises from pairwise interactions within an $\mathcal{O}(h)$ distance. The examples below are chosen in part to show that $\mathcal{O}(N^{-1∕2})$ convergence holds for $\nabla K$ with jumps at the origin.

5. Examples

We now demonstrate the successful identification of several particle systems in one and two spatial dimensions as well as the $\mathcal{O}(N^{-1∕2})$ convergence predicted in Theorem 1. In each case we use Algorithm 4.1 to discover a mean-field equation of the form (3.1) from discrete-time particle data. For each dataset we simulate the associated interacting particle system ${\mathbf{X}}_t$ given by (1.1) using the Euler–Maruyama scheme (initial conditions and timestep are given in each example). We assess the ability of WSINDy to select the correct model using the true positivity ratio¹⁹

$$\text{TPR}(\widehat{\mathbf{w}})=\frac{\text{TP}}{\text{TP}+\text{FN}+\text{FP}}$$ (5.1)

where TP is the number of correctly identified nonzero coefficients, FN is the number of coefficients falsely identified as zero, and FP is the number of coefficients falsely identified as nonzero [53]. To demonstrate the $\mathcal{O}(N^{-1∕2})$ convergence given by (4.10), for correctly identified models (i.e. $\text{TPR}(\widehat{\mathbf{w}})=1$) we compute the relative ${\ell }_2$-error of the recovered interaction force $\nabla \widehat{K}$, local force $\nabla \widehat{V}$, and diffusivity $\widehat{\sigma }$ over $\mathbf{C}-\mathbf{C}$ and $\mathbf{C}$, respectively, denoting this by $\Vert \cdot \Vert$ in the plots below. Results are averaged over 100 trials.

For the computational grid $\mathbf{C}$ we first compute the sample standard deviation $s$ of $\mathbb{Y}$ and we choose $\mathcal{D}$ to be the rectangular grid extending $3s$ from the mean of $\mathbb{Y}$ in each spatial dimension. We then set $\mathbf{C}$ to have 128 points in $x$ and $y$ for $d=2$ dimensions, and 256 points in $x$ for $d=1$, noting that these numbers are fairly arbitrary, and used to show that the grid need not be too large. We set the sparsity factors so that ${\log }_{10}(\lambda )$ contains 100 equally spaced points from −4 to 0. More information on the specifications of each example can be found in Appendix A.1. (MATLAB code used to generate examples is available at https://github.com/MathBioCU/WSINDy_IPS.)

5.1. Two-dimensional local model and homogenization

The first system we examine is a local model $(K^{\star }(x,y)=0)$ defined by the local potential $V^{\star }(x,y)=-x-y$ and diffusivity ${\sigma }^{\star }(x,y)=\sqrt{2(1+0.95cos(\omega x)cos(\omega y))}{\mathbf{I}}_2$, where ${\mathbf{I}}_2$ is the identity in ${\mathbf{R}}^2$. This results in a constant advection, variable diffusivity mean-field model²⁰

$${\partial }_t{\mu }_t=-{\partial }_x{\mu }_t-{\partial }_y{\mu }_t+\Delta [(1+0.95\cos (\omega x)\cos (\omega y)){\mu }_t].$$ (5.2)

The purpose of this example is three-fold. First, we are interested in the ability of Algorithm 4.1 to correctly identify a local model from a library containing both local and nonlocal terms. Next, we evaluate whether the $\mathcal{O}(N^{-1∕2})$ convergence is realized. Lastly, we investigate whether for large $\omega$ the weak-form identifies the associated homogenized equation (derived in Appendix A.2)

$${\partial }_t{\mu }_t=-{\partial }_x{\mu }_t-{\partial }_y{\mu }_t+\overline{\omega }\Delta {\mu }_t,$$ (5.3)

where $\overline{\omega }$ is given by the harmonic mean of diffusivity:

$$\overline{\omega }={\left({\int }_{\mathcal{D}}\frac{dxdy}{1+0.95\cos (x)\cos (y)}\right)}^{-1}.$$

For $\omega \in \{1,20\}$ we evolve the particles from an initial Gaussian distribution with mean zero and covariance ${\mathbf{I}}_2$ and record particle positions for 100 timesteps with $\Delta t=0.02$ (subsampled from a simulation with timestep 10⁻⁴). We use a rectangular domain $\mathcal{D}$ of approximate sidelength 10 and compute histograms with 128 bins in $x$ and $y$ for a spatial resolution of $\Delta x\approx 0.078$ (see Fig. 2 for solution snapshots), over which $\overline{\omega }\approx 0.62$. For $\omega =1$ we compare recovered equations with the full model (5.2), while for $\omega =20$ we compare with (5.3), for comparison computing $\overline{\omega }$ over each domain $\mathcal{D}$ using MATLAB’s integral2. Fig. 3 shows that as the particle number increases, we do in fact recover the desired equations, with $\text{TPR}(\widehat{\mathbf{w}})$ approaching one as $N$ increases. For $\omega =1$ we observe $\mathcal{O}(N^{-1∕2})$ convergence of the local potential $\widehat{V}$ and the diffusivity $\widehat{\sigma }$. For $\omega =20$, we observe approximate $\mathcal{O}(N^{-1∕2})$ convergence of $\widehat{V}$, and $\widehat{\sigma }$ converging to within 2% of $\sqrt{2\overline{\omega }}$, the homogenized diffusivity (higher accuracy can hardly be expected for $\omega =20$ since (5.3) is itself an approximation in the limit of infinite $\omega$).

Fig. 2.

Snapshots at time $t=2\Delta t=0.06$ (left) and $t=100\Delta t=2$ (right) of histograms computed with 128 bins in $x$ and $y$ from 16,384 particles evolving under (5.2) with $\omega =1$ (top) and $\omega =20$ (bottom).

Fig. 3.

Convergence of $\widehat{\sigma }$ (left) and $\nabla \widehat{V}$ (middle), recall $\Vert \cdot \Vert$ denotes the ${\ell }_2$ norm, for (5.2) with $\omega \in \{1,20\}$, as well as $\mathrm{TPR}(\widehat{\mathbf{w}})$ (right). For $\omega =1$, results are compared to the exact model (5.2), while for $\omega =20$ results are compared to the homogenized equation (5.3).

5.2. One-dimensional nonlocal model

We simulate the evolution of particle systems under the quadratic attraction/Newtonian repulsion potential 1 2

$$K_{\text{QANR}}(x)=\frac{1}{2}{x}^2-\mid x\mid$$ (5.4)

with no external potential ($V=0$). The $-\mid x\mid$ portion of $K_{\text{QANR}}$, leading to a discontinuity in $\nabla K$, is the one-dimensional free-space Green’s function for $-\Delta$. For $d\ge 1$, when replaced by the corresponding Green’s function in $d$ dimensions, the distribution of particles evolves under $K_{\text{QANR}}$ into the characteristic of the unit ball in ${\mathbb{R}}^d$, which has implications for design and control of autonomous systems [54]. We compare three diffusivity profiles, $\sigma (x)=0$ corresponding to zero intrinsic noise, $\sigma (x)=\sqrt{2(0.1)}$ leading to constant-diffusivity intrinsic noise, and $\sigma (x)=\sqrt{2(0.1)}\mid x-2\mid$ leading to variable-diffusivity intrinsic noise. With zero intrinsic noise $(\sigma (x)=0)$, we examine the effect of extrinsic noise on recovery, and assume uncertainty in the particle positions due to measurement noise at each timestep, $\mathbb{Y}=\mathbb{X}+\epsilon$, for $\epsilon \sim \mathcal{N}(0,{ϵ}^2\Vert {\mathbf{X}}_{\mathbf{t}}{\Vert }_{\text{RMS}}^2)$ i.i.d. and $ϵ\in$ {0.01, 0.0316, 0.1, 0.316}. In this way $ϵ$ is the noise ratio, such that $\Vert \epsilon {\Vert }_F∕\Vert \mathbb{X}{\Vert }_F\approx ϵ$ (computed with $\epsilon$ and $\mathbb{X}$ stretched into column vectors).

Measurement data consists of 100 timesteps at resolution $\Delta t=0.01$, coarsened from simulations with timestep 0.001. Initial particle positions are drawn from a mixture of three Gaussians each with standard deviation 0.005. Histograms are constructed with 256 bins of width $h=0.0234$. Typical histograms for each noise level are shown in Fig. 4 computed one experiment with $N=8000$ particles.

Fig. 4.

Histograms computed with 256 bins width $h=0.0234$ from 8000 particles in 1D evolving under $K^{\star }=K_{\mathrm{QANR}}(x)$ (5.4). Top left to top right: ${\sigma }^{\star }=0$, ${\sigma }^{\star }(x)=\sqrt{2(0.1)}$, ${\sigma }^{\star }(x)=\sqrt{2(0.1)}\mid x-2\mid$. Bottom: deterministic particles with i.i.d. Gaussian noise added to particle positions with resulting noise ratios (left to right) $ϵ=0.0316$, 0.1, 0.316.

For the case of extrinsic noise (Fig. 5), we use only one experiment ($M=1$) and examine the number of particles $N$ and the noise ratio $ϵ$. We find that recovery is accurate and reliable for $ϵ\le 0.1$, yielding correct identification of $K_{\text{QANR}}$ with less than 1% relative error in at least 98/100 trials. Increasing $N$ from 500 to 8000 leads to minor improvements in accuracy for $ϵ\le 0.1$, but otherwise has little effect, implying that for low to moderate noise levels the mean-field equations are readily identifiable even from smaller particle systems. For $ϵ={10}^{-1∕2}\approx 0.3162$ (see Fig. 4 (bottom right) for an example histogram), we observe a decrease in $\text{TPR}(\widehat{\mathbf{w}})$ (Fig. 5 middle panel) resulting from the generic identification of a linear diffusion term $v{\partial }_{xx}u$ with $v\approx 0.05$. Using that $\sqrt{2v}\approx \sqrt{2(0.05)}=ϵ$, we can identify this as the best-fit intrinsic noise model. Furthermore, increases in $N$ lead to reliable identification of the drift term, as measured by $\text{TPR}({\widehat{\mathbf{w}}}_{drift})$ (rightmost panel Fig. 5) which is the restriction of TPR to drift terms ${\mathbb{L}}_K$ and ${\mathbb{L}}_V$.

For constant diffusivity $\sigma (x)=\sqrt{2(0.1)}$ (Fig. 6), the full model is recovered with less than 3% errors in $\widehat{K}$ and $\widehat{\sigma }$ in at least 98/100 trials when the total particle count $NM$ is at least 8000, and yields errors less than 1% for $NM\ge 16,000$. The error trends for $\widehat{K}$ and $\widehat{\sigma }$ in this case both strongly agree with the predicted $\mathcal{O}(N^{-1∕2})$ rate. For non-constant diffusivity $\sigma (x)=\sqrt{2(0.1)}\mid x-2\mid$ (Fig. 7), we also observe robust recovery $(\text{TPR}(\widehat{\mathbf{w}})\ge 0.95)$ for $NM\ge 8000$ with error trends close to $\mathcal{O}(N^{-1∕2})$, although the accuracy in $\widehat{K}$ and $\widehat{\sigma }$ is diminished due to the strong order $\Delta t^{1∕2}$ convergence of Euler–Maruyama applied to diffusivities $\sigma$ that are unbounded in $x$ [55].

Fig. 6.

Recovery of (3.1) in one spatial dimension for $K^{\star }=K_{\mathrm{QANR}}$ and ${\sigma }^{\star }=\sqrt{2(0.1)}$.

Fig. 7.

Recovery of (3.1) in one spatial dimension for $K^{\star }=K_{\mathrm{QANR}}$ and ${\sigma }^{\star }=\sqrt{2(0.1)}\mid x-2\mid$.

5.3. Two-dimensional nonlocal model

We now discuss an example of singular interaction in two spatial dimensions using the logarithmic potential

$$K(x)=\frac{1}{2\pi }\log \mid x\mid$$ (5.5)

with constant diffusivity $\sigma (x)=\sigma \in \{0,\frac{1}{\sqrt{4\pi }}\}$. This example corresponds to the parabolic–elliptic Keller–Segel model of chemotaxis, where ${\sigma }_c≔\frac{1}{\sqrt{4\pi }}$ is the critical diffusivity such that $\sigma >{\sigma }_c$ leads diffusion-dominated spreading of particles throughout the domain (vanishing particle density at every point in ${\mathbb{R}}^2$) and $\sigma <{\sigma }_c$ leads to aggregation-dominated concentration of the particle density to the dirac-delta located at the center of mass of the initial particle density [44,56]. For $\sigma =0$ we examine the affect of additive i.i.d. measurement noise $\epsilon \sim \mathcal{N}(0,{ϵ}^2\Vert {\mathbf{X}}_{\mathbf{t}}{\Vert }_{\text{RMS}}^2)$ for $ϵ\in$ {0.01, 0.0316, 0.1, 0.316, 1}.

We simulate the particle system with a cutoff potential

$$K_{\delta }(x)=\{\begin{array}{cc}\frac{1}{2\pi }\left(\log (\delta )-1+\frac{\mid x\mid }{\delta }\right),\hfill & \mid x\mid <\delta \hfill \\ \frac{1}{2\pi }\log \mid x\mid ,\hfill & \mid x\mid \ge \delta \hfill \end{array}\phantom{\}}$$ (5.6)

with $\delta =0.01$, so that $K_{\delta }$ is Lipschitz and $\nabla K_{\delta }$ has a jump discontinuity at the origin. Initial particle positions are uniformly distributed on a disk of radius 2 and the particle position data consists of 81 timepoints recorded at a resolution $\Delta t=0.1$, coarsened from 0.0025. Histograms are created with 128 × 128 bins in $x$ and $y$ of sidelength $h=0.0469$ (see Fig. 8 for histogram snapshots over time). We examine $M=2^0,\dots ,2^6$ experiments with $N=2000$ or $N=4000$ particles.

Fig. 8.

Histograms created from 4000 particles evolving under logarithmic attraction (Eq. (5.5) with varying noise levels at times (left to right) $t=4$, $t=8$, and $t=12$. Top: $ϵ=0.316$, $\sigma =0$ (extrinsic only). Bottom: $ϵ=0$, $\sigma ={(4\pi )}^{-1∕2}\approx 0.28$ (intrinsic only).

In Fig. 9 we observe a similar trend in the $\sigma =0$ case as in the 1D nonlocal example, namely that recovery for $ϵ\le 0.01$ is robust with low errors in $\widehat{K}$ (on the order of 0.0032), only in this case the full model is robustly recovered up to $ϵ=0.316$. At $ϵ=1$, with $N=4000$ the method frequently identifies a diffusion term $v\Delta u$ with $v\approx 0.5={ϵ}^2∕2$, and for $N=2000$ the method occasionally identifies the backwards diffusion equation ${\partial }_t{\mu }_t=-\alpha \Delta {\mu }_t$, $\alpha >0$. This is easily prevented by enforcing positivity of $\sigma$, however we leave this and other constraints as an extension for future work.

With diffusivity $\sigma =\frac{1}{\sqrt{4\pi }}$, we obtain $\text{TPR}(\widehat{\mathbf{w}})$ approximately greater than 0.95 for $NM\ge 16,000$ (Fig. 10, right), with an error trend in $\widehat{K}$ following an $\mathcal{O}(N^{-1∕2})$ rate, and a trend in $\widehat{\sigma }$ of roughly $\mathcal{O}(N^{-2∕3})$. Since convergence in $M$ for any fixed $N$ is not covered by the theorem above, this shows that combining multiple experiments may yield similar accuracy trends for moderately-sized particle systems.

Fig. 10.

Recovery of (3.1) in two spatial dimensions with $K^{\star }$ given by (5.5) and ${\sigma }^{\star }=\frac{1}{\sqrt{4\pi }}$.

6. Discussion

We have developed a weak-form method for sparse identification of governing equations for interacting particle systems using the formalism of mean-field equations. In particular, we have investigated two lines of inquiry, (1) is the mean-field setting applicable for inference from medium-size batches of particles? And (2) can a low-cost, low-regularity density approximation such as a histogram be used to enforce weak-form agreement with the mean-field PDE? We have demonstrated on several examples that the answer is yes to both questions, despite the fact that the mean-field equations are only valid in the limit of infinitely many particles ($N\to \infty$). This framework is suitable for systems of several thousand particles in one and two spatial dimensions, and we have proved convergence in $N$ for the associated least-squares problem using simple histograms as approximate particle densities. In addition, the sparse regression approach allows one to identify the full system, including interaction potential $K$, local potential $V$, and diffusivity $\sigma$.

It was initially unclear whether the mean-field setting could be utilized in weak form for finite particle batches, hence this can be seen as a proof of concept for particle systems with $N$ in the range 10³ – 10⁵. With convergence in $N$ and low computational complexity, our weak-form approach is well-suited as is for much larger particle systems. In the opposite regime, for small fixed $N$, the authors of [26] show that their maximum likelihood-based method converges as $M\to \infty$ (i.e. in the limit of infinite experiments). While the same convergence does not hold for our weak-form method, the results in Section 5 suggest that in practice, combining $M$ independent experiments each with $N$ particles improves results. Furthermore, we include evidence in Appendix A.3 that even for small $N$, our method correctly identifies the mean-field model when $M$ is large enough, with performance similar to that in [26]. We leave a full investigation of the interplay between $M$ and $N$ to future work.

In the operable regime of $N>{10}^3$, there is potential for improvements and extensions in many directions. On the subject of density estimation, histograms are highly efficient, yet they lead to piecewise-constant approximations of ${\mu }_t$ and hence $\mathcal{O}(h)$ errors. Choosing a density kernel $G$ to achieve high-accuracy quadrature without sacrificing the $\mathcal{O}(N)$ runtime of histogram computation seems prudent, although one must be cautious about making assumptions on the smoothness of mean-field distribution ${\mu }_t$. For instance, in the 1D nonlocal example 5.2, discontinuities develop in ${\mu }_t$ for the case $\sigma =0$, hence a histogram approximation is more appropriate than using e.g. a Gaussian kernel.

The computational grid $\mathbf{C}$, quadrature method $\langle \cdot ,\cdot {\rangle }_{h,\Delta t}$, and reference test function $\psi$ may also be optimized further or adapted to specific problems. The approach chosen here of $\mathbf{C}$ equally-spaced and separable piecewise-polynomial $\psi$, along with integration using the trapezoidal quadrature, has several advantages, including high accuracy and fast computation using convolutions. However, this may need adjustment for higher dimensions. It might be advantageous to adapt $\mathbf{C}$ to the data $\mathbb{Y}$, however this may prevent one from evaluating ($\mathbf{G}$, $\mathbf{b}$) using the FFT if a non-uniform grid results, hence increases the overall computational complexity. One could also use multiple reference test functions $\psi$. The possibilities of varying the test functions (within the smoothness requirements of the library $\mathbb{L}$) have been largely unexplored in weak-form identification methods.

Several theoretical questions remain unanswered, namely model recovery statistics for finite $N$. As a consequence of Theorem 1, as well as convergence results on sequential thresholding [47], we have that $\mathbf{G}$ being full-rank and $\mathbb{L}$ containing the true model is sufficient to guarantee convergence $\widehat{\mathbf{w}}\to {\mathbf{w}}^{\star }$ as $N\to \infty$ at the rate $\mathcal{O}(N^{-1∕2})$. Noise, whether extrinsic or intrinsic, for finite $N$ may result in identification of an incorrect model when $\mathbf{G}$ is poorly-conditioned. The effect is more severe if the true model has a small coefficient, which requires a small threshold λ, which correspondingly may lead to a non-sparse solution. These are sensitivities of any sparse regression algorithm (see e.g. [57]) and accounting for the effect of noise and poor conditioning is an active area of research in equation discovery.

We also note that several researchers have focused on the uniqueness in kernel identifiability [34,58]. This issue does not directly apply to our scenario²¹ of identifying the triple ($K$, $V$, $\sigma$). Moreover, in the cases we considered, we do not see any identifiability issues (e.g. rank deficiency) even in the high noise case with low particle number. Quantifying the transition to identifiability as $N\to \infty$ as a function of the condition number $\kappa (\mathbf{G})$ is an important subject for future work.

For extensions, the example system (5.2) and resulting homogenization motivates further study of effective equations for systems with complex microstructure. In other fields this is described as coarse-graining. A related line of study is inference of 2nd-order particle systems, as explored in [32], which often lead to an infinite hierarchy of mean-field equations. Our weak-form approach may provide a principled method for truncated and closing such hierarchies using particle data. Another extension is to enforce convex constraints in the regression problem, such as lower bounds on diffusivity, or $K$ with long-range attraction depending on the distribution ${\rho }_{\pi }\in \mathcal{P}([0,\infty ))$ of pairwise distances (see [26] for further use of ${\rho }_{\pi }$). Finally, the framework we have introduced can easily be used to find nonlocal models from continuous solution data (e.g. given $\mathbf{U}$ instead of $\mathbb{Y}$), whereby questions of nonlocal representations of models can be investigated.

Lastly, we note that MATLAB code is available at https://github.com/MathBioCU/WSINDy_IPS.

Table 3.

Trial function library for nonlocal 1D example (Section 5.2).

Mean-field term	Trial function library
$\nabla \cdot (U\nabla K\ast U)$	$\nabla \cdot (U\nabla {\mid x\mid }^m\ast U)$, $m\in \{1,2,3,4,5,6,7\}$ {1, 2, 3, 4, 5, 6, 7}
$\nabla \cdot (U\nabla V)$	${\partial }_x(U{x}^m)$, $m\in \{0,2,3,4,5,6,7,8\}$ {0, 2, 3, 4, 5, 6, 7, 8}
$\frac{1}{2}{\sum }_{i,j=1}^d\frac{{\partial }^2{(U\sigma {\sigma }^T)}_{ij}}{\partial x_i\partial x_j}$	${\partial }_{xx}(U{x}^m)$, $m\in \{0,1,2,3,4,5,6,7,8\}$ {0, 1, 2, 3, 4, 5, 6, 7, 8}

Table 4.

Trial function library for nonlocal 2D example (Section 5.3). Interaction potentials ${[K]}_{\delta }$ indicate cutoff potentials of the form (5.6) with $\delta =0.01$ such that the resulting potential is Lipschitz.

Mean-field term	Trial function library
$\nabla \cdot (U\nabla K\ast U)$	$\{\begin{array}{cc}\nabla \cdot (U\nabla {\mid x\mid }^m\ast U)\hfill & m\in \{2,3,4,5,6\}\hfill \\ \nabla \cdot (U\nabla {\left[{\mid x\mid }^{1∕2}\right]}_{\delta }\ast U)\hfill & \hfill \\ \nabla \cdot (U\nabla {[\mid x\mid (\log \mid x\mid -1)]}_{\delta }\ast U)\hfill & \hfill \\ \nabla \cdot (U\nabla {[\log \mid x\mid ]}_{\delta }\ast U)\hfill & \hfill \end{array}\phantom{\}}$
$\nabla \cdot (U\nabla V)$	${\partial }_{x_i}\left(U{x}_1^m{x}_2^n\right)0\le m+n\le 5,i\in \{1,2\}$
$\frac{1}{2}{\sum }_{i,j=1}^d\frac{{\partial }^2{(U\sigma {\sigma }^T)}_{ij}}{\partial x_i\partial x_j}$	$\Delta (U\cos (m{x}_1)\cos (n{x}_2))$, $(m,n)\in \{0,1,2\}$

Appendix

A.1. Specifications for examples

In Tables 2-5 we include hyperparameter specifications and resulting attributes of Algorithm 4.1 applied to the three examples in Section 5. In particular, we report the typical walltime in Table 5, showing that on each example Algorithm 4.1 learns the mean-field equation from a dataset with ~64,000 particles in under 10 s.

Table 2.

Trial function library for local 2D example (Section 5.1).

Mean-field term	Trial function library
$\nabla \cdot (U\nabla K\ast U)$	$\nabla \cdot (U\nabla {\mid x\mid }^m\ast U)$, $m\in \{1,2,3,4,5,6,7\}$ {1, 2, 3, 4, 5, 6, 7}
$\nabla \cdot (U\nabla V)$	${\partial }_{x_i}(U\cos (m{x}_1)\cos (n{x}_2))$, $(m,n)\in \{0,1,2,3,4,5\}$ {0, 1, 2, 3, 4, 5}, $i\in \{1,2\}$ {1, 2}
$\frac{1}{2}{\sum }_{i,j=1}^d\frac{{\partial }^2{(U\sigma {\sigma }^T)}_{ij}}{\partial x_i\partial x_j}$	$\Delta (U\cos (m{x}_1)\cos (n{x}_2))$, $(m,n)\in \{0,1,2,3,4,5\}$ {0, 1, 2, 3, 4, 5}

Table 5.

Discretization parameters and general information for examples. The number of nonzeros in the true weight vector $\Vert {\mathbf{w}}^{\star }{\Vert }_0$ is given for each parameter set examined. Namely, for the local 2D example, $\omega =1$ results in a 4-term model, while the homogenized case $\omega =20$ results in a three-term model. For the nonlocal 1D example, $\sigma \in \{0,\sqrt{2(0.1)},\sqrt{2(0.1)}\mid x-2\mid \}$ result in 2-term, 3-term, and 5-term models, respectively, and for the nonlocal 2D example $\sigma \in \{0,(4\pi {)}^{-1}\}$ results in 1-term and 2-term models. The norm ${\Vert {\mathbf{G}}^{†}\Vert }_1$, condition number ${\kappa }_2(\mathbf{G})$ and walltime are listed for representative samples with 64,000 total particles.

Example	($m_x$, $m_t$)	($p_x$, $p_t$)	($s_x$, $s_t$)	size($\mathbf{U}$)	($h$, $\Delta t$)
Local 2D	(31,16)	(5,3)	(10,5)	128 × 128 × 101	(0.078, 0.02)
Nonlocal 1D	(29,8)	(5,3)	(5,1)	256 × 101	(0.023, 0.01)
Nonlocal 2D	(25,8)	(5,3)	(8,1)	128 × 128 × 81	(0.047, 0.1)
Example	$\Vert {\mathbf{w}}^{\star }{\Vert }_0$	size($\mathbf{G}$)	$\Vert {\mathbf{G}}^{†}{\Vert }_1$	${\kappa }_2(\mathbf{G})$	Walltime
Local 2D	{4, 3}	686 × 85	2.0 × 103	3.0 × 107	9.2 s
Nonlocal 1D	{2, 3, 5}	3400 × 24	1.3 × 105	8.7 × 108	0.7 s
Nonlocal 2D	{1, 2}	6500 × 59	1.1 × 104	6.4 × 106	8.5 s

A.2. Derivation of homogenized equation (5.3)

We briefly provide a derivation of the homogenized equation (5.3) in the static case. Let $\Omega \in {\mathbb{R}}^d$ be an open bounded domain with smooth boundary and ${\mathbb{T}}^d$ be the $d$-dimensional torus. Let $a(x,y):\Omega \times {\mathbb{T}}^d\to \mathbb{R}$ be continuous and uniformly bounded below,

$$a(x,y)\ge \alpha >0,(x,y)\in \Omega \times {\mathbb{T}}^d.$$

Then for any $f\in L^2(\Omega )$, the equation

$$-\Delta (a(x,x∕ϵ)u^{ϵ}(x))=f(x),u^{ϵ}{\mid }_{\partial \Omega }=0$$

has a unique weak solution $u^{ϵ}\in L^2(\Omega )$ given by

$$u^{ϵ}(x)=\frac{(Gf)(x)}{a(x,x∕ϵ)},$$

where $G$ is the Green’s function for $(-\Delta {)}^{-1}$ with homogeneous Dirichlet boundary conditions on $\partial \Omega$. By the coercivity of $a$ we have that $\Vert u^{ϵ}{\Vert }_{L^2(\Omega )}$ is uniformly bounded in $ϵ$. By the lemma in [59, Section 2.4], up to a subsequence $\{{ϵ}_j{\}}_{j\in \mathbb{N}}$, there exists a function $u(x,y)$ periodic in its second variable such that for any continuous function $\varphi (x,x∕ϵ)$, we have

$$\underset{ϵ\to 0}{\text{lim}}\int u^{ϵ}(x)\varphi (x,x∕ϵ)dx=\iint u(x,y)\varphi (x,y)dydx.$$

Setting $\varphi (x,y)=\varphi (x)$, we see that on the same subsequence, $u^{ϵ}\rightharpoonup \int u(x,y)dy$. Applying the same lemma to the constant series $u^{ϵ}=1$ and letting $\varphi (x,x∕ϵ)=\varphi (x)a^{-1}(x,x∕ϵ)$, we see that (up to possibly a second subsequence),

$$a^{-1}(x,x∕ϵ)\rightharpoonup \int \frac{dy}{a(x,y)}.$$

Letting $a^{\ast }(x)≔{\left(\int \frac{dy}{a(x,y)}\right)}^{-1}$ and putting together the previous limits, we see that

$$u^{ϵ}(x)\rightharpoonup u^{\ast }(x)≔\int u(x,y)dy=(Gf)(x)\int \frac{dy}{a(x,y)}≕\frac{(Gf)(x)}{a^{\ast }(x)},$$

and hence $u^{\ast }$ solves the homogenized equation

$$\Delta (a^{\ast }{u}^{\ast })=f.$$

A.3. Recovery for small $N$ and large $M$

The related maximum-likelihood approach [26] is shown to be suitable for small $N$ and large $M$, hence a natural line of inquiry is the performance of Algorithm 4.1 in this regime. Theorem 1 does not apply to this regime, and in fact convergence of the algorithm is not expected: letting $U_t^{M,N}=\frac{1}{M}{\sum }_{m=1}^M{U}_t^{(m),N}$ where $U_t^{(m),N}$ is the approximate density constructed from experiment $m$ with $N$ particles, we have the weak-measure convergence $U_t^{M,N}\to {\rho }_t^{(1),N}$ as $M\to \infty$, where ${\rho }_t^{(1),N}$ is the 1-particle marginal of the distribution of ${\mathbf{X}}_t$ in ${\mathbb{R}}^{Nd}$. Unlike the mean-field distribution ${\mu }_t$, ${\rho }_t^{(1),N}$ is not a weak solution to the mean-field Fokker–Planck equation (3.1), instead we have

$$\begin{gathered} {\partial }_t{\rho }_t^{(1),N}=\frac{N-1}{N}\nabla \cdot {\int }_{{\mathbb{R}}^d}\nabla K(x-y){\rho }_t^{(2),N}(x,y)dy+\nabla \cdot \left(\nabla V{\rho }_t^{(1),N}\right) \\ +\frac{1}{2}\sum _{i,j=1}^d{\partial }_{x_i{x}_j}(\sigma {\sigma }^T{\rho }_t^{(1),N}), \end{gathered}$$

holding weakly, which depends on the 2-particle marginal ${\rho }_t^{(2),N}$ [35]. Nevertheless, using the 1D nonlocal example in Section 5.2 with $\sigma =\sqrt{2(0.1)}\approx 0.45$, we observe in Fig. 11 (right panel) that our weak-form algorithm correctly identifies the model in > 96% of trials with just $N=10$ particles per experiment when $M\in [2^{10},2^{12}]$, and that error in $K$ (left panel) follows a $\mathcal{O}(M^{-1∕2})$ trend. At $M=4096\approx {10}^{3.61}$ experiments the error²² in $K$ is less than 1% and the runtime is approximately 0.9 s. The lack of convergence in $M$ is reflected in the diffusivity (middle panel of Fig. 11), where the error appears to plateau at around 1.7% for $h\approx 0.0468$ and at 3.5% for $h\approx 0.0234$. The lower resolution (larger binwidth $h$) appears to yield slightly better results, possibly indicating that larger $h$ produces a coarse-graining effect such that ${\rho }^{(2),N}\approx {\rho }^{(1),N}\otimes {\rho }^{(1),N}$ over larger distances, although this effect deserves more thorough study in future work.

A.4. Technical lemmas

We now prove Lemmas 2-4 under Assumption H. First, some consequences of Assumption H. (I) The $\eta$-Hölder continuity of sample paths (H.1) implies that for each $t\in [0,T]$,

$${\int }_{{\mathbb{R}}^d}\mid x{\mid }^{p}d{\mu }_t^N=\frac{1}{N}\sum _{i=1}^N\mid X_t^{(i)}{\mid }^p\le \frac{2^p}{N}\sum _{i=1}^N\mid X_0^{(i)}{\mid }^p+C_{\eta }{2}^p{t}^{p\eta }.$$

Together with the $p\mathrm{th}$ moment bound on ${\mu }_0$ (H.2), this implies

$$\mathbb{E}\left[\underset{t\le T}{\text{sup}}{\int }_{{\mathbb{R}}^d}\mid x{\mid }^{p}d{\mu }_t^N\right]\le 2^p(M_p+C_{\eta }{T}^{p\eta }),$$ (A.1)

independent of $N$. (II) The growth bounds on $\nabla K^{\star }$, $\nabla V^{\star }$, and ${\sigma }^{\star }$ (H.3)-(H.4) imply that for some $C>0$,

$$\mid \nabla K^{\star }(x)\mid +\mid \nabla V^{\ast }(x)\mid +\Vert {\sigma }^{\star }(x)({\sigma }^{\star }(x){)}^T{\Vert }_F\le C(1+\mid x{\mid }^p),$$ (A.2)

where ${\Vert \cdot \Vert }_F$ is the Frobenius norm.

Proof of Lemma 2. Applying Itô’s formula to the process $\frac{1}{N}{\sum }_{i=1}^N\psi (X^{(i)},t)$, we get that

$$\mathcal{L}({\mu }^N,\psi ,\langle \cdot ,\cdot \rangle )=\frac{1}{N}\sum _{i=1}^N{\int }_0^T\nabla \psi (X_t^{(i)},t{)}^T{\sigma }^{\star }(X_t^{(i)})d{B}_t^{(i)}.$$

Note that each integral on the right-hand side is a local martingale, since (A.2) and (H.5) ensure boundedness of $\nabla \psi {(x,t)}^T{\sigma }^{\star }(x)$ over any compact set in ${\mathbb{R}}^d$, hence has mean zero. By independence of the Brownian motions $B_t^{(i)}$, exchangeability of $X_t^{(i)}$, the moment bound (A.1), and the growth bounds on $\sigma$ (H.4), the Itô isometry gives us

$$\begin{gathered} \mathbb{E}[\mathcal{L}({\mu }^N,\psi ,\langle \cdot ,\cdot ,\rangle {)}^2] \\ =\frac{1}{N}{\int }_0^T{\mathbb{E}}_{X\sim {\rho }_t^{(1)}}\left[\mid \nabla \psi (X,t{)}^T{\sigma }^{\star }(X){\mid }^2\right]dt \\ =\frac{1}{N}{\int }_0^T\mathbb{E}\left[{\int }_{{\mathbb{R}}^d}\mid \nabla \psi (x,t{)}^T{\sigma }^{\star }(x){\mid }^{2}d{\mu }_t^N(x)\right]dt \\ \le \frac{C^{\prime }}{N}\Vert \nabla \psi {\Vert }_{2,\infty }^2{\int }_0^T\mathbb{E}\left[1+{\int }_{{\mathbb{R}}^d}\mid x{\mid }^{p}d{\mu }_t^N(x)\right]dt \\ \le C{N}^{-1} \end{gathered}$$

where $K^{\star }=K_{\mathrm{QANR}}$ depends on ${\sigma }^{\star }=\sqrt{2(0.1)}$, $N=10$, $C$, and $M_p$. The result follows from Jensen’s inequality.²³

FIG. 11.

Recovery of (3.1) in one spatial dimension for $C_p$ and $T$ with only $\psi$ particles per experiment.

Proof of Lemma 3. Using the notation $f^{\mathbf{C}}$ from Lemma 1 to denote piecewise constant approximation of a function $f$ over the domain $\mathcal{D}$ using the grid $\mathbf{C}$, we have

$$\begin{gathered} \mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_h)-\mathcal{L}({\mu }^N,\psi ,\langle \cdot ,\cdot \rangle ) \\ =-\underset{E_{\text{interact}}}{\underbrace{\left(\langle (\nabla \psi \cdot ((\nabla K^{\star }{)}^{\mathbf{C}}\ast {\mu }^N){)}^{\mathbf{C}},{\mu }^N\rangle -\langle \nabla \psi \cdot \nabla K^{\star }\ast {\mu }^N,{\mu }^N\rangle \right)}} \\ +\langle {\partial }_t{\psi }^{\mathbf{C}}-{\partial }_t\psi ,{\mu }^N\rangle -\langle (\nabla \psi \cdot \nabla V^{\ast }{)}^{\mathbf{C}}-\nabla \psi \cdot \nabla V^{\ast },{\mu }^N\rangle \\ +\frac{1}{2}\langle \text{Tr}({\nabla }^2\psi {\sigma }^{\star }({\sigma }^{\star }{)}^T{)}^{\mathbf{C}}-\text{Tr}({\nabla }^2\psi {\sigma }^{\star }({\sigma }^{\star }{)}^T),{\mu }^N\rangle \\ =E_{\text{interact}}+E_{\text{linear}}. \end{gathered}$$

The right-hand side includes an interaction error $E_{\text{interact}}$ followed by a sum $E_{\text{linear}}$ of terms that are linear in the difference between a locally Lipschitz function and its piecewise constant approximation. Hence, we can bound $E_{\text{linear}}$ using smoothness of $\psi$ (H.5), the moment assumptions on ${\mu }_t^N$ (H.2), and the growth assumptions on $V$ and $\sigma$ (H.3)-(H.4). Specifically, for $x\in B_k$ with center ${\mathbf{c}}_k$, the growth assumptions imply

$$\begin{gathered} \mid \nabla \psi (x)\cdot \nabla V^{\star }(x)-\nabla \psi ({\mathbf{c}}_k)\cdot \nabla V^{\star }({\mathbf{c}}_k)\mid \\ \le Ch\left((\Vert \nabla \psi {\Vert }_{2,\infty }+\text{Lip}(\nabla \psi ))(1+\mid x{\mid }^p)\right) \\ \mid \text{Tr}({\nabla }^2\psi (x){\sigma }^{\star }(x)({\sigma }^{\star }(x){)}^T)-\text{Tr}({\nabla }^2\psi ({\mathbf{c}}_k){\sigma }^{\star }({\mathbf{c}}_k)({\sigma }^{\star }({\mathbf{c}}_k){)}^T)\mid \\ \le C^{\prime }h\left((\Vert {\nabla }^2\psi {\Vert }_{F,\infty }+\text{Lip}({\nabla }^2\psi ))(1+\mid x{\mid }^p)\right) \end{gathered}$$

for $C$ and $C^{\prime }$ depending on $p$, $d$, and $C_p$, hence

$$\mid E_{\text{linear}}\mid \le C^{\prime \prime }\underset{\mid \alpha \mid \le 2}{\text{sup}}\text{Lip}({\partial }^{\alpha }\psi )\left(T+{\int }_0^T{\int }_{{\mathbb{R}}^d}\mid x{\mid }^{p}d{\mu }_t^{N}dt\right)h.$$ (A.3)

Similarly, for the interaction error we use that for $x\in B_k$ and $y\in B_j$ with centers ${\mathbf{c}}_k$ and ${\mathbf{c}}_j$, we have

$$\begin{gathered} \mid \nabla \psi ({\mathbf{c}}_k)\cdot \nabla K^{\star }({\mathbf{c}}_k-{\mathbf{c}}_j)-\nabla \psi (x)\cdot \nabla K^{\star }(x-y)\mid \\ \le \mid \nabla \psi ({\mathbf{c}}_k)\mid \mid \nabla K^{\star }({\mathbf{c}}_k-{\mathbf{c}}_j)-\nabla K^{\star }(x-y)\mid \\ +\mid \nabla \psi ({\mathbf{c}}_k)-\nabla \psi (x)\mid \mid \nabla K^{\star }(x-y)\mid \\ \le C^{″\prime }h\left(\Vert \nabla \psi {\Vert }_{2,\infty }+\text{Lip}(\nabla \psi )\right)(1+\mid x-y{\mid }^p) \end{gathered}$$

with $C^{″\prime }$ also depending on $p$, $d$, and $C_p$. From this we have

$$\mid E_{\text{interact}}\mid \le C^{\prime \prime \prime \prime }\left(T+{\int }_0^T{\int }_{{\mathbb{R}}^d}{\int }_{{\mathbb{R}}^d}\mid x-y{\mid }^{p}d{\mu }_t^N(y)d{\mu }_t^N(x)dt\right)h.$$ (A.4)

The result follows from taking expectation and using the moment bound (A.1), where the final constant $C$ depends on $p$, $d$, $C_p$, $M_p$, $T$, $\eta$, and $\psi$.

Proof of Lemma 4. Again rewriting the spatial trapezoidal-rule integration in the form ${\int }_{{\mathbb{R}}^d}{\phi }^{\mathbf{C}}(x)d{\mu }_t^N$, we see that

$$\mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_h)-\mathcal{L}(U,\psi ,\langle \cdot ,\cdot {\rangle }_{h,\Delta t})$$ (A.5)

reduces to four terms of the form

$$\begin{gathered} A(\phi )≔\frac{1}{N}\sum _{i=1}^N({\int }_0^T{\phi }^{\mathbf{C}}(X_t^{(i)})dt\phantom{)} \\ \phantom{(}-\frac{\Delta t}{2}\sum _{\ell =1}^L\left({\phi }^{\mathbf{C}}(X_{t_{\ell +1}}^{(i)})+{\phi }^{\mathbf{C}}(X_{t_{\ell }}^{(i)}\right)), \end{gathered}$$

for $\phi \in \left\{{\partial }_t\psi ,\nabla \psi \cdot \nabla V^{\star },\mathrm{Tr}({\nabla }^2\psi {\sigma }^{\star }{({\sigma }^{\star })}^T),\nabla \psi \cdot \nabla K^{\star }\ast {\mu }_t^N\right\}$. Similarly to the bounds derived for $\mid \phi (x)-{\phi }^{\mathbf{C}}(x)\mid$ in Lemma 3, the growth bounds on $K^{\star }$, $V^{\star }$ and ${\sigma }^{\star }$ imply in general that

$$\mid \phi (x)-\phi (y)\mid \le C\mid x-y\mid (1+\max \{\mid x\mid ,\mid y\mid {\}}^p).$$

Rewriting the summands in $A(\phi )$,

$$\begin{gathered} {\int }_0^T{\phi }^{\mathbf{C}}(X_t^{(i)})dt-\frac{\Delta t}{2}\sum _{\ell =1}^L\left({\phi }^{\mathbf{C}}(X_{t_{\ell +1}}^{(i)})+{\phi }^{\mathbf{C}}(X_{t_{\ell }}^{(i)})\right) \\ =\sum _{\ell =1}^L\underset{I_1}{\underbrace{{\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t-t_{\ell }}{\Delta t}\right)({\phi }^{\mathbf{C}}(X_t^{(i)})-{\phi }^{\mathbf{C}}(X_{t_{\ell +1}}^{(i)}))dt}} \\ +\underset{I_2}{\underbrace{{\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t_{\ell +1}-t}{\Delta t}\right)({\phi }^{\mathbf{C}}(X_t^{(i)})-{\phi }^{\mathbf{C}}(X_{t_{\ell }}^{(i)}))dt,}} \end{gathered}$$

and using

$$\begin{gathered} \mid {\phi }^{\mathbf{C}}(x)-{\phi }^{\mathbf{C}}(y)\mid \le \mid \phi (x)-\phi ({\mathbf{c}}_k)\mid +\mid \phi (x)-\phi (y)\mid +\mid \phi (y)-\phi ({\mathbf{c}}_{\ell })\mid \\ \le C(2h+\mid x-y\mid )(1+\max \{\mid x\mid ,\mid y\mid {\}}^p) \end{gathered}$$

where $x\in B_k$ and $y\in B_{\ell }$, we see that for $I_1$,

$$\begin{gathered} \mid {\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t-t_{\ell }}{\Delta t}\right)({\phi }^{\mathbf{C}}(X_t^{(i)})-{\phi }^{\mathbf{C}}(X_{t_{\ell +1}}^{(i)}))dt\mid \\ \le {\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t-t_{\ell }}{\Delta t}\right)C(2h+\mid X_t^{(i)}-X_{t_{\ell +1}}^{(i)}\mid )(1+\max \{\mid X_t^{(i)}\mid ,\mid X_{t_{\ell +1}}^{(i)}\mid {\}}^p)dt \\ \le {\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t-t_{\ell }}{\Delta t}\right)C^{\prime }(2h+\mid t_{\ell +1}-t{\mid }^{\eta }\mid )(1+\max \{\mid X_t^{(i)}\mid ,\mid X_{t_{\ell +1}}^{(i)}\mid {\}}^p)dt. \end{gathered}$$

Taking expectation on both sides and using the moment bound (A.1), we get

$$\begin{gathered} \mathbb{E}\left[\mid {\int }_{t_{\ell }}^{t_{\ell +1}}\left(\frac{t-t_{\ell }}{\Delta t}\right)({\phi }^{\mathbf{C}}(X_t^{(i)})-{\phi }^{\mathbf{C}}(X_{t_{\ell +1}}^{(i)}))dt\mid \right] \\ \le C(\Delta th+\Delta t^{1+\eta }). \end{gathered}$$

We get the same bound for $I_2$. Summing over $\ell$, and taking the average in $i$, we then get

$$\mathbb{E}[\mid A(\phi )\mid ]\le C(h+\Delta t^{\eta }),$$

which implies the desired bound on the difference (A.5).