WEAK SINDy: GALERKIN-BASED DATA-DRIVEN MODEL SELECTION

Abstract

We present a novel weak formulation and discretization for discovering governing equations from noisy measurement data. This method of learning differential equations from data fits into a new class of algorithms that replace pointwise derivative approximations with linear transformations and variance reduction techniques. Compared to the standard SINDy algorithm presented in [S. L. Brunton, J. L. Proctor, and J. N. Kutz, Proc. Natl. Acad. Sci. USA, 113 (2016), pp. 3932–3937], our so-called weak SINDy (WSINDy) algorithm allows for reliable model identification from data with large noise (often with ratios greater than 0.1) and reduces the error in the recovered coefficients to enable accurate prediction. Moreover, the coefficient error scales linearly with the noise level, leading to high-accuracy recovery in the low-noise regime. Altogether, WSINDy combines the simplicity and efficiency of the SINDy algorithm with the natural noise reduction of integration, as demonstrated in [H. Schaeffer and S. G. McCalla, Phys. Rev. E, 96 (2017), 023302], to arrive at a robust and accurate method of sparse recovery.

1. Problem statement.

Consider a first-order dynamical system in $D$ dimensions of the form

$$\frac{d}{dt}\mathbf{x}(t)=\mathbf{F}(x(t)),\mathbf{x}(0)={\mathbf{x}}_0\in {\mathrm{ℝ}}^D,0\le t\le T,$$ (1.1)

and measurement data $\mathbf{y}\in {\mathrm{ℝ}}^{M\times D}$ given at $M$ timepoints $\mathbf{t}={\left(t_1,\dots ,t_M\right)}^T$ by

$${\mathbf{y}}_{md}={\mathbf{x}}_d\left(t_m\right)+{ϵ}_{md},m\in [M],d\in [D],$$

where throughout we use the bracket notation $[M]:=\{1,\dots ,M\}$. The variable $ϵ\in {\mathrm{ℝ}}^{M\times D}$ represents a matrix of independent and identically distributed measurement noise. The focus of this article is the reconstruction of the dynamics (1.1) from the measurements $\mathbf{y}$.

The SINDy algorithm (sparse identification of nonlinear dynamics [4]) has been shown to be successful in solving this problem for sparsely represented nonlinear dynamics when noise is small and dynamic scales do not vary across multiple orders of magnitude. This framework assumes that the function $\mathbf{F}:{\mathrm{ℝ}}^D\to {\mathrm{ℝ}}^D$ in (1.1) is given componentwise by

$${\mathbf{F}}_d(\mathbf{x}(t))=\sum _{j=1}^J{\mathbf{w}}_{jd}^{\star }{f}_j(\mathbf{x}(t))$$ (1.2)

for some known family of functions ${\left(f_j\right)}_{j\in [J]}$ and a sparse weight matrix ${\mathbf{w}}^{\star }\in {\mathrm{ℝ}}^{J\times D}$. The problem is then transformed into solving for ${\mathbf{w}}^{\star }$ by building a data matrix $\text{Θ}(\mathbf{y})\in {\mathrm{ℝ}}^{M\times J}$ given by

$$\text{Θ}{(\mathbf{y})}_{mj}=f_j\left({\mathbf{y}}_m\right),{\mathbf{y}}_m:=\left({\mathbf{y}}_{m1},\dots ,{\mathbf{y}}_{mD}\right),$$

so that the candidate functions are directly evaluated at the noisy data. Solving (1.1) for $\mathbf{F}$ then reduces to identifying a sparse weight matrix $\widehat{\mathbf{w}}$ such that

$$\dot{\mathbf{y}}\approx \text{Θ}(\mathbf{y})\widehat{\mathbf{w}},$$ (1.3)

where $\dot{\mathbf{y}}$ is the numerical time derivative of the data $\mathbf{y}$. Sequential-thresholding least squares is then used to arrive at a sparse solution.

1.1. Background.

Research into statistically rigorous selection of mathematical models from data can be traced back to Akaike’s seminal work in the 1970s [1, 2]. In the last 20 years, there has been substantial work in this area at the interface between applied mathematics, computer science, and statistics (see [3, 11, 12, 19, 22, 23] for both theory and applications). More recently, the formulation of system discovery problems in terms of a candidate basis of nonlinear functions (1.2) and subsequent discretization (1.3) was introduced in [21] in the context of catastrophe prediction. The authors of [21] used compressed sensing techniques to enforce sparsity. Since then there has been an explosion of interest in the problem of identifying nonlinear dynamical systems from data, with some of the primary techniques being Gaussian process regression [15], deep neural networks [16], Bayesian inference [26, 27], and classical methods from numerical analysis [7, 9, 25]. These techniques have been successfully applied to the discovery of both ordinary and partial differential equations.

The variety of discovery algorithms qualitatively differ in the interpretability of the resulting data-driven dynamical system, the scope and efficiency of the algorithm, and the robustness to noise, scale separation, etc. For instance, a neural network based data-driven dynamical system does not easily lend itself to physical interpretation, while the SINDy algorithm identifies governing equations which can be analyzed directly. Moreover, it is also well-known that the training stage for neural networks and other iterative learning algorithms can be computationally costly. Concerning the scope of an algorithm, several methods have been independently developed to discover models under the assumption of some prior knowledge of the governing equations, notably for low-degree polynomial chaotic systems, cyclic ODEs, interacting particles, and Hamiltonian dynamics [20, 18, 13, 24]. In each of these cases the authors derive probabilistic recovery guarantees depending on the number of available trajectories, the size of the candidate model library, the level of incoherence of the data, and/or the sparsity of the governing equations.

The vast majority of algorithms and recovery guarantees assume that pointwise derivatives of the data either are available or can be reliably computed. This severely limits an algorithm’s robustness to noise and hence its applicability to real world data. Here we relax this assumption and provide rigorous justification for the weak formulation of the dynamics as a means to circumvent this ubiquitous problem in model selection. Building off of the SINDy framework, we present the robust discovery algorithm WSINDy (weak SINDy), which operates under the assumption that the time derivative is unavailable and that the only prior knowledge of the governing equations is their inclusion in a large model library. We also focus on the realistic scenario where only a single noisy trajectory of the state variable is available; however, extension to multiple trajectories is of course possible. For simplicity, we restrict numerical experiments to autonomous ODEs for their amenability to analysis. Natural next steps are to explore identification of PDEs and nonautonomous dynamical systems. We note that the use of integral equations for system identification was introduced in [17], where compressed sensing techniques were used to enforce sparsity, and that this technique can be seen as a special case of the method introduced here.

In section 2 we introduce the algorithm with analysis of the resulting error structure. Section 3 contains numerical results showing identification of six ODE systems over a range of noise levels and parameter regimes. In section 4, we provide concluding remarks as well as natural next directions for this line of research. In Appendix A we include a detailed comparison between WSINDy and SINDy as well as further information on the generalized least squares method.

2. WSINDy.

We approach the problem of system identification (1.3) from a nonstandard perspective by utilizing the weak form of the differential equation. Recall that for any smooth test function $\varphi :\mathrm{ℝ}\to \mathrm{ℝ}$ (absolutely continuous is sufficient) and interval $(a,b)\subset [0,T]$, (1.1) admits the weak formulation

$$\varphi (b)\mathbf{x}(b)-\varphi (a)\mathbf{x}(a)-{\int }_a^b{\varphi }^{\prime }(u)\mathbf{x}(u)du={\int }_a^b\varphi (u)\mathbf{F}(\mathbf{x}(u))du,0\le a<b\le T.$$ (2.1)

With $\varphi =1$, we arrive at the integral equation of the dynamics explored in [17]. If we instead take $\varphi$ to be nonconstant and compactly supported in $(a,b)$, we arrive at

$$-{\int }_a^b{\varphi }^{\prime }(u)\mathbf{x}(u)du={\int }_a^b\varphi (u)\mathbf{F}(\mathbf{x}(u))du.$$ (2.2)

Assuming a representation of the form (1.2), we then define the generalized residual $𝓡(\mathbf{w};\varphi )$ for a given test function $\varphi$ by replacing $\mathbf{F}$ with a candidate element from the span of ${\left(f_j\right)}_{j\in [J]}$ and $\mathbf{x}$ with $\mathbf{y}$ as follows:

$$𝓡(\mathbf{w};\varphi ):={\int }_a^b\left({\varphi }^{\prime }(u)\mathbf{y}(u)+\varphi (u)\left(\sum _{j=1}^J{\mathbf{w}}_j{f}_j(\mathbf{y}(u))\right)\right)du.$$ (2.3)

Clearly, with $\mathbf{w}={\mathbf{w}}^{\star }$ and $\mathbf{y}=\mathbf{x}(t)$ we have $𝓡(\mathbf{w};\varphi )=0$ for all $\varphi$ compactly supported in $(a,b)$; however, $y$ is a discrete set of data, so (2.3) can at best be approximated numerically. Measurement noise then presents a significant barrier to accurate indentification of ${\mathbf{w}}^{\star }$.

2.1. Method overview.

For analogy with traditional Galerkin methods, consider the forward problem of solving a dynamical system such as (1.1) for $\mathbf{x}$. The Galerkin approach is to seek a solution $\mathbf{x}$ represented in a chosen trial basis ${\left(f_j\right)}_{j\in [J]}$ such that the residual $𝓡$, defined by

$$𝓡=\int \varphi (t)(\dot{\mathbf{x}}(t)-\mathbf{F}(\mathbf{x}(t)))dt,$$

is minimized over all test functions $\varphi$ living in the span of a given test function basis ${\left({\varphi }_k\right)}_{k\in [K]}$. If the trial and test function bases are known analytically, inner products of the form $\langle f_j,{\varphi }_k\rangle$ appearing in the residual can be computed exactly. Thus, the computational error results only from representing the solution in a finite-dimensional function space.

The method we present here can be considered a data-driven Galerkin method of solving for $\mathbf{F}$ where the trial “basis” is given by the set of gridfunctions ${\left(f_j(\mathbf{y})\right)}_{j\in [J]}$ evaluated at the data and only the test function basis ${\left({\varphi }_k\right)}_{k\in [K]}$ is known analytically. In this way, inner products appearing in $𝓡(\mathbf{w};\varphi )$ must be approximated numerically, implying that the accuracy of the recovered weights $\widehat{\mathbf{w}}$ is ultimately limited by the quadrature scheme used to discretize inner products. Using Lemma 2 below, we show that the correct coefficients ${\mathbf{w}}^{\star }$ may be recovered to effective machine precision accuracy (given by the tolerance of the forward ODE solver) from noise-free trajectories $\mathbf{y}$ by discretizing (2.2) using the trapezoidal rule and choosing $\varphi$ to decay smoothly to zero at the boundaries of its support. Specifically, in this article we demonstrate this fact by choosing test functions from a particular family of unimodal piecewise polynomials $𝓢$ defined in (2.6).

Having chosen a quadrature scheme, the next accuracy barrier is presented by measurement noise, introducing randomness into the residuals $𝓡(\mathbf{w};\varphi )$. Numerical integration then couples residuals $𝓡\left(\mathbf{w};{\varphi }_1\right)$ and $𝓡\left(\mathbf{w};{\varphi }_2\right)$ whenever ${\varphi }_1$ and ${\varphi }_2$ have overlapping support. In this way, $𝓡(\mathbf{w};\varphi )$ does not have an ideal error structure for least squares but may be amenable to generalized least squares. Below we analyze the distribution of the residuals $𝓡(\mathbf{w};\varphi )$ to arrive at a generalized least squares approach where an approximate covariance matrix can be computed directly from the test functions. This analysis also suggests that placing test functions near steep gradients in the dynamics may improve recovery; hence we develop a derivative-free method for adaptively clustering test functions near steep gradients.

Remark 1.

The weak formulation of the dynamics introduces a wealth of information: given $M$ timepoints $\mathbf{t}={\left(t_m\right)}_{m\in [M]}$, (2.2) affords $K=M(M-1)/2$ residuals over all possible supports $(a,b)\subset \mathbf{t}\times \mathbf{t}$ with $a<b$. Of course, one could also assimilate the responses of multiple families of test functions $\left({\left\{{\varphi }_k^1\right\}}_{k\in \left[K_1\right]},{\left\{{\varphi }_k^2\right\}}_{k\in \left[K_2\right]},\dots \right)$; however, the computational complexity of such an exhaustive approach quickly becomes intractable. We stress that even with large noise, our proposed method identifies the correct nonlinearities with accurate weight recovery while keeping the number of test functions lower than the number of timepoints $(K<M)$.

2.2. Algorithm: WSINDy.

We state here the WSINDy algorithm in full generality. We propose a generalized least squares approach with approximate covariance matrix $\text{Σ}$. Below we derive a particular choice of $\text{Σ}$ which utilizes the action of the test functions ${\left({\varphi }_k\right)}_{k\in [K]}$ on the data $\mathbf{y}$. Sequential thresholding on the weight coefficients $\mathbf{w}$ with thresholding parameter $\lambda$ is used to enforce sparsity, where $\lambda \le {\min }_{{\mathbf{w}}^{\star }\ne 0}\left|{\mathbf{w}}^{\star }\right|$ is necessary for recovery. Lastly, an ${\ell }_2$-regularization term with coefficient $\gamma$ is included for problems involving rank deficiency. Methods of choosing optimal values of $\lambda$ and $\gamma$ directly from a given dataset do exist, for instance, by selecting the optimal position in a Pareto front [5]; however, this is not the focus of our current study, and thus we select values that work across multiple examples. Specifically, in the experiments below we set $\gamma =0$ with the exception of the nonlinear pendulum and the five-dimensional linear system, examples which show that regularization can be used to discover dynamics from excessively large libraries. For noise-free data the algorithm is only weakly dependent on $\lambda$ and so we use $\lambda =0.001$, while for noisy data we set $\lambda =\frac{1}{4}{\min }_{{\mathbf{w}}^{\star }\ne 0}\left|{\mathbf{w}}^{\star }\right|$

$$\widehat{\mathbf{w}}=\mathbf{\text{WSINDy}}\left(\mathbf{y},\mathbf{t};{\left({\varphi }_k\right)}_{k\in [K]},{\left(f_j\right)}_{j\in [J]},\text{Σ},\lambda ,\gamma \right):$$

1. Construct matrix of trial gridfunctions $\text{Θ}(\mathbf{y})=\left[f_1(\mathbf{y})|\dots |f_J(\mathbf{y})\right]$.

2. Construct integration matrices $\mathbf{V}$, ${\mathbf{V}}^{\prime }$ such that

   $${\mathbf{V}}_{km}=\text{Δ}t{\varphi }_k\left(t_m\right),{\mathbf{V}}_{km}^{\prime }=\text{Δ}t{\varphi }_k^{\prime }\left(t_m\right).$$

3. Compute Gram matrix $\mathbf{G}=\mathbf{V}\text{Θ}(\mathbf{y})$ and right-hand side $\mathbf{b}=-{\mathbf{V}}^{\prime }\mathbf{y}$ so that ${\mathbf{G}}_{kj}=\langle {\varphi }_k,f_j(\mathbf{y})\rangle$ and ${\mathbf{b}}_{kd}=-\langle {\varphi }_k^{\prime },{\mathbf{y}}_d\rangle$.

4. Solve the generalized least squares problem with ${\ell }_2$-regularization

   $$\widehat{\mathbf{w}}={\text{argmin}}_{\mathbf{w}}\left\{{(\mathbf{G}\mathbf{w}-\mathbf{b})}^T{\text{Σ}}^{-1}(\mathbf{G}\mathbf{w}-\mathbf{b})+{\gamma }^2\parallel \mathbf{w}{\parallel }_2^2\right\},$$

using sequential thresholding with parameter $\lambda$ to enforce sparsity.

With this as our core algorithm, we can now consider a residual analysis (section 2.3) leading to a generalized least squares framework. We can also develop theoretical results related to the test functions (section 2.4), yielding a more thorough understanding of the impact of using uniform (section 2.4.1) and adaptive (section 2.4.2) placement of test functions along the time axis.

2.3. Residual analysis.

Performance of WSINDy is determined by the behavior of the residuals

$$𝓡\left(\mathbf{w};{\varphi }_k\right):={(\mathbf{G}\mathbf{w}-\mathbf{b})}_k\in {\mathrm{ℝ}}^{1\times D},$$

denoted $𝓡(\mathbf{w})\in {\mathrm{ℝ}}^{K\times D}$ for the entire residual matrix. Here we analyze the residual for autonomous $\mathbf{F}$ to highlight key aspects for future analysis, as well as to arrive at an appropriate choice of approximate covariance $\sum$. We also provide a heuristic argument in favor of placing test functions near steep gradients in the dynamics.

A key difficulty in recovering the true weights ${\mathbf{w}}^{\star }$ is that for nonlinear systems the residual evaluated at the true weights ${\mathbf{w}}^{\star }$ is biased: $\mathbb{E}\left[𝓡\left({\mathbf{w}}^{\star }\right)\right]\ne 0$. Any minimization of $𝓡$ thus introduces a bias in the recovered weights $\widehat{\mathbf{w}}$. Nevertheless, we can understand how different test functions impact the residual by linearizing around the true trajectory $\mathbf{x}(t)$ and isolating the dominant error terms :

$$\begin{gathered} 𝓡\left(\mathbf{w};{\varphi }_k\right)=\langle {\varphi }_k,\text{Θ}(\mathbf{y})\mathbf{w}\rangle +\langle {\varphi }_k^{\prime },\mathbf{y}\rangle \\ =\langle {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)\rangle +\langle {\varphi }_k,\text{Θ}(\mathbf{y}){\mathbf{w}}^{\star }\rangle +\langle {\varphi }_k^{\prime },\mathbf{y}\rangle \\ =\langle {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)\rangle +\langle {\varphi }_k,\mathbf{F}(\mathbf{y})-\mathbf{F}(\mathbf{x})\rangle +\langle {\varphi }_k^{\prime },ϵ\rangle +I_k \\ =\underset{R_1}{\underbrace{\langle {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)\rangle }}+\underset{R_2}{\underbrace{\langle {\varphi }_k,ϵ\nabla \mathbf{F}(\mathbf{x})\rangle }}+\underset{R_3}{\underbrace{\langle {\varphi }_k^{\prime },ϵ\rangle }}+I_k+𝓞\left({ϵ}^2\right), \end{gathered}$$

where $\nabla F{(\mathbf{x})}_{d{d}^{\prime }}=\frac{\partial {\mathbf{F}}_{d^{\prime }}}{\partial {\mathbf{x}}_d}(\mathbf{x})$. The errors manifest in the following ways:

• $R_1$ is the misfit between $\mathbf{w}$ and ${\mathbf{w}}^{\star }$.

• $R_2$ results from measurement error in trial gridfunctions: $f_j(\mathbf{y})=f_j(\mathbf{x}+ϵ)\ne f_j(\mathbf{x})$.

• $R_3$ results from replacing $\mathbf{x}$ with $\mathbf{y}=\mathbf{x}+ϵ$ in the left-hand side of (2.2).

• $I_k$ is a deterministic integration error.

• $𝓞\left({ϵ}^2\right)$ is the remainder term in the truncated Taylor expansion of $\mathbf{F}(\mathbf{y})$ around $\mathbf{x}$:

  $$\mathbf{F}\left({\mathbf{y}}_m\right)=\mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)+{ϵ}_m\nabla \mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)+𝓞\left({\left|{ϵ}_m\right|}^2\right).$$

Clearly, recovery of $\mathbf{F}$ when $ϵ=0$ is straightforward: $R_1$ and $I_k$ are the only error terms; thus one only needs to select a quadrature scheme that ensures that the integration error $I_k$ is negligible and $\widehat{\mathbf{w}}={\mathbf{\text{w}}}^{\star }$ will be the minimizer. A primary focus of this study is the use of a specific family of piecewise polynomial test functions $𝓢$ defined below for which the trapezoidal rule is highly accurate (see Lemma 2). Figure 3.1 demonstrates this fact on noise-free data.

FIG. 3.1.

Noise-free data (${\sigma }_{NR}=0$): plots of relative coefficient error $E_2(\widehat{\mathbf{w}})$ (defined in (3.2)) vs. p. V1-V4 indicate different ODE parameters (see Table 2). For the Lorenz system the parameters are fixed, and 40 different initial conditions are sampled from a uniform distribution. In each case, the recovered coefficients $\widehat{\mathbf{w}}$ rapidly converge to within the accuracy of the ODE solver (10⁻¹⁰).

For $ϵ>0$, accurate recovery of $\mathbf{F}$ requires one to choose hyperparameters that exemplify the true misfit term $R_1$ by enforcing that the other error terms are of lower order. We look for ${\left({\varphi }_k\right)}_{k\in [K]}$ and $\text{Σ}=\mathbf{C}{\mathbf{C}}^T$ that approximately enforce ${\mathbf{C}}^{-1}𝓡\left({\mathbf{w}}^{\star }\right)\sim 𝓝\left(0,{\sigma }^2\mathbf{I}\right)$, justifying the least squares approach. In the next subsection we address the issue of approximating the covariance matrix, providing justification for using $\text{Σ}={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T$. The following subsection provides a heuristic argument for how to reduce corruption from the error terms $R_2$ and $R_3$ by placing test functions near steep gradients in the data.

2.3.1. Approximate covariance $\text{Σ}$.

Neglecting the deterministic integration error, which can be made small (see Lemma 2 below), and higher-order noise terms, the residual evaluated at the true weights is approximately

$$𝓡\left({\mathbf{w}}^{\star };{\varphi }_k\right)\approx R_2+R_3,$$

where $\mathbb{E}\left[R_2\right]=\mathbb{E}\left[R_3\right]=(0,\dots ,0)$ implies that $\mathbb{E}\left[𝓡\left({\mathbf{w}}^{\star }\right)\right]=\mathbf{0}$ to leading order. Given the variances

$$\mathbb{V}\left[R_2\right]=\mathbb{V}\left[\langle {\varphi }_k,ϵ\nabla \mathbf{F}(\mathbf{x})\rangle \right]=\text{Δ}t{\sigma }^2\left({\Vert {\varphi }_k\left|\nabla {\mathbf{F}}_1(\mathbf{x})\right|\Vert }_2^2,\dots ,{\Vert {\varphi }_k\left|\nabla {\mathbf{F}}_D(\mathbf{x})\right|\Vert }_2^2\right)$$

and

$$\mathbb{V}\left[R_3\right]=\mathbb{V}\left[\langle {\varphi }_k^{\prime },ϵ\rangle \right]=\text{Δ}t{\sigma }^2\left({\Vert {\varphi }_k^{\prime }\Vert }_2^2,\dots ,{\Vert {\varphi }_k^{\prime }\Vert }_2^2\right),$$

the true distribution of $𝓡\left({\mathbf{\text{w}}}^{\star }\right)$ depends on $\mathbf{F}$, which is not known a priori. If it holds that ${\Vert {\varphi }_k^{\prime }\Vert }_2\gg {\Vert {\varphi }_k\left|\nabla {\mathbf{F}}_d(\mathbf{x})\right|\Vert }_2,d\in [D]$, a leading order approximation to $\text{Cov}\left(𝓡\left({\mathbf{w}}^{\star }\right)\right)$ is

$$\text{Σ}:={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T\propto \text{Cov}\left(R_3\right),$$

using that $\text{Cov}{\left(R_3\right)}_{ij}=\text{Δ}t{\sigma }^2\langle {\varphi }_i^{\prime },{\varphi }_j^{\prime }\rangle$. For this reason, we employ localized test functions and adopt the heuristic $\text{Σ}={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T$ below.

2.3.2. Adaptive refinement.

Next we show that by localizing ${\varphi }_k$ around large $|\dot{\mathbf{\text{x}}}|$, we get an approximate cancellation of the error terms $R_2$ and $R_3$. Consider the one-dimensional case $(D=1)$ where $m$ is an arbitrary time index and ${\mathbf{y}}_m=\mathbf{x}\left(t_m\right)+ϵ$ is an observation. When $\left|\dot{\mathbf{x}}\left(t_m\right)\right|$ is large compared to $ϵ$, we approximately have

$${\mathbf{y}}_m=\mathbf{x}\left(t_m\right)+{ϵ}_m\approx \mathbf{x}\left(t_m+\delta t\right)\approx \mathbf{x}\left(t_m\right)+\delta t\mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)$$ (2.4)

for some small $\delta t$, i.e., the perturbed value ${\mathbf{y}}_m$ lands close to the true trajectory $\mathbf{x}$ at the time $t_m+\delta t$. To understand the heuristic behind this approximation, let $t_m+\delta t$ be the point of intersection between the tangent line to $\mathbf{x}(t)$ at $t_m$ and $\mathbf{x}\left(t_m\right)+ϵ$. Then

$$\delta t=\frac{ϵ}{\dot{\mathbf{x}}\left(t_m\right)};$$

hence $\left|\dot{\mathbf{x}}\left(t_m\right)\right|\gg ϵ$ implies that $\mathbf{x}\left(t_m\right)+ϵ$ will approximately lie on the true trajectory. As well, regions where $\left|\dot{\mathbf{x}}\left(t_m\right)\right|$ is small will not yield accurate recovery in the case of noisy data, since perturbations are more likely to exit the relevant region of phase space. If we linearize $\mathbf{F}$ using the approximation (2.4) we get

$$\mathbf{F}\left({\mathbf{y}}_m\right)\approx \mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)+\delta t{\mathbf{F}}^{\prime }\left(\mathbf{x}\left(t_m\right)\right)\mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)=\mathbf{F}\left(\mathbf{x}\left(t_m\right)\right)+\delta t\ddot{\mathbf{x}}\left(t_m\right).$$ (2.5)

Assuming ${\varphi }_k$ is sufficiently localized around $t_m$, (2.4) also implies that

$$\langle {\varphi }_k^{\prime },\mathbf{x}\rangle +\underset{R_3}{\underbrace{\langle {\varphi }_k^{\prime },ϵ\rangle }}=\langle {\varphi }_k^{\prime },\mathbf{y}\rangle \approx \langle {\varphi }_k^{\prime },\mathbf{x}\rangle +\delta t\langle {\varphi }_k^{\prime },\mathbf{F}(\mathbf{x})\rangle ;$$

hence $R_3\approx \delta t\langle {\varphi }_k^{\prime },\mathbf{F}(\mathbf{x})\rangle$, while (2.5) implies

$$\begin{gathered} \langle {\varphi }_k,\text{Θ}(\mathbf{y})\mathbf{w}\rangle =\underset{=R_1}{\underbrace{\langle {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)\rangle }}+\langle {\varphi }_k,\mathbf{F}(\mathbf{y})\rangle \\ \approx {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)+\langle {\varphi }_k,\mathbf{F}(\mathbf{x})\rangle +\underset{\approx R_2}{\underbrace{\delta t\langle {\varphi }_k,\ddot{\mathbf{x}}\rangle }} \\ =\langle {\varphi }_k,\text{Θ}(\mathbf{y})\left(\mathbf{w}-{\mathbf{w}}^{\star }\right)\rangle +\langle {\varphi }_k,\mathbf{F}(\mathbf{x})\rangle -\delta t\langle {\varphi }_k^{\prime },\mathbf{F}(\mathbf{x})\rangle , \end{gathered}$$

having integrated by parts. Collecting the terms together yields that the residual takes the form

$$𝓡\left(\mathbf{w};{\varphi }_k\right)=\langle {\varphi }_k^{\prime },\mathbf{y}\rangle +\langle {\varphi }_k,\text{Θ}(\mathbf{y})\mathbf{w}\rangle \approx R_1,$$

and we see that $R_2$ and $R_3$ have effectively cancelled. In higher dimensions this interpretation does not appear to be as illuminating, but nevertheless, for any given coordinate ${\mathbf{x}}_d$, it does hold that terms in the error expansion vanish around points $t_m$ where $\left|{\dot{\mathbf{\text{x}}}}_d\right|$ is large, precisely because ${\mathbf{x}}_d\left(t_m\right)+ϵ\approx {\mathbf{x}}_d\left(t_m+\delta t\right)$.

2.4. Test function basis ${\left({\varphi }_k\right)}_{k\in [K]}$

Here we introduce a test function space $𝓢$ and quadrature scheme to minimize integration errors and enact the heuristic arguments above, which rely on ${\varphi }_k$ having fast decay to its support boundaries and being sufficiently localized to ensure ${\Vert {\varphi }_k^{\prime }\Vert }_2^2\gg {\Vert {\varphi }_k\Vert }_2^2$. We define the space $𝓢$ of unimodal piecewise polynomials of the form

$$\varphi (t)=\{\begin{array}{ll}C{(t-a)}^p{(b-t)}^q& t\in [a,b],\\ 0& \text{otherwise,}\end{array}$$ (2.6)

where $(a,b)\subset \mathbf{t}\times \mathbf{t}$ satisfies $a<b$ and $p,q\ge 1$. The normalization

$$C=\frac{1}{p^p{q}^q}{\left(\frac{p+q}{b-a}\right)}^{p+q}$$

ensures that $\parallel \varphi {\parallel }_{\infty }=1$. Functions $\varphi \in 𝓢$ are nonnegative, unimodal, and compactly supported in $[0,T]$ with $\lfloor \min \{p,q\}\rfloor -1$ continuous derivatives. Larger $p$ and $q$ imply faster decay towards the endpoints of the support. For $p=q$, we refer to $p$ as the degree of $\varphi$.

To ensure the integration error in approximating inner products $\langle f_j,{\varphi }_k\rangle$ is negligible, we rely on the following lemma, which provides a bound on the error in discretizing the weak derivative relation

$$-\int {\varphi }^{\prime }fdt=\int \varphi f^{\prime }dt$$ (2.7)

using the trapezoidal rule for compactly supported $\varphi$. Following the lemma we introduce two strategies for choosing the parameters of the test functions ${\left({\varphi }_k\right)}_{k\in [K]}\subset 𝓢$.

Lemma 2 (numerical error in weak derivatives).

Let $f,\varphi$ have continuous derivatives of order $p$, and define $t_j=a+j\frac{b-a}{N}=a+j\text{Δ}t$. If $\varphi$ has roots $\varphi (a)=\varphi (b)=0$ of multiplicity $p$, then

$$\frac{\text{Δ}t}{2}\sum _{j=0}^{N-1}\left[g\left(t_j\right)+g\left(t_{j+1}\right)\right]=𝓞\left(\text{Δ}{t}^{p+1}\right),$$ (2.8)

where $g(t)={\varphi }^{\prime }(t)f(t)+\varphi (t)f^{\prime }(t)$. In other words, the composite trapezoidal rule discretizes the weak derivative relation (2.7) to order $p+1$.

Proof.

This is a simple consequence of the Euler-Maclaurin formula. If $g:[a,b]\to \mathrm{ℂ}$ is a smooth function, then the following asymptotic expansion holds:

$$\frac{\text{Δ}t}{2}\sum _{j=0}^{N-1}\left[g\left(t_j\right)+g\left(t_{j+1}\right)\right]\sim {\int }_a^{b}g(t)dt+\sum _{k=1}^{\infty }\frac{\text{Δ}{t}^{2k}{B}_{2k}}{(2k)!}\left(g^{(2k-1)}(b)-g^{(2k-1)}(a)\right),$$

where $B_{2k}$ are the Bernoulli numbers. The asymptotic expansion provides corrections to the trapezoidal rule that realize machine precision accuracy up until a certain value of $k$, after which terms in the expansion grow and the series diverges [6, Chapter 3]. In our case, $g(t)={\varphi }^{\prime }(t)f(t)+\varphi (t)f^{\prime }(t)$, where the root conditions on $\varphi$ imply that

$${\int }_a^{b}g(t)dt=0\text{and}{g}^{(k)}(b)=g^{(k)}(a)=0,0\le k\le p-1.$$

So for $p$ odd, we have that

$$\begin{gathered} \frac{\text{Δ}t}{2}\sum _{j=0}^{N-1}\left[g\left(t_j\right)+g\left(t_{j+1}\right)\right]\sim \sum _{k=(p+1)/2}^{\infty }\frac{\text{Δ}{t}^{2k}{B}_{2k}}{(2k)!}\left(g^{(2k-1)}(b)-g^{(2k-1)}(a)\right) \\ =\frac{B_{p+1}}{(p+1)!}\left({\varphi }^{(p)}(b)f(b)-{\varphi }^{(p)}(a)f(a)\right)\text{Δ}{t}^{p+1}+𝓞\left(\text{Δ}{t}^{p+2}\right). \end{gathered}$$

For even $p$, the leading term is $𝓞\left(\text{Δ}{t}^{p+2}\right)$ with a slightly different coefficient.

For $\varphi \in 𝓢$ with $p=q$, the exact leading order error in term in (2.8) is

$$\frac{2^p{B}_{p+1}}{p+1}(f(b)-f(a))\text{Δ}{t}^{p+1},$$ (2.9)

which is negligible for a wide range of reasonable $p$ and $\text{Δ}t$ values. The Bernoulli numbers eventually start growing like $p^p$, but for smaller values of $p$ they are moderate. For instance, with $\text{Δ}t=0.1$ and $f(b)-f(a)=1$, this error term is $o(1)$ up until $p=85$, where it takes the value 0.495352, while for $\text{Δ}t=0.01$, the error is below machine precision for all $p$ between 7 and 819. For these reasons, in what follows we choose test functions ${\left({\varphi }_k\right)}_{k\in [K]}\subset 𝓢$ and discretize all integrals using the trapezoidal rule. Unless otherwise stated, each function ${\varphi }_k$ satisfies $p=q$ and so is fully determined by the tuple $\left\{p_k,a_k,b_k\right\}$ indicating its polynomial degree and support. In the next two subsections we propose two different strategies for determining ${\varphi }_k$ using the data $\mathbf{y}$.

2.4.1. Strategy 1: Uniform grid.

The simplest strategy for choosing a basis of test functions ${\left({\varphi }_k\right)}_{k\in [K]}\subset 𝓢$ is to place ${\varphi }_k$ uniformly on the interval $[0,T]$ with fixed degree $p$ and fixed support size

$$L:=\text{\#}\left\{\mathbf{t}\cap \text{supp}\left({\varphi }_k\right)\right\}$$

(i.e., $L$ is the number of timepoints in $\mathbf{t}$ that ${\varphi }_k$ is supported on). The triple $(L,p,K)$ then defines the scheme, where each piece effects the distribution of the residual $𝓡(\mathbf{w})$.

Step 1: Choosing L.

Heuristically, the support size of ${\varphi }_k$ relates to the Fourier transform of the data. If $\text{supp}\left({\varphi }_k\right)$ is small compared to the dominant wavemodes in the dynamics, then high-frequency noise will dominate the values of the inner products $\langle {\varphi }_k^{\prime },\mathbf{y}\rangle$. If $\text{supp}\left({\varphi }_k\right)$ is much larger than the dominant wavemodes, then too much averaging may occur, leading to unresolved dynamics. A natural choice is then to set $L$ equal to the period of a known active wavemode^{1 $k$}:

$$L=\lfloor \frac{1}{\text{Δ}t}\frac{2\pi }{(2\pi T/k)}\rfloor =\lfloor \frac{M}{k}\rfloor .$$

In the noise-free and small-noise experiments below we set $L=\lfloor \frac{M}{25}\rfloor$ and leave optimal selection of $L$ based on Fourier analysis to future work.

Step 2: Determining $p$.

In light of the derivation above of the approximate covariance matrix $\text{Σ}={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T$, we define the parameter $\rho :={\Vert {\varphi }_k^{\prime }\Vert }_2/{\Vert {\varphi }_k\Vert }_2$, which serves as an estimate for the ratio $\sqrt{\mathbb{V}\left[R_3\right]/\mathbb{V}\left[R_2\right]}$ between the standard deviations of the two dominant error terms $R_3$ and $R_2$ in the residual $𝓡\left({\mathbf{\text{w}}}^{\star }\right)$. Larger $\rho$ indicates better agreement with the approximate covariance matrix $\text{Σ}$, since $\text{Σ}\propto \text{Cov}\left(R_3\right)$. Furthermore, for ${\varphi }_k\in 𝓢$ we have the exact formula

$${\rho }^2=\frac{8p^2}{{(b-a)}^2}\left(\frac{\text{Γ}(2p-1)\text{Γ}\left(2p+\frac{1}{2}\right)}{\text{Γ}(2p+1)\text{Γ}\left(2p+\frac{3}{2}\right)}\right)=\frac{p}{{(b-a)}^2}\left(\frac{4p+1}{p-\frac{1}{2}}\right),$$

where $\text{Γ}(z)={\int }_0^{\infty }{t}^{z-1}{e}^{-t}dt$ is the gamma function. Given ${\rho }^2\ge (5+2\sqrt{6})/{(b-a)}^2$, a polynomial degree $p$ may be selected from $\rho$ using the formula

$$\lfloor p=\frac{1}{8}\left(\left({(b-a)}^2{\rho }^2-1\right)+\sqrt{{\left({(b-a)}^2{\rho }^2-1\right)}^2-8{(b-a)}^2{\rho }^2}\right)\rfloor .$$

Step 3: Determining $K$.

Next we introduce the shift parameter $s\in [0,1]$ defined by

$$s:={\varphi }_k\left(t^{\ast }\right)\text{s.t.}{\varphi }_k\left(t^{\ast }\right)={\varphi }_{k+1}\left(t^{\ast }\right),$$

which determines $K$ from $p$ and $L$. In words, $s$ is the height of intersection between ${\varphi }_k$ and ${\varphi }_{k+1}$ and measures the amount of overlap between successive test functions. More overlap increases the correlation between rows in the residual $𝓡(\mathbf{w})$ and hence leads to larger off-diagonal elements in the covariance matrix $\text{Σ}$. Larger $s$ implies that neighboring functions overlap on more points, with $s=1$ indicating that ${\varphi }_k={\varphi }_{k+1}$. Specifically, neighboring test functions overlap on $\lfloor L\left(1-\sqrt{1-s^{1/p}}\right)\rfloor$ timepoints. In Figures 3.2 and 3.3 we vary the parameters $\rho$ and $s$ and observe that results agree with intuition: larger $\rho$ (better agreement with $\text{Σ}$) and larger $s$ (more test functions) lead to better recovery of ${\mathbf{w}}^{\star }$. We summarize the uniform grid algorithm below.

FIG. 3.2.

Small-noise regime: dynamic recovery of the Duffing equation with $\beta =1$. Top: heat map of the ${\log }_{10}$ average error $E_2(\widehat{\mathbf{\text{w}}})$ (left) and sample standard deviation of $E_2(\widehat{\mathbf{\text{w}}})$ (right) over 200 instantiations of noise with ${\sigma }_{NR}=0.04$ (4% noise) vs. $\rho$ and $s$. Bottom:$E_2(\widehat{\mathbf{\text{w}}})$. $\rho$ for fixed $s=0.5$ and various ${\sigma }_{NR}$. For $\rho >3$ the average error is roughly an order of magnitude below ${\sigma }_{NR}$.

FIG. 3.3.

Small-noise regime: dynamic recovery of the van der Pol oscillator with $\beta =4$. Top: heat map of the ${\log }_{10}$ average error $E_2(\widehat{\mathbf{\text{w}}})$ (left) and sample standard deviation of $E_2(\widehat{\mathbf{\text{w}}})$ (right) over 200 instantiations of noise with ${\sigma }_{NR}=0.04$ (4% noise) vs. $\rho$ and $s$. Bottom: $E_2(\widehat{\mathbf{\text{w}}})$ vs. $\rho$ for fixed $s=0.5$ and various ${\sigma }_{NR}$. Similar to the Duffing equation, average error falls to roughly an order of magnitude below ${\sigma }_{NR}$, although for van der Pol this regime is reached when $\rho \approx 6$.

$\widehat{\mathbf{w}}=\mathbf{WSINDy}\_\mathbf{\text{UG}}\left(\mathbf{y},\mathbf{t};{\left(f_j\right)}_{j\in [J]},L,\rho ,s,\lambda ,\gamma \right)$:

1. Construct matrix of trial gridfunctions $\text{Θ}(\mathbf{y})=\left[f_1(\mathbf{y})|\dots |f_J(\mathbf{y})\right]$.

2. Construct integration matrices $\mathbf{\text{V}},{\mathbf{\text{V}}}^{\prime }$ such that

   $${\mathbf{V}}_{km}=\text{Δ}t{\varphi }_k\left(t_m\right),{\mathbf{V}}_{km}^{\prime }=\text{Δ}t{\varphi }_k^{\prime }\left(t_m\right)$$

   with the test functions ${\left({\varphi }_k\right)}_{k\in [K]}$ determined by $L,\rho ,s$ as described above.

3. Compute Gram matrix $\mathbf{G}=\mathbf{V}\text{Θ}(\mathbf{y})$ and right-hand side $\mathbf{b}=-{\mathbf{V}}^{\prime }\mathbf{y}$ so that ${\mathbf{G}}_{kj}=\langle {\varphi }_k,f_j(\mathbf{y})\rangle$ and ${\mathbf{b}}_{kd}=-\langle {\varphi }_k^{\prime },{\mathbf{y}}_d\rangle$.

4. Compute approximate covariance and Cholesky factorization $\text{Σ}={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T=\mathbf{C}{\mathbf{C}}^T$.

5. Solve the generalized least squares problem with ${\ell }_2$-regularization

   $$\widehat{\mathbf{w}}={\text{argmin}}_{\mathbf{w}}\left\{{(\mathbf{G}\mathbf{w}-\mathbf{b})}^T{\text{Σ}}^{-1}(\mathbf{G}\mathbf{w}-\mathbf{b})+{\gamma }^2\parallel \mathbf{w}{\parallel }_2^2\right\},$$

   using sequential thresholding with parameter $\lambda$ to enforce sparsity.

2.4.2. Strategy 2: Adaptive grid.

Motivated by the arguments above, we now introduce an algorithm for constructing a test function basis localized near points of large change in the dynamics. This occurs in three steps: (1) construct a weak approximation to the derivative of the dynamics $\mathbf{v}\approx \dot{\mathbf{x}}$, (2) sample $K$ points $\mathbf{c}$ from a cumulative distribution $\psi$ with density proportional to the total variation $|\mathbf{v}|$, and (3) construct test functions centered at $\mathbf{c}$ using a width-at-half-max parameter $r_{whm}$ to determine the parameters $\left(p_k,a_k,b_k\right)$ of each function ${\varphi }_k$. Each of these steps is numerically stable and carried out independently along each coordinate of the dynamics. A visual diagram is provided in Figure 2.1.

FIG. 2.1.

Adaptive grid construction used on data from the Duffing equation with 10% noise (${\sigma }_{NR}=0.1$). As desired, the centers $\mathbf{c}$ are clustered near steep gradients in the dynamics despite large measurement noise. ($\mathit{\text{Note}}-\varphi (t{)}^{\prime }/10$ is plotted in the upper-left instead of $\varphi (t{)}^{\prime }$ in order to visualize both $\varphi$ and ${\varphi }^{\prime }$.)

Step 1: Weak derivative approximation.

Define $\mathbf{v}:=-{\mathbf{V}}_w^{\prime }\mathbf{y}$, where the matrix $-{\mathbf{V}}_w^{\prime }$ enacts a linear convolution with the derivative of a chosen test function $\varphi \in 𝓢$ of degree $p_w$ and support size $L_w$ so that

$${\mathbf{v}}_m=-\langle {\varphi }^{\prime },\mathbf{y}\rangle =\langle \varphi ,\dot{\mathbf{y}}\rangle \approx {\dot{\mathbf{y}}}_m.$$

The parameters $L_w$ and $p_w$ are chosen by the user, with $L_w=5$ and $p_w\ge 2$ corresponding to taking a centered finite difference derivative with a 3-point stencil. Smaller $p_w$ results in more smoothing and minimizes the corruption from noise while still accurately locating steep gradients in the dynamics. For the examples below we arbitrarily² use $p_w=2$ and $L_w=17$.

Step 2: Selecting $c$.

Having computed $v$, define $\psi$ to be the cumulative sum of $|\mathbf{v}|$ normalized so that max $\psi =1$. In this way $\psi$ is a valid cumulative distribution function with density proportional to the total variation of $\mathbf{y}$. We then find $\mathbf{c}$ by sampling from $\psi$. Let $U=\left[0,\frac{1}{K},\frac{2}{K},\dots ,\frac{K-1}{K}\right]$ with $K$ being the number of the test functions; we then define $\mathbf{c}={\psi }^{-1}(U)$, or numerically,

$$c_k=\min \left\{t\in \mathbf{t}:\psi (t)\ge U_k\right\}.$$

This stage requires the user to select the number of test functions $K$.

Step 3: Construction of test functions ${\left({\varphi }_k\right)}_{k\in [K]}$.

Having chosen the location $c_k$ of the centerpoint for each test function ${\varphi }_k$, we are left to choose the degree $p_k$ of the polynomial and the supports $\left[a_k,b_k\right]$. The degree is chosen according to the width-at-half-max parameter $r_{whm}$, which specifies the difference in timepoints between each center $c_k$ and ${\arg }_t\left\{{\varphi }_k(t)=1/2\right\}$, while the supports are chosen such that ${\varphi }_k\left(b_k-\text{Δ}t\right)={10}^{-16}$. This gives us a nonlinear system of two equations in two unknowns which can be easily solved (i.e., using fzero in MATLAB). This can be done for one reference test functions and the rest of the weights obtained by translation. The optimal value of $r_{whm}$ depends on the timescales of the dynamics and can be chosen from the data using the Fourier transform as in the uniform grid case; however, for simplicity we set $r_{whm}=\lfloor M/100\rfloor$ in the large-noise examples below.

The adaptive grid WSINDy algorithm is summarized as follows: $\widehat{\mathbf{w}}=\mathbf{WSINDy}\_\mathbf{\text{AG}}\left(\mathbf{y},\mathbf{t};{\left(f_j\right)}_{j\in [J]},p_w,L_w,K,r_{whm},\lambda ,\gamma \right)$:

1. Construct matrix of trial gridfunctions $\text{Θ}(\mathbf{y})=\left[f_1(\mathbf{y})|\dots |f_J(\mathbf{y})\right]$.

2. Construct integration matrices $\mathbf{\text{V}},{\mathbf{\text{V}}}^{\prime }$ such that

   $${\mathbf{V}}_{km}=\text{Δ}t{\varphi }_k\left(t_m\right),{\mathbf{V}}_{km}^{\prime }=\text{Δ}t{\varphi }_k^{\prime }\left(t_m\right),$$

   with test functions ${\left({\varphi }_k\right)}_{k\in [K]}$ determined by $p_w,L_w,K,r_{whm}$ as described above.

3. Compute Gram matrix $\mathbf{G}=\mathbf{V}\text{Θ}(\mathbf{y})$ and right-hand side $\mathbf{b}=-{\mathbf{V}}^{\prime }\mathbf{y}$ so that ${\mathbf{G}}_{kj}=\langle {\varphi }_k,f_j(\mathbf{y})\rangle$ and ${\mathbf{b}}_{kd}=-\langle {\varphi }_k^{\prime },{\mathbf{y}}_d\rangle$.

4. Compute approximate covariance and Cholesky factorization $\text{Σ}={\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T=\mathbf{C}{\mathbf{C}}^T$

5. Solve the generalized least squares problem with ${\ell }_2$-regularization

   $$\widehat{\mathbf{w}}={\text{argmin}}_{\mathbf{w}}\left\{{(\mathbf{G}\mathbf{w}-\mathbf{b})}^T{\text{Σ}}^{-1}(\mathbf{G}\mathbf{w}-\mathbf{b})+{\gamma }^2\parallel \mathbf{w}{\parallel }_2^2\right\},$$

   using sequential thresholding with parameter $\lambda$ to enforce sparsity.

3. Numerical experiments.

We now show that WSINDy is capable of recovering the correct dynamics to high accuracy over a range of noise levels. We examine the systems in Table 1 which exhibit several canonical dynamics, namely growth and decay, nonlinear oscillations and chaotic dynamics, in dimensions $D\in \{2,3,5\}$. To generate true trajectory data we use ode45 in MATLAB with absolute and relative tolerance 10⁻¹⁰ and collect $M$ samples uniformly³ in time with sampling rate $\text{Δ}t$. The parameters $M$ and $\text{Δ}t$ are chosen to provide a balance between illustrating ODE behaviors and avoiding an overabundance of observations. Gaussian white noise with mean zero and variance ${\sigma }^2$ is added to the exact trajectories, where $\sigma$ is computed by specifying a noise ratio ${\sigma }_{NR}$ and setting

$$\sigma ={\sigma }_{NR}\frac{\parallel \mathbf{x}{\parallel }_F}{\sqrt{MD}},$$ (3.1)

where the Frobenius norm of a matrix $\mathbf{x}\in {\mathrm{ℝ}}^{M\times D}$ is defined by

$$\parallel \mathbf{x}{\parallel }_F:=\sqrt{\sum _{m=1}^M\sum _{d=1}^D{\left|{\mathbf{x}}_{md}\right|}^2}.$$

The ratio of noise to signal is then approximately equal to the square root of the variance: $\parallel ϵ{\parallel }_F/\parallel \mathbf{x}{\parallel }_F\approx \sigma$.

TABLE 1.

ODEs used in numerical experiments. For Linear 5D, Duffing, van der Pol, and Lotka–Volterra we measure the accuracy in the recovered system as the parameter $\beta$ varies (see Table 2).

Name	Governing equations	$M$	$\text{Δ}t$
Linear 5D	$\{\begin{array}{l}{\dot{x}}_1=-x_5+\beta x_1+x_2,\hfill \\ {\dot{x}}_i=-x_{i-1}+\beta x_i+x_{i+1},i=2,3,4\hfill \\ {\dot{x}}_5=-x_4+\beta x_5+x_1\hfill \end{array}$	1401	0.025
Duffing	$\{\begin{array}{l}{\dot{x}}_1=x_2,\hfill \\ {\dot{x}}_2=-0.2x_2-0.2x_1-\beta x_1^3\hfill \end{array}$	3001	0.01
Van der Pol	$\{\begin{array}{l}{\dot{x}}_1=x_2,\hfill \\ {\dot{x}}_2=\beta x_2\left(1-x_1^2\right)-x_1\hfill \end{array}$	3001	0.01
Lotka–Volterra	$\{\begin{array}{l}{\dot{x}}_1=3x_1-\beta x_1{x}_2,\hfill \\ {\dot{x}}_2=\beta x_1{x}_2-6x_2\hfill \end{array}$	1001	0.01
Nonlinear pendulum	$\{\begin{array}{l}{\dot{x}}_1=x_2,\hfill \\ {\dot{x}}_2=-\sin \left(x_1\right)\hfill \end{array}$	501	0.1
Lorenz	$\{\begin{array}{l}{\dot{x}}_1=10\left(x_2-x_1\right),\hfill \\ {\dot{x}}_2=x_1\left(28-x_3\right)-x_2,\hfill \\ {\dot{x}}_3=x_1{x}_2-\frac{8}{3}{x}_3\hfill \end{array}$	10001	0.001

We measure the accuracy in the recovered dynamical system using the relative $\parallel \cdot {\parallel }_F$ error in the recovered coefficients,

$$E_2(\widehat{\mathbf{w}})=\frac{{\Vert \widehat{\mathbf{w}}-{\mathbf{w}}^{\star }\Vert }_F}{{\Vert {\mathbf{w}}^{\star }\Vert }_F},$$ (3.2)

and the relative $\parallel \cdot {\parallel }_F$ error between the noise-free data $\mathbf{x}$ and the data-driven dynamics ${\mathbf{x}}_{dd}$ along the same timepoints:

$${𝓔}_2\left({\mathbf{x}}_{dd}\right)=\frac{{\Vert {\mathbf{x}}_{dd}-\mathbf{x}\Vert }_F}{\parallel \mathbf{x}{\parallel }_F}.$$ (3.3)

The collection of ODEs in Table 1 are all first-order autonomous systems; however, they exhibit a diverse range of dynamics. The Linear 5D system (for $\beta <0$) and Duffing’s equation are both examples of damped oscillators, showing that WSINDy is able to discern whether such motion is governed by linear or nonlinear coupling between variables. For $\beta >0$, the Linear 5D system exhibits exponential growth. The van der Pol oscillator, Lotka–Volterra system, and nonlinear pendulum demonstrate that a stable limit cycle with abrupt changes may manifest from vastly different nonlinear mechanisms, which turn out to be identifiable using the weak form. Finally, the Lorenz system exhibits deterministic chaos, and hence the dynamics cover a wide range of Fourier modes, which easily become corrupted with noise.

3.1. Noise-free data.

The goal of the following noise-free experiments is to demonstrate convergence of the recovered weights $\widehat{\mathbf{w}}$ to the true weights ${\mathbf{w}}^{\star }$ to within the accuracy tolerance of the ODE solver (fixed 10⁻¹⁰ throughout). In light of Lemma 2, this should occur as the decay rate of the test functions ${\left({\varphi }_k\right)}_{k\in [K]}$ is increased, which for test functions in class $𝓢$ (see (2.6)) is realized by increasing the polynomial degree $p$. Hence, over the range of parameter values in Table 2, for each system we test convergence as $p$ increases. We use the uniform grid approach with shift parameter $s$ chosen such that the number of test functions equals to the number of trial functions $(K=J)$, resulting in square Gram matrices $\mathbf{G}=\mathbf{V}\text{Θ}(\mathbf{y})$. The support of the basis functions along the timegrid $\mathbf{t}$ is set to $L=\lfloor \frac{M}{25}\rfloor$ points. The data-driven trial basis ${\left(f_j\right)}_{j\in [J]}$ includes all monomials in the state variables up to degree 5 as well as the trigonometric terms $\cos \left(n{\mathbf{y}}_d\right)$, $\sin \left(n{\mathbf{y}}_d\right)$ for $n=1,2$ and $d\in [D]$. We set the regularization parameter to zero $(\gamma =0)$, with the exception of the nonlinear pendulum, where $\gamma ={10}^{-8}$, and the sparsity threshold to $\lambda =0.001$. We note that a nonzero $\gamma$ is always necessary to discover the nonlinear pendulum from combined trigonometric and polynomial libraries since $\sin \left(x_1\right)$ is well-approximated by polynomial terms; however, the same is not true for low-order polynomial systems. In cases considered here, sequential thresholding successfully removes trigonometric library terms for ODE systems with polynomial dynamics despite initially ill-conditioned Gram matrices $G$ resulting from combining polynomial and trigonometric terms.

TABLE 2.

Specifications for parameters used in illustrating simulations in Figure 3.1.

ODE	$\beta$	$\mathbf{x}(0)$	$L$	$\text{Δ}L$	$J(=K)$
Linear 5D	(−0.3, −0.2, −0.1,0.1)	${(10,0,0,0,0)}^T$	57	5	252
Duffing	(0.01,0.1,1,10)	${(0,2)}^T$	121	99	29
Van der Pol	(0.01,0.1,1,10)	${(0,1)}^T$	121	99	29
Lotka–Volterra	(0.005,0.01,0.1,1)	${(1,1)}^T$	41	33	29
Pendulum	—	$\begin{array}{c}{x}_2(0)=0,\\ x_1(0)\in \left\{\frac{15}{16}\pi ,\frac{10}{16}\pi ,\frac{5}{16}\pi ,\frac{1}{16}\pi \right\}\end{array}$	21	16	29
Lorenz	—	$\sim U_{{[-15,15]}^2\times [10,40]}$	401	141	68

Figure 3.1 shows that in the limit of large $p$, WSINDy recovers the correct weight matrix ${\mathbf{w}}^{\star }$ of each system in Table 1 to an accuracy of $𝓞\left({10}^{-10}\right)$. For the Linear 5D system, we vary the growth/decay parameter, showing that the system is identifiable to high accuracy despite an excessively large trial library (252 terms). For Duffing’s equation and the van der Pol oscillator, the same convergence trend is observed for $\beta$ values spanning several orders of magnitude. Accuracy is slightly worse for the Lotka–Volterra equation when $\beta =0.005$, which corresponds to highly infrequent predator-prey interactions and leads to solutions with large amplitudes and gradients. For the nonlinear pendulum, we test that WSINDy is able to identify the $\sin \left(x_1\right)$ nonlinearity for both large and small initial amplitudes, noting that $x_1(0)=\frac{15}{16}\pi \approx \pi$ produces strongly nonlinear oscillations, while $x_1(0)=\frac{1}{16}\pi$ produces small-angle oscillations where $\sin \left(x_1\right)\approx x_1$. In addition, for the pendulum we use fewer samples $(M=501)$ and a larger time step $\text{Δ}t=0.1$ and hence observe a decreased convergence rate. For the Lorenz equations we vary the initial conditions, generating 40 random initial conditions from a region covering the strange attractor, and show convergence over all cases.

3.2. Small-noise regime.

We now turn to the case of low to moderate noise levels, examining a noise ratio ${\sigma }_{NR}$ in the range $\left[{10}^{-5},0.04\right]$ for the van der Pol oscillator and Duffing’s equation. We examine $\rho \in [1,7]$ and $s\in [0.3,0.95]$, where $\rho :={\Vert {\varphi }_k^{\prime }\Vert }_2/{\Vert {\varphi }_k\Vert }_2$ and $s$ is the height of intersection of two neighboring test functions ${\varphi }_k$ and ${\varphi }_{k+1}$ (with $s=1$ leading to ${\varphi }_k={\varphi }_{k+1}$ and $s=0$ indicating $\text{supp}\left({\varphi }_k\right)\cap \text{supp}\left({\varphi }_{k+1}\right)=\varphi$. Using the analysis from section 2.3, increasing $\rho$ affects the distribution of the residual $𝓡(\mathbf{w})$ by magnifying the portion $R_3=\langle {\varphi }_k^{\prime },ϵ\rangle$ that is linear in the noise. For $\varphi \in 𝓢$, larger $\rho$ corresponds to a higher polynomial degree $p$, with $\rho \in [1,7]$ leading to $p\in [2,98]$. Larger shift parameter $s$ corresponds to more test functions (higher $K$) but also to higher correlation between rows in $G$, as $\langle {\varphi }_k,f_j(\mathbf{y})\rangle \approx \langle {\varphi }_{k+1},f_j(\mathbf{y})\rangle$ when the supports of ${\varphi }_k$ and ${\varphi }_{k+1}$ sufficiently overlap. Here $s\in [0.3,0.95]$ corresponds to $K\in [14,451]$. We again use the uniform grid approach with $\gamma =0$ and $\lambda =\frac{1}{4}{\min }_{{\mathbf{w}}_j^{\star }\ne 0}\left|{\mathbf{w}}_j^{\star }\right|$. For each system we generate 200 instantiations of noise and record the coefficient error over the range of $s$ and $\rho$ values.

From Figures 3.2 and 3.3 we observe two properties. Firstly, the coefficient error $E_2(\widehat{\mathbf{\text{w}}})$ monotonically deceases with increasing $s$ and $\rho$; hence accurate recovery re quires sufficient overlap between test functions (large enough shift parameter $s$) and sufficiently localized test functions that amplify the portion of the residual that is linear in the noise. Secondly, for large enough $\rho$ and $s$, the error in the coefficients scales linearly with ${\sigma }_{NR}$, leading to an accuracy of $E_2(\widehat{\mathbf{\text{w}}})\approx 0.1{\sigma }_{NR}$, or $-{\log }_{10}\left(0.1{\sigma }_{NR}\right)$ significant digits in the recovered coefficients. In Appendix A we show that this second property does not hold for standard SINDy; in particular, the method of differentiation must change depending on the noise level in order to reach a desired accuracy.

3.3. Large-noise regime.

Figures 3.4 to 3.9 show that adaptive placement of test functions (Strategy 2) can be employed to discover dynamics in the large-noise regime with fewer test functions. We test that each system in Table 1 can be discovered under ${\sigma }_{NR}=0.1$ (10% noise) from only 250 test functions distributed near steep gradients in $\mathbf{\text{y}}$, which are located using the scheme in section 2.4.2 with $p_w=2$ and $L_w=17$. We set the width-at-half-max of the test functions to $r_{whm}=\lfloor M/100\rfloor$ timepoints. To exemplify the separation of scales and the severity of the corruption from noise, the noisy data $\mathbf{\text{y}}$, true data $\mathbf{\text{x}}$, and trajectories ${\mathbf{x}}_{dd}$ from the learned dynamical systems are shown in dynamo view and in phase space (for $D\le 3$). We extend ${\mathbf{x}}_{dd}$ by 50% to show that the data-driven system captures the true limiting behavior. We set the sparsity to $\lambda =\frac{1}{4}{\min }_{{\mathbf{w}}^{\star }\ne 0}\left|{\mathbf{w}}^{\star }\right|$ and $\gamma =0$ except in the Linear 5D and nonlinear pendulum examples, where $\gamma =\sqrt{{\sigma }_{NR}}\approx 0.32$. For the trial basis we use all monomials up to degree 5 in the state variables, and for the pendulum we include the trigonometric terms $\sin \left(k{\mathbf{y}}_d\right),\cos \left(k{\mathbf{y}}_d\right)$ for $k=1,2$ and $d=1,2$.

FIG. 3.4.

Large-noise regime: Linear 5D system with damping $\beta =-0.2$. All correct terms were identified with an error in the weights of $E_2(\widehat{\mathbf{w}})=0.0064$ and a trajectory error of ${𝓔}_2(\widehat{\mathbf{w}})=0.013$.

FIG. 3.9.

Large-noise regime: Lorenz system with ${\mathbf{x}}_0={(-8,7,27)}^T$. All correct terms were identified with an error in the weights of $E_2(\widehat{\mathbf{w}})=0.0084$ and trajectory error $𝓔(\widehat{\mathbf{w}})=0.56$. The large trajectory error is expected due to the chaotic nature of the solution. Using data up until $t=1.5$ (first 1500 timepoints) the trajectory error is 0.027.

In each case the correct terms are identified with coefficient error $E_2(\widehat{\mathbf{w}})<{10}^{-2}$, in agreement with the trend $E_2(\widehat{\mathbf{w}})\approx 0.1{\sigma }_{NR}$ observed in the small-noise regime. For the Linear 5D, Duffing, and Lotka–Volterra systems (Figures 3.4, 3.5, and 3.7) the data-driven trajectory ${\mathbf{x}}_{dd}$ is indistinguishable from the true data to the eye, with trajectory error ${𝓔}_2(\widehat{\mathbf{\text{w}}})<0.02$. For the van der Pol oscillator and nonlinear pendulum (Figures 3.6 and 3.8), ${\mathbf{x}}_{dd}$ follows a limit cycle with an attractor that is indistiguishable from the true data (see phase plane plots); however, an error in the period of oscillation of roughly 0.6% leads to a larger trajectory error. The data-driven trajectory for the Lorenz equation diverges from the true trajectory around $t=2.5$ (Figure 3.9), which is expected from chaotic dynamics, but still remains close to the Lorenz attractor.

FIG. 3.5.

Large-noise regime: Duffing equation, $\beta =1$. All correct terms were identified with an error in the weights of $E_2(\widehat{\mathbf{w}})=0.0075$ and a trajectory error of ${𝓔}_2(\widehat{\mathbf{w}})=0.014$.

FIG. 3.7.

Large-noise regime: Lotka–Volterra system with $\beta =1$. All correct nonzero terms were identified with an error in the weights of $E_2(\widehat{\mathbf{w}})=0.0013$ and trajectory error ${𝓔}_2(\widehat{\mathbf{w}})=0.0082$.

FIG. 3.6.

Large-noise regime: van der Pol oscillator, $\beta =4$. All correct terms were identified with coefficient error $E_2(\widehat{\mathbf{w}})=0.0073$ and trajectory error ${𝓔}_2(\widehat{\mathbf{w}})=0.32$. The data-driven trajectory ${\mathbf{x}}_{dd}$ has a slightly shorter oscillation period of 10.14 time units compared to the true 10.2, resulting in an eventual offset from the true data $\mathbf{x}$ and hence a larger trajectory error. Measured over the time interval [0, 8] the trajectory error is 0.065.

FIG. 3.8.

Large-noise regime: nonlinear pendulum with initial conditions $\mathbf{x}(0)={(15\pi /16,0)}^T$. All correct nonzero terms were identified with an error in the weights of $E_2(\widehat{\mathbf{w}})=0.0089$ and an error between ${𝓔}_2(\widehat{\mathbf{w}})=0.076$.

4. Concluding remarks.

We have developed and investigated a data-driven model selection algorithm based on the weak formulation of differential equations. The algorithm utilizes the reformulation of the model selection problem as a sparse regression problem for the weights ${\mathbf{w}}^{\star }$ of a candidate function basis ${\left(f_j\right)}_{j\in [J]}$ introduced in [21] and generalized in [4] as the SINDy algorithm. Our WSINDy algorithm can be seen as a generalization of the sparse recovery scheme using integral terms found in [17], where dynamics were recovered from noisy data using the integral equation. We have shown that by extending the integral equation to the weak form and using test functions with certain localization and smoothness properties, one may discovery the dynamics over a wide range of noise levels, with accuracy scaling favorably with noise: $E_2(\widehat{\mathbf{w}})\approx 0.1{\sigma }_{NR}$.

A natural line of inquiry is to consider how WSINDy compares with conventional SINDy. There are several notable advantages of WSINDy; in particular, by considering the weak form of the equations, WSINDy completely avoids approximation of pointwise derivatives which significantly reduce the accuracy in conventional SINDy. When using SINDy, one must choose an appropriate numerical differentiation scheme depending on the noise level (e.g., finite differences are not robust to large noise but work well for small noise). For WSINDy, test functions from the space $𝓢$ (see section 2.4) together with the trapezoidal rule are effective in both low-noise and high-noise regimes. We demonstrate these observations in Appendix A by comparing WSINDy to SINDy under several numerical differentiation schemes. On the other hand, it may be the case that less data is required by standard SINDy. For the examples shown here, WSINDy works optimally for test functions supported on at least 15 timepoints, while many derivative approximations require fewer consecutive points.

WSINDy also utilizes the linearity of inner products with test functions to estimate the covariance structure of the residual, performing model selection in a generalized least squares framework. This is a much more appropriate setting given that residuals are neither independent nor uniformly distributed; however, we note that our implementations in this article employ approximate covariance matrices and could benefit from further refinements and investigation. In Appendix B we show that using generalized least squares with approximate covariance improves some results over ordinary least squares, but not significantly. We leave incorporation of more detailed knowledge of the covariance structure to future work. In addition, generalized least squares could potentially improve traditional model selection algorithms that rely on pointwise derivative estimates by similarly exploiting linear operators. Ultimately, a thorough analysis of the advantages of generalized least squares for model selection deserves further study.

Lastly, the most obvious extensions lie in generalizing the WSINDy method to spatiotemporal datasets. WSINDy as presented here in the context of ODEs is an exciting proof of concept with natural extensions to spatiotemporal and multiresolution settings building upon the extensive results in numerical and functional analysis for weak and variational formulations of physical problems.

Appendix A. Comparison between WSINDy and SINDy.

Here we compare WSINDy and SINDy using the van der Pol oscillator, Lotka–Volterra system, and Lorenz equation. For WSINDy we place test functions along the time axis according to the uniform grid strategy. For SINDy, we examine three differentiation methods: total variation regularized derivatives (SINDy-TV), centered second-order finite difference (SINDy-FD-2), and centered fourth-order finite difference (SINDy-FD-4). For SINDy-TV we use default settings and set the regularization parameter equal to the time step.

For each system and noise level we generate 200 independent instantiations of noise and record the average coefficient error $E_2(\widehat{\mathbf{\text{w}}})$ (3.2) as well as the average true positivity ratio (TPR) [10]:

$$\text{TPR}(\widehat{\mathbf{w}})=\frac{\text{TP}(\widehat{\mathbf{w}})}{\text{TP}(\widehat{\mathbf{w}})+\text{FP}(\widehat{\mathbf{w}})+\text{FN}(\widehat{\mathbf{w}})},$$ (A.1)

where $\text{TP}(\widehat{\mathbf{w}})$ is the number of correctly identified nonzero terms, $\text{FP}(\widehat{\mathbf{\text{w}}})$ is the number of falsely identified nonzero terms, and $\text{FN}(\widehat{\mathbf{w}})$ is the number of terms that are falsely identified as having a coefficient of zero. Since the feasible range of sparsity thresholds $\lambda$ depends on the noise level, we adopt the selection methodology in [14] to choose an appropriate $\lambda$ value for each instantiation of noise: $\lambda$ is chosen from the set ${10}^{\left\{-5+\frac{i}{10},i\in \{0,\dots ,50\}\right\}}$ (i.e., the 51 values from 10⁻⁵ to 1 equally spaced ${\log }_{10}$) as the minimizer of the loss function

$$𝓛(\lambda )=\frac{{\Vert \mathbf{A}{\mathbf{w}}^{\lambda }-\mathbf{A}{\mathbf{w}}^0\Vert }_2}{{\Vert \mathbf{A}{\mathbf{w}}^0\Vert }_2}+\frac{\text{\#}\left\{j:{\mathbf{w}}_j^{\lambda }\ne 0\right\}}{J},$$

where $\mathbf{A}=\mathbf{V}\text{Θ}(\mathbf{y})$ for WSINDy and $\mathbf{A}=\text{Θ}(\mathbf{y})$ for $\text{SINDy;}{\mathbf{w}}^{\lambda }$ is the sequential-thresholding least squares solution for sparsity threshold $\lambda$, and $J$ is the number of terms in the model library (for further details see [14]).

From Figures A.1, A.2, and A.3 we observe that for small noise (up to ${\sigma }_{NR}={10}^{-1}$), the coefficient error for WSINDy follows the linear trend $E_2(\widehat{\mathbf{w}})\approx 0.1{\sigma }_{NR}$ (observed in the text) and that SINDy-FD-4 behaves similarly but with slightly worse accuracy. For larger noise, SINDy diverges in accuracy and identification of the correct nonzero terms for each differentiation scheme, while WSINDy maintains a TPR of at least 0.8 up to 40% noise for each system. WSINDy thus provides an advantage across the entire noise spectrum examined, all while employing the same weak discretization scheme.

FIG. A.1.

Comparison between WSINDy and SINDy: van der Pol. Clockwise from top left: small-noise $TPR(\widehat{\mathbf{w}})$ (defined in (A.1)), large-noise $TPR(\widehat{\mathbf{w}})$, large-noise $E_2(\widehat{\mathbf{\text{w}}})$ (defined (3.2)), small-noise $E_2(\widehat{\mathbf{\text{w}}})$.

FIG. A.2.

Comparison between WSINDy and SINDy: Lotka–Volterra. Clockwise from top left: small-noise $TPR(\widehat{\mathbf{w}})$ (defined in (A.1)), large-noise $TPR(\widehat{\mathbf{w}})$, large-noise $E_2(\widehat{\mathbf{\text{w}}})$ (defined (3.2)), small-noise $E_2(\widehat{\mathbf{\text{w}}})$.

FIG. A.3.

Comparison between WSINDy and SINDy: Lorenz system. Clockwise from top left: small-noise $TPR(\widehat{\mathbf{w}})$ (defined in (A.1)), large-noise $TPR(\widehat{\mathbf{w}})$, large-noise $E_2(\widehat{\mathbf{\text{w}}})$ (defined (3.2)), small-noise $E_2(\widehat{\mathbf{\text{w}}})$.

Appendix B. Generalized least squares vs. ordinary least squares.

FIG. B.1.

Comparison between WSINDy with GLS and WSINDy with ordinary least squares using the Duffing equation. Results are averaged over 200 instantiations of noise.

Generalized least squares (GLS) aims to account for correlations between the residuals [8]. Given a linear model $y=\mathbf{X}\beta +ϵ$, where $\text{Cov}(ϵ)=\text{Σ}$ and $\mathbb{E}[ϵ\mid \mathbf{X}]=0$, the GLS estimator of the parameters $\beta$ upon observing $\widehat{y}$ is

$$\widehat{\beta }={\left({\mathbf{X}}^T{\text{Σ}}^{-1}\mathbf{X}\right)}^{-1}{\mathbf{X}}^T{\text{Σ}}^{-1}\widehat{y}.$$

This provides the best linear unbiased estimator of $\beta$ in the sense that if $\tilde{\beta }$ is any other unbiased estimator, then $\tilde{\beta }$ has lower variance: $\mathbb{V}\left[{\widehat{\beta }}_i\right]\le \mathbb{V}\left[{\tilde{\beta }}_i\right],i=1,\dots ,n$.

Above we derived an approximate covariance matrix $\text{Σ}\approx {\mathbf{V}}^{\prime }{\left({\mathbf{V}}^{\prime }\right)}^T$ to use in the GLS implementation of WSINDy, although the true covariance depends on the underlying unknown dynamical system and hence is unattainable. In addition, since in our case $\mathbf{X}=\mathbf{G}=\mathbf{V}\text{Θ}(\mathbf{y})$ depends on the noise $ϵ$, the assumption $\mathbb{E}[ϵ\mid \mathbf{X}]=0$ is violated. Nevertheless, we find that the noise regime ${\sigma }_{NR}\in [0.01,0.3]$ does benefit from using GLS over ordinary least squares. Figure B.1 shows that for the Duffing equation, GLS extends the region $\left\{{\sigma }_{NR}\mid \text{TPR}(\widehat{\mathbf{w}})>0.95\right\}$ from ${\sigma }_{NR}\le 0.05\text{to}{\sigma }_{NR}\le 0.15$, as well as increases the accuracy in the recovered coefficients. This suggests that further improvements can be made with a more refined covariance matrix.