Learning partial differential equations via data discovery and sparse optimization

Abstract

We investigate the problem of learning an evolution equation directly from some given data. This work develops a learning algorithm to identify the terms in the underlying partial differential equations and to approximate the coefficients of the terms only using data. The algorithm uses sparse optimization in order to perform feature selection and parameter estimation. The features are data driven in the sense that they are constructed using nonlinear algebraic equations on the spatial derivatives of the data. Several numerical experiments show the proposed method's robustness to data noise and size, its ability to capture the true features of the data, and its capability of performing additional analytics. Examples include shock equations, pattern formation, fluid flow and turbulence, and oscillatory convection.

1. Introduction

Differential equations provide fundamental models in the physical and life sciences. Many examples of partial differential equations (PDEs) exist in the physical sciences, for example Maxwell's equations for electromagnetism, Einstein's equation for general relativity, Schrödinger's equations in quantum physics and chemistry, Boltzmann's and Vlasov's equations in plasma physics, and the Navier–Stokes and Euler equations in fluid and gas dynamics. These equations are often considered essential descriptors of important physical phenomena. In the life sciences, PDEs typically describe dynamic processes such as the Keller–Segel model for chemotaxis, predator–prey equations, reaction–diffusion equations, and the Hodgkin–Huxley and FitzHugh–Nagumo models for neural excitement. PDEs also model complex behaviours in the social sciences, for example conservation laws for traffic flow, systems of equations for population dynamics, epidemic models and financial markets. The original discoveries of these equations (typically) required a grasp of mathematics, an understanding of theory, and supportive evidence from experimental data. In this work, we provide a computational approach for learning the underlying PDE that governs a particular phenomenon only using the data.

Machine learning and regression provide numerical tools to explore large datasets and learn the underlying behaviours. These methods can supply researchers with important techniques to support and augment their path to scientific discovery. Two important contributions to the discovery of physical laws from experimental data can be found in [1,2]. The main approach is to use symbolic regression in order to recover the underlying physical equations that govern the structure of the phenomena. The idea in [1,2] is to compare numerical derivatives of the experimental data with analytic derivatives of candidate functions. The regression approach balances between the efficiency of the candidate model and the accuracy of the fit. In a recent study [3], an alternative approach using sparse regression is proposed. Rather than using symbolic regression, a sequential thresholded least-squares algorithm is applied to a set of candidate functions (polynomials) in order to approximate the governing equations for a nonlinear dynamical system. The idea in [3] gives a direct approach to identifying key terms in the dynamical system within a set space of candidate polynomials. In this work, we propose a sparse optimization method for identifying the underlying PDE to a given dataset.

Sparsity plays a key role in optimization and data sciences. The L¹ norm is often used as a proxy for sparsity, because it allows for the construction and implementation of feasible numerical schemes. The least absolute shrinkage and selection operator (LASSO) [4] proposed augmenting the least-squares problem, commonly used in data fitting, with the L¹ norm on the unknown coefficients. The L¹ norm is used to penalize the number of non-zero coefficients in order to promote coefficient sparsity. The L¹ penalty formulation has yielded a rich body of research and applications. One application is in compressive sensing, which (in general) is the recovery of a signal from random measurements and results in an under-determined system of linear equations (see [5,6]). The linear system is typically solved via an L¹ (or related) regularized least-squares minimization and (in some cases) can be shown to exactly recover the sparse solution [7]. Sparsity is also extremely advantageous for learning key features from datasets. One popular example is its use in dictionary learning, where one learns a sparse representation for a given dataset (typically images) while simultaneously learning the features (referred to as the dictionary) themselves (see [8,9]).

In recent years, sparsity and sparse optimization have been applied to problems in differential equations and scientific computing. Some examples include modelling multi-physics and multi-scale phenomena related to nonlinear PDEs [10–13] in the construction of basis functions for quantum mechanical models [14–16] and in reduced-order modelling of dynamic systems [17–20]. The interaction between PDEs and the L¹ terms has been studied in [21–23], providing some properties of solutions.

The paper is outlined as follows. In §2, a motivating example of the proposed method is provided and its construction is detailed. The numerical method and algorithm, including the construction of the features used in the learning step, are provided. In §3, the data simulation is detailed. Several numerical experiments are presented, measuring the robustness of the method to data noise and size, the method's ability to capture the true features of the simulated data, and its capability to perform other important tasks. We end with some concluding remarks in §4.

2. Sparse reconstruction of partial differential equations

Before detailing the method and algorithm, we will derive the proposed method for a specific example.

(a). Motivation and model

Consider the problem of discovering the underlying PDE when data were obtained from the viscous Burgers’ equation:

$$u_t={\left(\frac{u^2}{2}\right)}_x+\nu u_{xx}.$$

Assume that the form of the equation is unknown to the user, and instead we consider the general form for second-order PDEs with quadratic nonlinearity (in space)

$$u_t={\alpha }_1+{\alpha }_{2}u+{\alpha }_3{u}^2+{\alpha }_4{u}_x+{\alpha }_5{u}_x^2+{\alpha }_{6}u{u}_x+{\alpha }_7{u}_{xx}+{\alpha }_8{u}_{xx}^2+{\alpha }_{9}u{u}_{xx}+{\alpha }_{10}{u}_x{u}_{xx}.$$

This assumption can be made by a user based on some physically relevant models related to the data/simulation. The r.h.s. can also be obtained by approximating a general second-order evolution equation u_t=F(u,u_x,u_xx) via a second-order Taylor expansion. The PDE above can be written as the product of (u,u_x,u_xx)-dependent terms with fixed coefficients

$$u_t=[1u{u}^2{u}_x{u}_x^{2}u{u}_x{u}_{xx}{u}_{xx}^{2}u{u}_{xx}{u}_x{u}_{xx}]\cdot \mathit{\alpha }.$$ 2.1

This equation holds at all points (x,t) in the domain. Each of the (u,u_x,u_xx)-dependent terms in equation (2.1) represents a potential feature that describes the intrinsic dynamics of the data. Define the feature vectors f₁(t), f₂(t),…, and f₁₀(t), by

$$f_1(t)=\left[\begin{array}{c}|\\ 1\\ |\end{array}\right],f_2(t)=\left[\begin{array}{c}|\\ u\\ |\end{array}\right],f_3(t)=\left[\begin{array}{c}|\\ u^2\\ |\end{array}\right],\dots ,f_{10}(t)=\left[\begin{array}{c}|\\ u_x\hspace{0.17em}{u}_{xx}\\ |\end{array}\right],$$

where each time-dependent feature vector is the vectorization of the terms in equation (2.1). Specifically, each component of the feature vector f_ℓ(t) is the evaluation of the corresponding term from equation (2.1) at a pre-determined point in space. For example, given n spatial points, the third feature vector is defined as

$$f_3(t)=\left[\begin{array}{c}{u}^2(x_1,t)\\ |\\ u^2(x_j,t)\\ |\\ u^2(x_n,t)\end{array}\right].$$

The collection of feature vectors defines the feature matrix, i.e. F_u(t)=[f_ℓ(t)] or (explicitly)

$$F_u(t)=\left[\begin{array}{cccccccccc}|& |& |& |& |& |& |& |& |& |\\ 1& u& u^2& u_x& u_x^2& u{u}_x& u_{xx}& u_{xx}^2& u{u}_{xx}& u_x{u}_{xx}\\ |& |& |& |& |& |& |& |& |& |\end{array}\right].$$

Let V (t) be the ‘velocity’ vector, i.e.

$$V(t)=\left[\begin{array}{c}|\\ u_t\\ |\end{array}\right].$$

Then for t>0, the following system of equations holds:

$$V(t)=F_u(t)\hspace{0.17em}\mathit{\alpha }.$$ 2.2

The terms V and F are known while the coefficients α are unknown. Note that the problem is linear in the unknown α. As the feature matrix and the velocity vector are data dependent (i.e. generated from a particular initial condition), the inverse problem (equation (2.2)) may not produce a unique solution α (for example, consider the trivial case u(x,t)=0 for all t>0). Also, because the data may contain noise, the inverse problem can be ill-posed. To deal with this ill-posedness, we will use the entire dataset along with a regularizer.

The linear system in equation (2.2) holds for all time, so we have

$$V(t_1)=F_u(t_1)\hspace{0.17em}\mathit{\alpha },\hspace{0.17em}V(t_2)=F_u(t_2)\hspace{0.17em}\mathit{\alpha },...,V(t_M)=F_u(t_M)\hspace{0.17em}\mathit{\alpha }.$$ 2.3

If the number of columns in F_u(t) is less than the number of discrete grid points in x, then both equations (2.2) and (2.3) are over-determined. In the ‘perfect’ case, where u(x,t) is exactly generated by the given PDE, the features are exact. Thus, the inverse problem can be solved directly and it may be possible to recover the correct coefficients (under some mild assumptions). However, if the data are collected from a simulation or experiment, direct solutions (for example, using the pseudo-inverse) yield the incorrect coefficients. Instead, to solve equation (2.3), we use an L¹-regularized least-squares minimization

$$\underset{\mathit{\alpha }}{min}\frac{1}{2}\sum _{k=1}^M\parallel V(t_k)-F_u(t_k)\hspace{0.17em}\mathit{\alpha }{\parallel }_2^2+\lambda \parallel \mathit{\alpha }{\parallel }_{1,}$$ 2.4

where λ>0 is a balancing parameter. The least-squares term in equation (2.4) is taken over several discrete time stamps t_k. Without a regularization term, the noise makes the solution meaningless. The L¹ regularization is chosen to promote sparsity in the vector α, with the modelling assumption that the underlying dynamics are governed by a few terms. A visual guide to this algorithm is provided in figure 1.

Figure 1.

Visual algorithm. The data are given as a sequence at different time stamps (a). Then, the data are converted to the Fourier domain (b) and the spatial derivatives are calculated (c). The derivatives are combined to construct features, which are then vectorized (d). An L¹ optimization algorithm is used to extract the sparsity coefficients α that fit the features to the velocity. The coefficients α are unravelled to reveal the underlying model (e). (Online version in colour.)

(b). Numerical method and algorithm

Given data w(x,t) at (possibly non-uniform) grid points x_j for 1≤j≤N and time stamps t_k for 1≤k≤M, we approximate the underlying evolution equation w_t=G(w) using the idea from §2a. In this section, we will make a distinction between the given (discrete) data w(x,t) and the computed variable u(x,t) that is generated by a particular PDE. The following steps are required to approximate the form of the evolution equation w_t=G(w): first compute a numerical approximation to the spatial derivatives of w(x,t), then construct a collection of feature vectors, and lastly compute the coefficients of the terms using the sparse least-squares method.

First, we set the highest order spatial derivative expected in the model and calculate all derivatives up to that order. To demonstrate the details of this algorithm, we will focus on the two-dimensional case where w has periodic boundary conditions on a square ${[0,L]}^2\subset {\mathbb{R}}^2$ and thus construct derivative approximations using the spectral method [24]. Note that this method can be applied to more general dimensions and that examples in §3 contain both one-dimensional and two-dimensional datasets. Setting the wavenumbers to k_x=−πN/L,…,πN/L and k_y=−πN/L,…,πN/L, the pth spatial derivatives are approximated using the following equations: (D_x)^pw≈ifft((ik_x)^p fft(w)) and (D_y)^pw≈ifft((ik_y)^p fft(w)). Alternative approximations, such as finite differences or finite elements, can be used. If the data w(x,t) are very noisy, then preprocessing and/or regularization is needed to calculate the spatial derivatives (see example 3.9).

The feature vectors f_ℓ(t) can be found by taking polynomials of the derivative terms. The features can be determined by using the terms of a Taylor expansion and/or by collecting common terms from known models with similar behaviour. It is assumed that the number of features (i.e. the size of the feature space) is larger than the number of terms in the underlying dynamics and that the feature space includes the true terms in the underlying equations. Because the feature space is constructed using algebraic equations on the data and its spatial derivatives, the space can be arbitrarily large and redundant. Note that the features themselves are problem dependent—explicit examples are provided §3. Each feature is normalized (by dividing the feature vector by the maximum magnitude of the feature vector over all time stamps), in order to prevent biasing in the optimization. The sparse optimization problem (equation (2.4)) penalizes the magnitude of the coefficients directly, thus the approximation of the coefficients could be biased by the high variance among the magnitudes of the vectors. For example, if one of the feature vectors is arbitrarily small, then its inclusion may not be ‘costly’ with respect to the optimization. Once the feature vectors are computed and normalized, the feature matrix is constructed by appending them together column-wise: F_w(t)=[ f_ℓ(t)]. After the coefficients are computed, we will need to rescale them in order to invert the normalization.

The time derivative of the data w_t is either given or computed numerically using finite differences. Vectorizing w_t forms the ‘velocity’ vector V (t). The values of w_t may be noisy due to the data acquisition process, computational error or approximation error. This method does not require any preprocessing of the velocity vector. The coefficients are found via a sparse least-squares minimization

$$\underset{\mathit{\alpha }}{min}\frac{1}{2}\sum _{k=1}^M\parallel V(t_k)-F_w(t_k)\hspace{0.17em}\mathit{\alpha }{\parallel }_2^2+\lambda \parallel \mathit{\alpha }{\parallel }_1,$$ 2.5

where λ>0 is a balancing parameter. The minimizer satisfies the following inclusion relation [25]:

$$\sum _{k=1}^M{F}_w^T(t_k)(F_w(t_k)\hspace{0.17em}\mathit{\alpha }-V(t_k))+\lambda \mathrm{\partial }\parallel \mathit{\alpha }{\parallel }_1\ni 0,$$

which can be written as

$$(\sum _{k=1}^M{F}_w^T(t_k)F_w(t_k))\mathit{\alpha }+\sum _{k=1}^M{F}_w^T(t_k)V(t_k)+\lambda \mathrm{\partial }\parallel \mathit{\alpha }{\parallel }_1\ni 0.$$ 2.6

The number of equations given in equation (2.6) is equal to the number of unknown coefficients. Therefore, the size of the problem can be relatively small if the number of features is small or arbitrarily large if the feature space contains many terms. There are several ways to solve equation (2.6) in practice (see [26–30]). We use the Douglas–Rachford algorithm [31,32]. Define the two terms

$$H_1(\alpha )=\lambda \parallel \mathit{\alpha }{\parallel }_1,H_2(\alpha )=\frac{1}{2}\sum _{k=1}^M\parallel V(t_k)-F_w(t_k)\hspace{0.17em}\mathit{\alpha }{\parallel }_2^2.$$

The proximal operator for a functional H(x) is defined as

$${\text{prox}}_{\gamma H}(x)=\underset{y}{\mathrm{argmin}}\hspace{0.17em}\gamma H(y)+\frac{1}{2}\parallel x-y{\parallel }^2.$$

The proximal operators for these functions can be written as [33]

$${\text{prox}}_{\gamma H_1}(x)=\underset{y}{\mathrm{argmin}}\hspace{0.17em}\gamma \lambda \parallel \mathit{\text{y}}{\parallel }_1+\frac{1}{2}\parallel x-y{\parallel }^2=\text{max}(|x|-\gamma \lambda ,0)\hspace{0.17em}\text{sign}(x),$$

which is commonly known as the shrink operator, and

$$\begin{array}{rl}{\text{prox}}_{\gamma H_2}(x)& =\underset{y}{\mathrm{argmin}}\hspace{0.17em}\frac{\gamma }{2}\sum _{k=1}^M\parallel V(t_k)-F_w(t_k)y{\parallel }_2^2+\frac{1}{2}\parallel x-y{\parallel }^2\\ & ={(I+\gamma \sum _{k=1}^M{F}_w^T(t_k)\hspace{0.17em}{F}_w(t_k))}^{-1}(x+\gamma \sum _{k=1}^M{F}_w^T(t_k)V(t_k)),\end{array}$$

which is a regularization of the standard normal equation. The Douglas–Rachford iteration is defined by the following two-step method [32]:

$$\begin{array}{rl}{\tilde{\alpha }}^{k+1}& =(1-\frac{\mu }{2}){\tilde{\alpha }}^k+\frac{\mu }{2}\hspace{0.17em}{\text{rprox}}_{\gamma H_2}({\text{rprox}}_{\gamma H_1}({\tilde{\alpha }}^k))\\ \text{and}{\alpha }^{k+1}& ={\text{prox}}_{\gamma H_1}({\tilde{\alpha }}^{k+1})\end{array}\},$$ 2.7

where rprox_γH(x)=2prox_γH(x)−x. Note that the order of the operators matters, and it is typical to use the shrink operator as the second step to maintain sparsity. For γ>0 and μ∈[0,2], the iterates α^k converge to the minimizer of equation (2.5). For more details on the convergence of the Douglas–Rachford algorithm, see [34,35].

The coefficients identified by equation (2.5), denoted α*, will typically have some bias due to the shrinkage function. Define the support set of the minimizer of equation (2.5), i.e. $\mathcal{I}=\text{supp}({\mathit{\alpha }}^{\ast })$. If the restriction of the matrix $\sum _{k=1}^M{F}_w^T(t_k)F_w(t_k)$ onto the support set $\mathcal{I}$ is well conditioned, we can refine the solution α* by a second optimization over all vectors with the same support

$$\underset{\text{supp}({\mathit{\alpha }}^{\ast })\hspace{0.17em}=\hspace{0.17em}\mathcal{I}}{min}\hspace{0.17em}\frac{1}{2}\sum _{k=1}^M\parallel V(t_k)-F_w(t_k)\hspace{0.17em}\mathit{\alpha }{\parallel }_2^2.$$ 2.8

This step removes any bias in the value of the coefficients while retaining the features. Altogether, the proposed algorithm is as follows:

Partial differential equations, sparse learning algorithm

• Step 1: Given w(x,t), calculate all spatial derivatives (up to some order) using the spectral method.

• Step 2: Construct feature vectors via algebraic equations of the spatial derivatives. Normalize feature vectors to have maximum value 1.

• Step 3: Use the Douglas–Rachford algorithm, equation (2.7), to solve the L¹ least-squares problem (equation (2.5)). Extract the support of the vector.

• Step 4: (Optional) If the matrix $\sum _{k=1}^M{F}_w^T(t_k)\hspace{0.17em}{F}_w(t_k)$ restricted to the support set $\mathcal{I}$ is well conditioned, then refine the solution of equation (2.5) by solving equation (2.8).

Note that the normalization in step 2 must be ‘undone’ after step 4 in order to recover the actual coefficient corresponding to a particular feature. Also, to compute non-trivial solutions, the parameter λ should be in the interval $(0,\parallel \sum _{k=1}^M{F}_w^T(t_k)V(t_k){\parallel }_{\mathrm{\infty }})$; see [36] for related work.

3. Simulation and numerical experiments

To test the method, data are simulated using an accurate method on a higher resolution grid. Then, the simulated data u(x,t) are restricted onto a coarse grid and are assigned to the variable w⁰(x,t). The noise-free velocity $w_t^0$ is computed by a first-order backward difference

$$w_t^0(x_j,t_k)\approx \frac{w^0(x_j,t_k)-w^0(x_j,t_{k-1})}{\mathrm{d}t},$$

where dt>0 is the time step. The computed velocity w_t is corrupted by additive Gaussian noise, which models possible corruption in the velocity measurement or its calculation. The noisy velocity is assigned to w_t. The noise level is displayed for all examples in this section by measuring the ratio between the L² norm of the noise and the L² norm of the data, i.e.

$$\text{noise level}=\frac{\parallel w_t-w_t^0{\parallel }_2}{\parallel w_t^0{\parallel }_2}\times 100\mathrm{\%}.$$

All features are computed using w⁰(x_j,t_k); therefore the fitting is done on a backward Euler step at each time stamp. It is worth noting that the approximations used to simulate the data and the approximation used to learn the features are not the same.

Here, we include several examples which represent various phenomena and applications. In all examples, the parameters for the Douglas–Rachford algorithm are set to: $\gamma =\frac{1}{2}$, μ=1 and the maximum number of iterations to 5000. Convergence of the iterates is achieved before the maximum iteration bound is reached.

(a). Examples and applications

Example 3.1 (viscous Burgers' equation) —

The first example is the viscous Burgers' equation

$$u_t={\left(\frac{u^2}{2}\right)}_x+\nu u_{xx},$$

where ν>0 is the viscosity and x∈[0,1]. The data are simulated using the Crank–Nicolson method with 1024 grid points and with $\nu =\frac{1}{10}$. The data are simulated from t∈[0,0.05]. The spectral simulation is computed in conservative form although the feature vectors will not be in conservative form. The balance parameter is fixed to λ=500.

In table 1, the feature space is made up of the terms from a third-order Taylor expansion using up to second-order derivatives. The proposed algorithm is applied to the data with various levels of noise. For all noise levels, 2.0%, 19.5% and 94.5%, the coefficients are identified to within 1% in relative error. As the noise level increases, the approximation to the coefficients stays within two significant digits. This example shows that the method seems robust to noise. Also, as all spatial derivatives are approximated using the spectral method and the features are computed via direct products, each term is not necessarily in the proper form (i.e. the features are non-conservative, non-monotone, etc.). Yet, the method is able to detect the correct features and approximate the true value of the coefficients.

Table 1.

Example 3.1: viscous Burgers’ equation. The feature space is made up of the terms from a third-order Taylor expansion using up to second-order derivatives. The coefficients are identified to within 1% in relative error. As the noise level increases, the approximation to the coefficients stays within two significant digits.

terms	coefficients, noise=2.0%	noise=19.5%	noise=94.5%
1	0	0	0
u	0	0	0
u2	0	0	0
u3	0	0	0
ux	0	0	0
$u_x^2$	0	0	0
$u_x^3$	0	0	0
uux	0.9953	0.9949	0.9948
u2ux	0	0	0
$u{u}_x^2$	0	0	0
uxx	0.0099	0.0099	0.0099
$u_{xx}^2$	0	0	0
$u_{xx}^3$	0	0	0
uuxx	0	0	0
u2uxx	0	0	0
$u{u}_{xx}^2$	0	0	0
uxuxx	0	0	0
$u_x^2{u}_{xx}$	0	0	0
$u_x{u}_{xx}^2$	0	0	0
uuxuxx	0	0	0

In figure 2, the coefficients learned from the data with 94.5% noise (column 3 of table 1) are used to compare the original solution with the learned solution. In figure 2a is the original simulation (black, solid) using 1024 grid points and the learned solution (blue, dotted) using 256 grid points at time T=0.1 (which is well beyond the interval used in the learning algorithm). Figure 2b is the absolute value of the error between the true and learned solutions projected onto the coarse grid. The error between the original simulation and the learned solution is small in norm: the L¹ error is 0.0128, the L² error is 0.0014 and the $L^{\mathrm{\infty }}$ error is 0.0200. The error between the learned solution and the original data incorporates both the error in the learned parameters and the difference between the two resolutions. The projection error (i.e. the difference between the original simulation using 1024 grid points and a simulation using 256 grid points with the exact parameters) in the various norms is: the L¹ error is 0.0114, the L² error is 0.0012, and the $L^{\mathrm{\infty }}$ error is 0.0195. In figure 3, the space–time dynamics are plotted to provide a qualitative comparison between the original simulation (figure 3a) and the learned dynamics (figure 3b). This shows that the parameter estimation leads to a close approximation to the original dynamics in both quantitative and qualitative senses. The main interest here is the identification of the correct terms in the underlying dynamics and the approximation of the coefficients of the equation.

Figure 2.

Example 3.1: viscous Burgers’ equation with noise level =94.5%. In (a) is the original simulation (black, solid) using 1024 grid points and the learned solution (blue, dotted) using 256 grid points at time T=0.1. In (b) is the absolute value of the error between the true and learned solutions projected onto the coarse grid. (Online version in colour.)

Figure 3.

Example 3.1: viscous Burgers’ equation with noise level =94.5%. The space–time dynamics are plotted to provide a qualitative comparison between the original simulation (a) and the learned dynamics (b). This shows that the parameter estimation leads to a close approximation to the original dynamics. The horizontal axis is space and the vertical axis is time. (Online version in colour.)

Example 3.2 (inviscid Burgers' equation) —

To generate the solution to the inviscid Burgers' equation, we simulate the viscous equation with 0<ν≪1 and x∈[0,1]. The data are simulated using the Crank–Nicolson method with 1024 grid points, ν=9.8×10⁻⁴ (near the vanishing limit), from t∈[0,0.05]. The spectral method is applied to the conservative form of the PDE. The balance parameter is fixed to λ=500, as in example 3.1.

In table 2, the feature space is made up of the terms from a third-order Taylor expansion using up to second-order derivatives. The algorithm is applied to the data, with various levels of noise. For all noise levels, 1.5%, 15.0% and 73.7%, the coefficients are identified to within 2% in relative error. As the noise level increases, the identification of the correct features stays stable, while a slight decrease occurs in the approximation of the coefficients. This shows the robustness of the method to noisy data.

Table 2.

Example 3.2: inviscid Burgers' equation. The feature space is made up of the terms from a third-order Taylor expansion using up to second-order derivatives. The coefficients are identified to within 2% in relative error. As the noise level increases, the identification of the correct features stays stable. Note that the terms used in the feature space are not necessarily in the correct form (they are non-conservative, non-monotone, etc.), yet the correct coefficients are detected and resolved.

terms	coefficients, noise=1.5%	noise=15.0%	noise=73.7%
1	0	0	0
u	0	0	0
u2	0	0	0
u3	0	0	0
ux	0	0	0
$u_x^2$	0	0	0
$u_x^3$	0	0	0
uux	0.9989	0.9987	0.9972
u2ux	0	0	0
$u{u}_x^2$	0	0	0
uxx	0.0010	0.0010	0.0010
$u_{xx}^2$	0	0	0
$u_{xx}^3$	0	0	0
uuxx	0	0	0
u2uxx	0	0	0
$u{u}_{xx}^2$	0	0	0
uxuxx	0	0	0
$u_x^2{u}_{xx}$	0	0	0
$u_x{u}_{xx}^2$	0	0	0
uuxuxx	0	0	0

In figure 4, the coefficients learned from the data with 73.7% noise (column 3 of table 2) are used to compare the original solution with the learned solution. Figure 4a is the original simulation (black, solid) using 1024 grid points and the learned solution (blue, dotted) using 256 grid points at time T=0.1. Figure 4b is the absolute value of the error between the two solutions projected onto the coarse grid, which shows that the learned dynamics agree with the original data everywhere except at the shock. In figure 5, the space–time dynamics are plotted to provide a qualitative comparison between the original simulation (figure 5a) and the learned dynamics (figure 5b) over time. The error between the original simulation and the learned solution is small in norm: the L¹ error is 0.0163, the L² error is 0.0193 and the $L^{\mathrm{\infty }}$ error is 0.2705, which also includes the projection errors (difference in grid sizes). The $L^{\mathrm{\infty }}$ error is larger in example 3.2 than in example 3.1 because the solution in example 3.2 contains shocks whose locations strongly depend on the values of the coefficients in the PDE. In both examples 3.1 and 3.2, the dynamics are learned from t∈[0,0.05] but the comparisons are done at T=0.1, showing the possibility of leveraging this type of learning in order to extrapolate dynamics and predict critical events (such as shock formation).

Figure 4.

Example 3.2: inviscid Burgers' equation with noise =73.7%. Plot (a) is the original simulation (black, solid) using 1024grid points and the learned solution (blue, dotted) using 256 grid points at time T=0.1. Plot (b) is the absolute value of the error between the two solutions projected onto the coarse grid, which shows that the learned dynamics agree with the original data everywhere except at a few grid points near the shock. (Online version in colour.)

Figure 5.

Example 3.2: inviscid Burgers' equation with noise =73.7%. The space–time dynamics are plotted to provide a qualitative comparison between the original simulation (a) and the learned dynamics (b) over time. The error between the original simulation and the learned solution is small in norm: the L¹ error is 0.0163, the L² error is 0.0193 and the $L^{\mathrm{\infty }}$ error is 0.2705, which also includes the projection errors (difference in grid sizes). The horizontal axis is space and the vertical axis is time. (Online version in colour.)

Example 3.3 (Swift–Hohenberg equation) —

The Swift–Hohenberg equation is a scalar evolution equation with fourth-order derivatives

$$u_t=-{(1+\mathrm{\Delta })}^{2}u+\alpha u+\beta u^2-u^3,$$

with x∈[0,50π]². It is a model for thermal convection and solutions form various patterns depending on the choice of α and β. In this example, we will consider three parameter regimes where different patterns form: (α,β)=(0.1,0), which yield curves; (α,β)=(0.5,0), which yields a coral-like structure; and (α,β)=(0.05,0.5), which yields a hexagonal structure. For clarity, the Swift–Hohenberg equation can be written term by term as

$$u_t=-2\mathrm{\Delta }u-{\mathrm{\Delta }}^{2}u+(\alpha -1)u+\beta u^2-u^3.$$

These basic terms are used as some of the features in the learning method.

The input data for each column of table 3 use (α,β)=(0.1,0), (0.5,0) and (0.05,0.5) (respectively) and were simulated up to T=500, 5 and 15 (respectively). The noise levels are equal to 15.0%, 8.3% and 7.9% (respectively) and the learning parameter λ equals 0.5, 25 and 1 (respectively). The terms are all identified correctly and the coefficients are within 2.8% of the true value. This example shows that, although the feature space contains redundancies (because Δu=u_xx+u_yy), the method is able to find the underlying dynamics.

Table 3.

Example 3.3: Swift–Hohenberg. The true coefficients for each column are (α,β)=(0.1,0), (0.5,0) and (0.05,0.5) (respectively). The terms are all identified correctly and the coefficients are within 2.8% of the true value. This example shows that, although the feature space contains redundancies (because Δu=u_xx+u_yy), the method is able to find the underlying dynamics.

terms	curves, noise=15.0%	coral, noise=8.3%	hexagonal, noise=7.9%
u	−0.8999	−0.5001	−0.9500
u2	0	0	0.4999
u3	−0.9717	−0.9751	−0.9797
u4	0	0	0
ux	0	0	0
uy	0	0	0
$u_x^2$	0	0	0
$u_y^2$	0	0	0
uyux	0	0	0
uxx	0	0	0
uyy	0	0	0
uxy	0	0	0
Δu	−1.9842	−1.9832	−1.9843
Δ2u	−0.9843	−0.9838	−0.9844

Figure 6a shows the original dynamics on a 65 536 point grid and figure 6b uses the coefficients from table 3 to simulate the data on a 16 384 point grid. The solutions are generated using a backward Euler step on the linear terms and a forward Euler step on the nonlinear terms. The solutions are plotted at T=250, 100 and 150, respectively. The learned solutions are qualitatively very similar to the original ones, thereby demonstrating the proposed method's ability to identify the correct parameter regimes for the various patterns. This could have applications to bifurcation and structure predictions.

Figure 6.

Example 3.3: Swift–Hohenberg. (a) The original dynamics on a 65 536 point grid and (b) the coefficients from table 3 used to simulate the data on a 16 384 point grid. The solutions are plotted at T=250, 100 and 150, respectively. (Online version in colour.)

Example 3.4 (Cahn–Hilliard equation) —

The Cahn–Hilliard equation,

$$u_t=-\gamma {\mathrm{\Delta }}^{2}u+\mathrm{\Delta }(u^3-u)$$

with x∈[0,50π]², is a model for phase separation in fluids. In this example, we will assume that the model takes the form: u_t=ΔF(u), and use this as our ansatz to construct a feature space. In order to validate the proposed method's robustness to the size of the data used in the learning, we will vary the number of time stamps inputted in the algorithm.

The data are simulated up to T=2 using the Crank–Nicolson method with γ=0.5. In table 4, the terms are all identified correctly and the coefficients are within 1% of the true value. Similar to example 3.3, the learning is stable with respect to redundancies in the feature space. The input data to the learning method use uniform time steps with a total of 385, 85 and 25 time stamps. In all cases, the noise level is kept at 5.0%. The learned coefficients displayed in table 4 show that the method is robust to the number of time stamps used in the learning process.

Figure 7a shows the original data at T=20 and figure 7b uses the coefficients from column 3 of table 4 to simulate the data on 65 536 grid points. The solutions are generated using the Crank–Nicolson method. The learned solution, which uses the coefficients learned from 25 time stamps, is close to the original data: the L¹ error is 0.0058 and the L² error is 0.0329. Therefore, this suggests that the proposed method is robust to the size of the data.

Table 4.

Example 3.4: Cahn–Hilliard. The noise level is fixed at 5.0% for all three columns. The coefficients are within 1% of the true value. By varying the number of time steps retained in the simulation and inputted into the learning algorithm, we see that the method seems robust to the size of the data.

terms	coefficients, 385 time steps	85 time steps	25 time steps
Δu	−0.9963	−0.9936	−0.9964
Δ(u2)	0	0	0
Δ(u3)	0.9952	0.9911	0.9927
Δ(u4)	0	0	0
Δux	0	0	0
Δuy	0	0	0
Δuxx	0	0	0
Δuyy	0	0	0
Δuxy	0	0	0
Δ2u	−0.0996	−0.0992	−0.0994
Δ2uxx	0	0	0
Δ2uxy	0	0	0
Δ2uyy	0	0	0
Δ3u	0	0	0

Figure 7.

Example 3.4: Cahn–Hilliard equation. Plot (a) is the original data at T=20 and (b) uses the coefficients from column 3 of table 4 to simulate the data on 65 536 grid points. The learned solution, which uses the coefficients learned from 25 time stamps, is close to the original data. (Online version in colour.)

Example 3.5 (Navier–Stokes, low Reynolds number) —

The vorticity formulation for the two-dimensional Navier–Stokes equation is

$$u_t+({\mathrm{\nabla }}^{\perp }{\mathrm{\Delta }}^{-1}u)\cdot \mathrm{\nabla }u=\frac{1}{Re}\mathrm{\Delta }u.$$

Using this equation, the vorticity is simulated with x∈[0,2π]² and is computed up to T=0.25 using the Crank–Nicolson method with various Reynolds numbers (Re>0). We will first consider the low Reynolds number case. Using Re=4 and Re=100, the learned coefficients can be found in columns 2 and 3 of table 5. The terms are all identified correctly and the coefficients are within 1% of the true value. In these two cases, the solutions exhibit smooth flows. Similar to examples 3 and 4, the learning is stable with respect to redundancies in the feature space. In column 1 of table 5, the Reynolds number is chosen to be small enough so that the dynamics are dominated by viscous flow. The coefficients in column 1 of table 5 show that the method identifies the viscous flow model, which is the true dominant behaviour of the data.

Table 5.

Example 3.5: Navier–Stokes, low Reynolds number. Note that −Δv=u. In the first column, the Reynolds number is chosen to be small enough so that the dynamics are dominated by viscous flow. The coefficients in the first column show that the method identifies the viscous flow model, which is the true dominant behaviour of the data. The coefficients are within 1% of the true value.

terms	$Re=\frac{1}{10}$, noise=56.0%	Re=4, noise=56.8%	Re=100, noise=60.4%
u	0	0	0
u2	0	0	0
u3	0	0	0
ux	0	0	0
uy	0	0	0
$u_x^2$	0	0	0
$u_y^2$	0	0	0
uyux	0	0	0
vxux	0	0	0
vyux	0	−1.0047	−1.0004
vxuy	0	1.0057	1.0006
vyuy	0	0	0
uxx	0	0	0
uyy	0	0	0
uxy	0	0	0
Δu	10.0563	0.2513	0.0101
Δ2u	0	0	0

Example 3.6 (Navier–Stokes, high Reynolds number) —

Using the vorticity formulation, we examine the turbulent flow case (i.e. Re≫1). The data are simulated using 1 048 576 grid points with Re=500 (very high Reynolds number relative to the coarse grid) and Re=1024 (vanishing viscosity on the coarse grid). It is known that the limiting flow (as the Reynolds number goes to infinity) of the Navier–Stokes flow on the torus is Euler's equation

$$u_t+({\mathrm{\nabla }}^{\perp }{\mathrm{\Delta }}^{-1}u)\cdot \mathrm{\nabla }u=0.$$

Thus, one can argue that the underlying dynamics are truly governed by Euler's equation. In table 6, the learning is done using 65 536 grid points. The method identifies the correct terms (with respect to Euler's equation) and the coefficients are within 1% of the true value. The method picks out the coefficients to the intrinsic dynamics and not the equations used to simulate the data.

Figure 8a shows the original simulation using 1 048 576 grid points and figure 8b is the learned solution using the coefficients from tables 5 and 6 on a 65 536 point grid. From left to right the columns use: $Re=\frac{1}{10}$, Re=4, Re=100 and Re=1024. The results of figure 8 show that the learned PDEs yield qualitatively similar solutions.

Table 6.

Example 3.6: Navier–Stokes, high Reynolds number. The simulations are done on a 1 048 576 point grid and the learning is performed using 65 536 grid points.The method identifies the correct terms (with respect to Euler's equation) and the coefficients are within 1% of the true value. The method picks out the coefficients to the intrinsic dynamics and not the equations used to simulate the data. Note that −Δv=u.

terms	Re=500, noise=15.3%	Re=1024, noise 14.9%
u	0	0
u2	0	0
u3	0	0
ux	0	0
uy	0	0
$u_x^2$	0	0
$u_y^2$	0	0
uyux	0	0
vxux	0	0
vyux	−0.9945	−0.9931
vxuy	0.9976	0.9928
vyuy	0	0
uxx	0	0
uyy	0	0
uxy	0	0
Δu	0	0
Δ2u	0	0

Figure 8.

Example 3.6 Navier–Stokes, high Reynolds number. (a) shows the original simulation using 1 048 576 grid points and (b) uses the coefficients from tables 5 and 6 to compute the solution with 65 536 grid points. From left to right the columns use: $Re=\frac{1}{10}$, Re=4, Re=100 and Re=1024. (Online version in colour.)

Example 3.7 (homogenization of convection equation) —

One possible application of this method is the homogenization of highly oscillatory or random dynamics. As an example, consider the convection equation with oscillatory velocity

$$u_t^{ϵ}+a\left(\frac{x}{ϵ}\right)\cdot \mathrm{\nabla }{u}^{ϵ}=0,$$

with initial data u₀(x,x/ϵ) and x∈[0,2π]. The velocity field is assumed to be periodic and divergence free, i.e. div(a)=0. This is a linear model related to the homogenization of the incompressible Euler equation. Assuming sufficient smoothness, as $ϵ\to 0$, the solution homogenizes: $u^{ϵ}\to u$, where u solves $u_t+\overline{a}(x)\cdot \mathrm{\nabla }u=0$, with initial data ${\overline{u}}_0$, where $\overline{v}$ is the average of v over the domain (see [37]).

For this example, the oscillatory velocity field is defined as

$$a(\frac{x}{ϵ},\frac{y}{ϵ})={[1+\frac{1}{2}\sin \left(\frac{x}{ϵ}\right)\cos \left(\frac{y}{ϵ}\right),\hspace{0.17em}-2-\frac{1}{2}\cos \left(\frac{x}{ϵ}\right)\sin \left(\frac{y}{ϵ}\right)]}^t$$

and the data are in t∈[0,0.25]. In table 7, the data are simulated using 1 048 576 grid points with $ϵ=\frac{1}{16}$ and $ϵ=\frac{1}{64}$. The larger value of ϵ yields an oscillatory solution while the smaller value is closer (numerically) to the homogenized dynamics (see figure 9a,b). Step 4 of the method (the de-biasing step) is not used because the restriction of the normal equations to the identified terms is not well conditioned. The learning is done using 65 536 grid points. In the first column of table 7, the method identifies the correct terms but does not approximate the correct values of the coefficients because the velocity field is far from uniform and the approximation with this feature space is insufficient. In the second column of table 7, the method identifies the correct terms and approximates the coefficients to within 0.005% in relative error. Even though the data are not completely homogenized and no de-biasing is used, the method can approximate the underlying uniform dynamics.

Table 7.

Example 3.7: homogenization of the convection equation. The data are simulated using 1 048 576 grid points with $ϵ=\frac{1}{16}$ and $ϵ=\frac{1}{64}$. Step 4 of the method (the de-biasing step) is not used because the restriction of the normal equations to theidentified terms is not well conditioned. The learning is done using 65 536 grid points. In the first column, the method identifies the correct terms but does not approximate the correct values of the coefficients because the solution is far from homogenized. In the second column, the method is within 0.005% of the true coefficients. No de-biasing is used because the restriction of the matrix $\sum _{k=1}^M{F}_w^T(t_k)F_w(t_k)$ onto the support set $\mathcal{I}$ is ill conditioned.

terms	$ϵ=\frac{1}{16}$, noise=9.5%	$ϵ=\frac{1}{64}$, noise=9.4%
u	0	0
u2	0	0
u3	0	0
ux	−1.3166	−1.0000
uy	1.3537	1.9999
$u_x^2$	0	0
$u_y^2$	0	0
uyux	0	0
uxx	0	0
uyy	0	0
uxy	0	0
Δu	0	0
Δ2u	0	0

Figure 9.

Example 3.7: homogenization of the convection equation. (a,b) are the simulations computed using 1 048 576 grid points with $ϵ=\frac{1}{16}$ (a) and $ϵ=\frac{1}{64}$ (b). The larger value of ϵ yields an oscillatory solution while the smaller value is closer (numerically) to the homogenized dynamics. (c) is the learned solution using 65 536 grid points. (Online version in colour.)

Example 3.8 (travelling waves, solitons and ambiguity) —

As the coefficients and governing equations are learned from data, there is the potential for ambiguity when different equations generate similar datasets. For example, consider the Korteweg–de Vries (KdV) equation

$$u_t+u_{xxx}+6{(u^2)}_x=0,$$ 3.1

on x∈[0,1] with initial data $u(x,0)=500\hspace{0.17em}{\text{sech}}^2\left(5\sqrt{100}(x-\frac{1}{2})\right)$, whose solution is given by

$$u(x,t)=500\hspace{0.17em}{\text{sech}}^2\left(5\sqrt{100}(x-1000t-\frac{1}{2})\right).$$ 3.2

Equation (3.2) is also a solution of the one-way wave equation

$$u_t+1000u_x=0,$$ 3.3

which leads to a potential issue with the recovery of the governing equation. As the one-way wave equation has fewer terms (a sparser governing equation), we expect the method to choose equation (3.3) over equation (3.2).

In table S1 in the electronic supplementary material, the data are simulated by equation (3.1) over t∈[0,0.1] using 512 grid points. In the first column of table S1 in the electronic supplementary material, we see that the method selects the one-way wave equation as the learned governing equation. This is indeed the sparsest governing equation (because it only uses one term in the model). In the second column of table S1 in the electronic supplementary material, u_x is removed as a potential feature and the method is applied to the remaining terms. This results in the correct identification of the (intended) governing equation. This example shows that, when the dynamics governing the data can be represented by multiple equations, the method may not select the intended equation. However, the method will select a valid model equation (in this case the one-way wave equation). Model ambiguity is a potential issue with this approach when a large feature space is used.

Example 3.9 (additive noise on the data) —

In examples 3.1–3.8, the noise is assumed to be sampled from a Gaussian distribution and added directly to the velocity estimate. The underlying assumption is that the velocity is more severely corrupted by noise than the given data. In this example, Gaussian noise is added directly to the data themselves. As the data themselves are noisy, direct velocity estimation using finite differences is unstable.

Consider repeating example 3.1, where the data are simulated using

$$u_t={\left(\frac{u^2}{2}\right)}_x+\nu u_{xx}$$

with $\nu =\frac{1}{10}$, t∈[0,0.05] and x∈[0,1]. The data are corrupted directly and the velocity is calculated from the noisy data. Applying the method to this test case shows a recovery transition: the method produces accurate results before noise ≈3% and inaccurate results after ≈3% (the noise level is computed in the same way as before). This is due to an inaccurate approximation to the temporal and spatial derivatives. Data smoothing (or denoising) is probably needed to produce an accurate approximation. One possible smoothing approach is as follows: given noisy data uⁿ, a smooth approximation u^s is constructed by $u^s=\underset{w}{\mathrm{argmin}}\hspace{0.17em}\beta \parallel \mathrm{\nabla }w{\parallel }^2+\parallel w-u^n{\parallel }^2$ with a smoothing parameter β>0. The solution is given by u^s=(I−βΔ)⁻¹uⁿ and is particularly simple to implement in the spectral domain (see [24] for more on spectral filtering). The learning algorithm is then applied to the filtered data u^s.

In table S2 in the electronic supplementary material, data from example 3.1 are corrupted by various noise levels and smoothed using β=5×10⁻⁵. When the noise level between the smoothed velocity and the velocity calculated from the noisy data is about 50%, the learned model is accurate (see the first column of table S2 in the electronic supplementary material). In figure S1 in the electronic supplementary material, a visual comparison between the estimated velocities is provided. Note that spectral filtering preserves the general spatial structure but may still contain some noisy frequencies. When the noise level between the smoothed velocity and the velocity calculated from the noisy data is large, the learned model loses accuracy and introduces false terms to compensate for the noise (see the second column of table S2 in the electronic supplementary material). This shows a potential issue with this approach when applied directly to noisy data. In particular, as derivatives must be calculated, instabilities are likely to form. Smoothing approaches using other regularizers, such as the total variation semi-norm, could be used to better approximate the derivative of noisy data (see [38]).

4. Conclusion

Sparse regression and other machine learning methods can provide scientists with important techniques to support their path to scientific discovery. This work presented one approach along this direction, in particular the use of L¹ regularized least-squares minimization to select the correct features that learn the governing PDE. The identification of the terms in the PDE and the approximation of the coefficients come directly from the data. The numerical experiments show that the proposed method is robust to data noise and size, is able to capture the true features of the data, and is capable of performing additional tasks such as extrapolation, prediction and (simple) homogenization.

Competing interests

I declare I have no competing interests.

Funding

H.S. was supported by NSF 1303892.