Trend to equilibrium for the kinetic Fokker-Planck equation via the neural network approach

Highlights

• •Deep Neural Network approach to solve the kinetic Fokker-Planck equation in a bounded interval.
• •Theoretical evidence on the relationship between the Deep Neural Network solutions and the a priori analytic solutions.
• •There exists a sequence of weights such that the total sum of loss functions converges to 0.
• •The neural networks equipped with such weights converge to an analytic solution.
• •The long-time asymptotics of the neural network solutions to the Fokker-Planck equation under various boundary conditions.

Abstract

The issue of the relaxation to equilibrium has been at the core of the kinetic theory of rarefied gas dynamics. In the paper, we introduce the Deep Neural Network (DNN) approximated solutions to the kinetic Fokker-Planck equation in a bounded interval and study the large-time asymptotic behavior of the solutions and other physically relevant macroscopic quantities. We impose the varied types of boundary conditions including the inflow-type and the reflection-type boundaries as well as the varied diffusion and friction coefficients and study the boundary effects on the asymptotic behaviors. These include the predictions on the large-time behaviors of the pointwise values of the particle distribution and the macroscopic physical quantities including the total kinetic energy, the entropy, and the free energy. We also provide the theoretical supports for the pointwise convergence of the neural network solutions to the a priori analytic solutions. We use the library PyTorch, the activation function tanh between layers, and the Adam optimizer for the Deep Learning algorithm.

1. Introduction

1.1. Motivation

One of the main questions of interest in the study of the dynamics of rarefied gas particles is on the time-asymptotic behaviors of the particle density distribution and its macroscopic quantities. Since the era of Ludwig Boltzmann, the validity of the time-irreversibility and the entropy production of the Boltzmann kinetic equation has long been a bone of contention due to the Poincaré recurrence theorem. Indeed, it is an important problem to show that the time-scale of the convergence towards the equilibrium is much smaller than the time-scale of the validity of the Boltzmann equation.

In this work, we study the time-irreversibility and the entropy production of the kinetic Fokker-Planck equation, which is a fundamental model for a physical plasma. This question has been heavily studied in both analytic and numerical aspects. The Lyapunov functional for the kinetic Fokker-Planck equation is given by the relative entropy functional with respect to the steady-state, which we will describe in more detail below, and we provide a newly-devised numerical method of using a machine learning algorithm for the study of the large-data asymptotic behaviors of the Deep Neural Network (DNN) solutions to the kinetic Fokker-Planck equation in a bounded domain.

In fact, it has been considered numerically difficult to simulate an initial boundary value problem for a kinetic partial differential equation. One of the difficulties arises from the fact that a numerical solution is closely related to a computation in a bounded domain by nature, whereas the boundary of a kinetic equation still consists of the whole space in the momentum variable v. In order to resolve this issue, people (cf. [31]) have considered the decaying properties of the solutions to a kinetic equation; i.e., if one could observe that the solution decays fastly enough in time outside a compact set, then the size (in a specific sense) of solution outside the compact region is close to zero so one can treat the difference as a small error numerically. Another difficulty arises from the huge computational cost on the simulation of a kinetic partial differential equation due to the extra dimensions from the momentum variable v. This becomes worse when we consider an integral-based operator such as the classical Boltzmann collision operator or the Landau-Boltzmann collision operator.

The Deep Learning method is a new approach for solving partial differential equations that can resolve some of the issues that we listed above. The Deep Learning algorithm has an advantage of being intuitive and easy to be executed via the backpropagation method. The algorithm produces approximated solutions that can be differentiated continuously on domains. Also, it is relatively easier to put information in the algorithm by adding a term to the loss function via the Deep Learning approach. For example, we can simply include a term regarding the conservation of the total mass of the system in the total Loss function of the algorithm, as the scheme is proposed to conserve the total mass under several boundary conditions. In addition, the Deep Learning algorithm can be extended to arbitrary domains so it is not necessary to worry about how to split a domain into triangles as in the numerical methods. Although our DNN algorithm uses the uniform grids of domains, using sampling random points from a domain can be applied to a special domain in higher dimension equations [64].

However, there are also some weaknesses of the approach that one should be careful of. Firstly, there is no guarantee that the Deep Learning algorithm will converge and it is theoretically difficult to show the convergence of the Deep Learning Algorithm. Also, it is hard to evaluate the accuracy of the Deep Learning algorithm in contrast with the numerical methods. So far, lots of varied measures are suggested to express the performance of a DNN model. Due to the randomly initialized parameters in a Deep Learning algorithm, each learning could give slightly different solutions, while the numerical methods are deterministic.

1.2. A brief history of the past results

1.2.1. Mathematical results on the Fokker-Planck equation

The existence and the uniqueness of the solutions to the Fokker-Planck equation have been heavily studied. Dita [30] constructs an analytic solution of the stationary 1D Fokker-Planck equation with an absorbing boundary. Then Protopopescu [57] deals with the stationary 1D Fokker-Planck equation under some velocity-dependent external forces with some boundary conditions. DiPerna-Lions [29] established stability results for sequences of solutions and global existence for the Cauchy problem of the Fokker-Planck equation with large data. Desvillettes and Villani [28] showed that a polynomial decay for the solutions with suitable initial conditions to a global equilibrium with the help of logarithmic Sobolev inequalities. The irregular coefficients in Fokker–Planck, and transport equations were also studied in [12], [51]. Later, Mischler [53] provided the stability results of DiPerna-Lions renormalized solutions for Fokker-Planck equations with Maxwell boundary conditions. [62] showed that the well-posedness of the steady Fokker–Planck solutions in smooth domains with absorbing boundary conditions.

Regarding the hypoellipticity of the equation, Hwang-Jang-Velazquez in [40] showed the hypoellipticity properties for the Fokker-Planck equation with absorbing boundary conditions, and this has been generalized in [39] by Hwang-Jang-Jung. Also, Hwang-Phan [41] extended the results of [40] to the inflow boundary conditions.

Regarding the Vlasov-Poisson-Fokker-Planck system, Victory and O'Dwyer [65] proved the existence of local in time solutions to the VPFP system. Neunzert, Pulvirenti, and Triolo in [54] used a probabilistic method to prove the global existence of smooth solutions in one and two dimensions. Degond and Pierre [26] proposed a fully deterministic proof of the existence of global in time smooth solutions for the Vlasov-Fokker-Planck equations in one and two dimensions. They also proved that the solution of the VPFP equation converges to the solution of the VP equation as the coefficients in the FP operator term go to zero. Bouchut in [9], [10] showed the existence and uniqueness of strong and global in time solutions to the three-dimensional VPFP equation. The asymptotic behavior and the convergence to the equilibrium of the solutions to the Vlasov(-Poisson)-Fokker-Planck equation were studied in [8], [16], [18], [19], [20]. The stationary states and large time behavior of the Wigner-Fokker-Planck equation were studied in [3]. The low and high field scaling limits of Vlasov-Poisson-Fokker-Planck system were considered in [2]. The global existence and uniqueness of weak solutions to kinetic Kolmogorov–Vicsek models were considered in [34]. The Vlasov-Poisson-Fokker-Planck system with uncertainty and multiple scales was studied in [43], [70]. Regarding the recent development in the qualitative properties of the VPFP system, F. Bouchut and J. Dolbeault in [11] showed the large time behaviors and the steady-states for the solutions of the VPFP equation in the case that the particles occupy the whole space ${\mathbb{R}}^3$. Another related result is in [8] which studied the large time asymptotics for the VPFP system in a bounded domain with the reflection type boundary conditions.

1.2.2. Numerical results on kinetic equations

In this section, we would like to introduce a few past results on the numerical analysis of kinetic equations. Regarding past numerical results on the Fokker-Planck equation, we would like to start with some early developments via the conservative finite element method [5], [15], [21], [27], [48]. Regarding the (spatially homogeneous and inhomogeneous) nonlinear Landau collision equation, which is a generalized version of the linear Fokker-Planck equation, we have the early developments via the conservative finite element method [7], [13], [14], [24], [49], [56]. We also record a result via the spectral method [33].

We note that many methods have been proposed to solve the Vlasov-Poisson-Fokker-Planck equation and the Fokker-Planck-Landau equation. Allen-Victory [1] and Havlak-Victory [36] proposed the random particle method with the analysis and the computational study of its method. The finite difference scheme was also used to solve the VPFP with the periodic 1D case in [22], [60], [61]. Another approach is the deterministic particle methods which are based on the characteristic trajectories for the transport term of the Vlasov-Poisson-Fokker-Planck equation [37]. Wollman and Ozizmir [67], [68] combined the deterministic method with the periodic regriding of the distribution function to approximate the solutions more stable and accurate. This method is extended to the two-dimensional case in [69]. The fast spectral method, which is also an alternative method for the numerical approximation was introduced in [55]. In [32], Filbet and Pareschi presented a new spectral method that could be extended to the nonhomogeneous situation.

1.2.3. Neural networks and the Cauchy problem of a PDE

The neural network architecture was first introduced in [52]. Then, Cybenko [25] established sufficient conditions for which a continuous function can be approximated by finite linear combinations of single hidden layer neural networks with the same univariate function. Hornik-Stinchcombe-White [38] also showed measurable functions can be approximated by the multi-layer feedforward networks with a monotone sigmoid function. Then Cotter [23] extended the result of [38] to a new architecture, and later Li [50] proved that the multi-layer network with one hidden layer can approximate a target function and its higher partial derivatives on a compact set.

Though the theory of artificial neural networks as an approximation to solutions of differential equations has such a long history, the actual implementation of the ideas has a relatively short history due to the technical and algorithmical issues. Solving differential equations using an artificial neural network with architecture including one single layer and ten units were studied in [46], and the results were extended to a domain with complex boundaries in [47]. Then Jianyu et al. [42] replaced an activation function with a radial basis and solved the Poisson equation.

More recently, Berg-Nyström [6] used a DNN to solve steady problems in 1D and 2D space dimensions with complex geometry. Han-Jentzen-Weinan [35] then applied DNNs to the stochastic process for solving high dimensional differential equations. Very recently, Raissi-Perdikaris-Karniadakis [58] suggested an algorithm that solves both forward and inverse problems. Sequentially, Jo et al. [44] gave a theoretical reason that neural networks converge to analytic solutions in forward and inverse problems. Also, the application of other architectures or learning strategies of a convolutional neural network or reinforcement learning has been studied in [63], [66]. Also, the neural network approaches to solve a partial differential equation have been proposed in [59], [64].

1.3. The Fokker-Planck equation

The d−dimensional kinetic Fokker-Planck equation reads as

$${\partial }_{t}f+v\cdot {\mathrm{\nabla }}_{x}f={\mathrm{\nabla }}_v\cdot (\sigma {\mathrm{\nabla }}_{v}f+\beta vf),(t,x,v)\in [0,T]\times \mathrm{\Omega }\times {\mathbb{R}}^d,$$ (1.1)

where $\mathrm{\Omega }\subset {\mathbb{R}}^d$, $\sigma >0$ is the diffusion coefficient, $\beta \ge 0$ is the friction coefficient, and $f=f(t,x,v)$ is the probabilistic density distribution of particles. In this paper, we consider the following 1-dimensional kinetic Fokker-Planck equation in a bounded interval $\mathrm{\Omega }=(-1,1)$ and $d=1$ as

$${\partial }_{t}f+v{\partial }_{x}f={\partial }_v(\sigma {\partial }_{v}f+\beta vf),(t,x,v)\in (0,T)\times \mathrm{\Omega }\times \mathbb{R},$$ (1.2)

subject to the initial condition

$$f(0,x,v)=f_0(x,v)\ge 0,(x,v)\in \mathrm{\Omega }\times \mathbb{R}.$$ (1.3)

The boundary conditions that we impose will be introduced in Section 1.4.

1.4. Boundary conditions

We define the outward normal vector $n_x$ on the boundary ∂Ω as

$$n_x\stackrel{\text{def}}{=}\frac{\mathrm{\nabla }\zeta (x)}{|\mathrm{\nabla }\zeta (x)|}.$$ (1.4)

Since our $\mathrm{\Omega }=(-1,1)$ and hence $\partial \mathrm{\Omega }=\{-1,1\}$, our $n_x=\hat{x}$ at $x=1$ and $=-\hat{x}$ at $x=-1$. Throughout this paper, we will denote the phase boundary of $\partial \mathrm{\Omega }\times \mathbb{R}$ as $\gamma \stackrel{\text{def}}{=}\partial \mathrm{\Omega }\times \mathbb{R}$. Additionally we split this boundary into an outgoing boundary ${\gamma }_{+}$, an incoming boundary ${\gamma }_{-}$, and a singular boundary ${\gamma }_0$ for grazing velocities, defined as

$${\gamma }_{+}\stackrel{\text{def}}{=}\{(x,v)\in \mathrm{\Omega }\times \mathbb{R}:n_x\cdot v>0\},{\gamma }_{-}\stackrel{\text{def}}{=}\{(x,v)\in \mathrm{\Omega }\times \mathbb{R}:n_x\cdot v<0\},{\gamma }_0\stackrel{\text{def}}{=}\{(x,v)\in \mathrm{\Omega }\times \mathbb{R}:n_x\cdot v=0\}.$$ (1.5)

Since our $\mathrm{\Omega }=(-1,1)$ and $\partial \mathrm{\Omega }=\{-1,1\}$, we have

$${\gamma }_{+}=\{(1,v)|v>0\}\cup \{(-1,v)|v<0\},{\gamma }_{-}=\{(1,v)|v<0\}\cup \{(-1,v)|v>0\},{\gamma }_0=\{(1,0),(-1,0)\}.$$ (1.6)

In terms of the probability density function f, we formulate the following four physical boundary conditions throughout the paper.

1.4.1. Specular reflection boundary condition

$$f(t,x,v){|}_{{\gamma }_{-}}=f(t,x,-v),$$ (1.7)

for $x=-1$ and 1.

1.4.2. Diffusive reflection boundary condition

$$f(t,x,v){|}_{{\gamma }_{-}}=C\mu (v)\underset{w\cdot n_x>0}{\int }f(t,x,w)|w\cdot n_x|dw,$$ (1.8)

for $x=-1$ and 1, where

$$C={(\underset{v\cdot n<0}{\int }\mu (v)|v\cdot n|dv)}^{-1}\text{and}\mu (v)\stackrel{\text{def}}{=}{e}^{-\frac{v^2}{2}},$$

for both $n=\hat{x}$ and $=-\hat{x}$.

1.4.3. Periodic boundary condition

$$f(t,x,v){|}_{{\gamma }_{-}}=f(t,-x,v),$$ (1.9)

for $x=-1$ and 1.

1.4.4. Absorbing boundary condition

$$f(t,x,v){|}_{{\gamma }_{-}}=0,$$ (1.10)

for $x=-1$ and 1.

1.4.5. Inflow boundary condition

$$f(t,x,v){|}_{{\gamma }_{-}}=g(t,x,v),\text{for}x=-1\text{ and }1,$$ (1.11)

with a given function g.

1.5. The equilibrium state, the Lyapunov functional, and the balance laws

It is well-known that the linear Fokker-Planck equation (1.2) has a global equilibrium solution. This is called the global Maxwellian solution and the form of the steady-state was introduced in [8, Theorem 1.2, p. 1349] as follows:

$$f_{\infty }(v)=\frac{M}{C{(2\pi \frac{\sigma }{\beta })}^{0.5}}\exp (-\frac{\beta }{\sigma }\frac{|v{|}^2}{2})$$ (1.12)

where $M={\Vert f_0(\cdot ,\cdot )\Vert }_{L_{x,v}^1}$ and $C=|\mathrm{\Omega }|=2$. The Lyapunov functional $\eta (t)$ is defined by the relative entropy of the solution f with respect to the stationary distribution $f_{\infty }$ as

$$\eta (t)\stackrel{\text{def}}{=}\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }f\log \left(\frac{f}{f_{\infty }}\right)dxdv=\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }f\log fdxdv+\frac{\beta }{2\sigma }\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }|v{|}^{2}fdxdv+\log \left(\frac{C{(2\pi \frac{\sigma }{\beta })}^{0.5}}{M}\right)\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }fdxdv=-\text{Ent}(t)+\frac{\beta }{\sigma }\text{KE}(t)+\log \left(\frac{C{(2\pi \frac{\sigma }{\beta })}^{0.5}}{M}\right)\text{Mass}(t),$$ (1.13)

where the entropy of the system “Ent”, the total kinetic energy “KE”, and the total mass “Mass” of the system are defined as

$$\text{Ent}(t)\stackrel{\text{def}}{=}-\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }f\log fdxdv,$$

$$\text{KE}(t)\stackrel{\text{def}}{=}\frac{1}{2}\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }|v{|}^{2}fdxdv,$$

and

$$\text{Mass}(t)\stackrel{\text{def}}{=}\underset{\mathrm{\Omega }\times \mathbb{R}}{\int }fdxdv.$$

We define the free energy functional “FE” as

$$\text{FE}(t)\stackrel{\text{def}}{=}\text{KE}(t)-\frac{\sigma }{\beta }\text{Ent}(t),$$

whose time-derivative is equivalent to that of the Lyapunov functional η if the total mass is conserved. In the cases of the specular reflection, the diffusive reflection, and the periodic boundaries, we expect that the Lyapunov functional satisfies ${\eta }^{\prime }(t)\le 0$, which is a manifestation of the second law of thermodynamics.

The following balance laws on the macroscopic quantities [8, p. 1350] are also well-known:

• •
  Balance of the total mass:

  $$\frac{d}{dt}{\Vert f(t,\cdot ,\cdot )\Vert }_{L_{x,v}^1}=-\underset{(-1,1)\times \mathbb{R}}{\int }dxdv(v\cdot n_x)\gamma f(t,x,v).$$ (1.14)
• •
  Balance of the total kinetic energy:

  $$\frac{d}{dt}(\frac{1}{2}\underset{(-1,1)\times \mathbb{R}}{\int }dxdv|v{|}^{2}f(t,x,v))=-\frac{1}{2}\underset{\{-1,1\}\times \mathbb{R}}{\int }dxdv(v\cdot n_x)|v{|}^{2}f(t,x,v)-\beta \underset{(-1,1)\times \mathbb{R}}{\int }dxdv|v{|}^{2}f(t,x,v)+\sigma {\Vert f(0,\cdot ,\cdot )\Vert }_{L_{x,v}^1}.$$ (1.15)
• •
  Balance of the entropy:

  $$\frac{d}{dt}\underset{(-1,1)\times \mathbb{R}}{\int }dxdv(-f(t,x,v)\log f)(t,x,v)=-\beta {\Vert f(0,\cdot ,\cdot )\Vert }_{L_{x,v}^1}+\underset{\{-1,1\}\times \mathbb{R}}{\int }dxdv(v\cdot n_x)f(t,x,v)\log f(t,x,v)+4\sigma \underset{(-1,1)\times \mathbb{R}}{\int }dxdv{\left|{\mathrm{\nabla }}_v\sqrt{f}\right|}^2.$$ (1.16)

1.6. Main results, difficulties, and our strategy

The main results of the paper consist of two parts. One is on the theoretical evidence on the relationship between the DNN solutions and the a priori analytic solutions. More precisely, we first prove in Theorem 3.4 that a sequence of neural network approximated solutions, which make the total loss term converge to zero, exists if a ${\hat{C}}^{(1,1,2)}$ solution to the equation exists. Namely, the theorem implies we can always find appropriate weights of the neural network which reduce the total error functions as much as we want. However, since this does not guarantee the convergence of the approximated solutions as the total loss term converges to zero, we prove an additional Theorem 3.6 which provides us that the neural network solutions indeed converge to the analytic solution as the total loss term vanishes. This is proved under the suitable smallness on the boundary condition (3.3) which corresponds to the truncation of the v domain in the choice of our grid points, as introduced in Section 2.2. Though the equation is linear, the derivation of an energy inequality for the convergence includes the boundary integrations not just in x variable but also in v variable due to the truncation of v space and hence we needed the smallness condition on the difference of the approximated solutions and the analytic solution as in the condition (3.3) to deal with the boundary integrations in v variable. Also, in order to create a sufficiently large dissipation term on the left-hand side of the energy inequality, we follow the transformation of the Fokker-Planck equation motivated by Carrillo [17].

On the other hand, we provide in Section 4 the time-asymptotic behaviors of the neural network approximated solutions and the macroscopic physical quantities to the kinetic Fokker-Planck equation under the various types of initial and boundary conditions with different values of diffusion and friction coefficients. The learning algorithms of our DNN algorithm are based on the development of the total loss function that contains the information of the Fokker-Planck equation and the initial-boundary conditions and the appropriate selection of the grid points and the weights for each loss term so they fit to the physical conditions. Our algorithm uses the library PyTorch and the hyper-tangent tanh activation function between the layers. For the purpose of minimizing the total loss term, we use the Adam optimizer which is based on the stochastic gradient descent. One of the reasons that many of the existing numerical methods fail to simulate the solutions to several kinetic equations is on the issue of the conservation of the total mass. In order to deal with the difficulty, we remark that we increased the learning efficiency of the model by adding the mass conservation law to the total loss function. It is indeed a huge advantage of using the neural network approach that we can intuitively add any physically relevant conditions on the necessary quantities when we design the total loss function, though the reduction of the total loss function is another issue of concern that we need to take care of.

In addition, our numerical simulations indeed provide several predictions on the time-asymptotic behaviors of the a priori analytic solutions to the kinetic Fokker-Planck equation and the physically relevant macroscopic quantities in the pointwise sense. Namely, we provide the time-asymptotic plots on the actual pointwise value at each grid under varied types of conditions, and these definitely contain much more information than the graphs on the asymptotic behaviors of the general weighted $L^p$-moments of the solution for some p. To the best of authors' knowledge, there have been no numerical plots of the pointwise values of the approximated solutions for a kinetic equation under the varied types of the physical boundary conditions, such as the inflow-type and the reflection-type boundary conditions. Via the numerical simulations, we have observed the asymptotic behaviors of the solutions to the equation that were theoretically proved under some specific situations; our numerical simulations predict the pointwise convergence of the neural network solutions to the global Maxwellian for varied types of the boundary conditions and the varied coefficients for the diffusion and the friction. Under the specular reflection boundary condition, we also provide the different rates of convergence under the varied choices of the friction coefficient β.

1.7. Outline of the paper

In Section 2, we will introduce in detail our DNN architecture and method to solve the Cauchy problem to the Fokker-Planck equation. This will include the detailed descriptions on the hidden layers and the definitions of grid points and loss functions. In Section 3.1, we will first prove that there exists a sequence of weights such that the total sum of loss functions converges to 0 and that neural networks equipped with such weights converge to the analytic solutions. In Section 4, we provide our numerical simulations and the result for each initial and boundary condition. Several plots will show the actual values of the distribution function at each time and spatial variable and will provide the asymptotic behaviors of macroscopic quantities as well as the actual values of the distribution. Finally, in Section 5, we complete the paper by summarizing our methods and the results.

2. Methodology: neural network approach

In this section, we introduce our DNN structure and method to solve the Cauchy problem to the 1-dimensional kinetic Fokker-Planck equation (1.2).

2.1. Our Deep Learning algorithm and the architecture

A Deep Learning algorithm is a non-linear function approximation method via a DNN structure. A DNN consists of several layers, and each layer has several neurons which are connected to pre- and post- layer neurons. Connected neurons have a relation with an affine transformation and a non-linear activation function. We denote the approximated function as $f^{nn}(t,x,v;m,w,b)$ and we suppose our DNN has L layers; in other words, our DNN has an input layer, $L-1$ hidden layers, and an output layer. The input layer takes $(t,x,v)$ as input and the final layer gives $f^{nn}(t,x,v;m,w,b)$ as the output. We denote the relation between the l-th layer and the $(l+1)$-th layer ($l=1,2,...,L-1$) as

$$z_j^{(l+1)}=\sum _{i=1}^{m_l}{w}_{ji}^{(l+1)}{\overline{\sigma }}_l(z_i^l)+b_j^{(l+1)},$$

where $m=(m_0,m_1,m_2,...,m_{L-1})$, $w={\{w_{ji}^{(k)}\}}_{i,j,k=1}^{m_{k-1},m_k,L}$, $b={\{b_j^{(k)}\}}_{j=1,k=1}^{m_k,L}$, and

• • $z_i^l$: the i-th neuron in the l-th layer
• • ${\overline{\sigma }}_l$: the activation function in the l-th layer
• • $w_{ji}^{(l+1)}$: the weight between the i-th neuron in the l-th layer and the j-th neuron in the $(l+1)$-th layer
• • $b_j^{(l+1)}$: the bias of the j-th neuron in the $(l+1)$-th layer
• • $m_l$: the number of neurons in the l-th layer.

Note that the relation between the input layer and the first-hidden layer is expressed as follows:

$$z_j^1=\sum _{i=1}^3{w}_{ji}^1{z}_i^0+b_j^1,$$

where $(z_1^0,z_2^0,z_3^0)=(t,x,v)$.

We use the library PyTorch and the hyper-tangent tanh activation function for our fully connected DNN. Our DNN has four layers which each layer has 3-128-256-128-2 neurons as shown in the following figure:

Regarding the optimization algorithm, we use Adam optimization algorithm, which is an extended algorithm of the stochastic gradient descent and is heavily used in the applications of the deep learning. We use a powerful technique called the automatic differentiation (AD), which computes the derivatives of the neural network output with respect to its input coordinates. The AD is the one of the most practical tools in scientific computing that differ from the usual numerical differentiations (such as finite difference) or the symbolic differentiations (expression manipulation). Baydin et al. [4] explains that “all numerical computations are ultimately compositions of a finite set of elementary operations for which derivatives are known, and combining the derivatives of the constituent operations through the chain rule gives the derivative of the overall composition.” This allows us to take the derivatives of any order of particular functions relatively easily. In particular, we use the Autograd in PyTorch package (Python library).

The reason that we make the second output is to approximate ${\partial }_{vv}{f}^{nn}(t,x,v;m,w,b)$ via the reduction of order technique. We approximate ${\partial }_{vv}{f}^{nn}(t,x,v;m,w,b)$ by

$${\partial }_v{h}^{nn}(t,x,v;m,w,b)$$

using the second output

$${\partial }_v{f}^{nn}(t,x,v;m,w,b)=h^{nn}(t,x,v;m,w,b)$$

contained in the loss term; see Section 2.3. It is also possible to get the value of ${\partial }_{vv}{f}^{nn}$ directly applying the AD method twice for $f^{nn}$. However, we use the AD method only once for $f^{nn}$ and $h^{nn}$ by adding the one more output neuron and adding the loss function for $h^{nn}$. This is because we realized that the modification reduces the computational costs dramatically than the direct one of using the second derivative. Also, the DNN architecture in Fig. 1 shares the same weights for the output f and the derivatives of f at the same time, and this allows us to design and modify the DNN corresponding to the purpose.

Fig. 1.

The DNN structure.

2.2. Grid points

To approximate the kinetic solution $f(t,x,v)$ via the Deep Learning algorithm, we make the data of grid points for each variable domain. We choose T as 5 or 10 for each boundary condition and each collision coefficient. We truncate the momentum space for the v variable as $[-20,20]$, make the grid points for v in $[-10,10]$ for the training, and assume that $f^{nn}(t,x,v;m,w,b)$ is 0 if $|v|>10$. More precisely, the grid points for the training are chosen uniformly as follows:

$${\{(t_i,x_j,v_k)\}}_{i,j,k}\in [0,T]\times [-1,1]\times [-10,10]\text{with}\mathrm{\Delta }t=0.01,\mathrm{\Delta }x=0.02,\mathrm{\Delta }v=0.2.$$

We use the grids

$${\{(t=0,x_j,v_k)\}}_{j,k}$$

for the initial condition and

$${\{(t_i,x=1\text{or}-1,v_k)\}}_{i,k}$$

for the boundary condition.

2.3. Loss functions

In the algorithm, the Adam optimizer finds the optimal parameters $w_{ji}^{(l+1)}$ and $b_j^{(l+1)}$ to minimize loss functions using the back-propagation method. Thus, we need to define loss functions for our 1-dimensional kinetic Fokker-Planck equation: $Los{s}_{GE}$ for the Fokker-Planck equation (1.2), $Los{s}_{IC}$ for the initial condition (1.3) and $Los{s}_{BC}$ for the boundary conditions defined as in Section 1.4.

Firstly, we define a loss function for the governing equation (1.2). We use the reduction-of-order technique for the second-order term as follows:

$$Los{s}_{GE}^1=\underset{(0,T)}{\int }dt\underset{(-1,1)}{\int }dx\underset{V}{\int }dv|{\partial }_t{f}^{nn}(t,x,v;m,w,b)+v{\partial }_x{f}^{nn}(t,x,v;m,w,b)-(\sigma {\partial }_v{h}^{nn}(t,x,v;m,w,b)+\beta {\partial }_v(v{f}^{nn})(t,x,v;m,w,b)){|}^2,Los{s}_{GE}^2=\underset{(0,T)}{\int }dt\underset{(-1,1)}{\int }dx\underset{V}{\int }dv|h^{nn}(t,x,v;m,w,b)-{\partial }_v{f}^{nn}(t,x,v;m,w,b){|}^2,$$

where $V\stackrel{\text{def}}{=}[-10,10]$. Then we define $Los{s}_{GE}$ as

$$Los{s}_{GE}=Los{s}_{GE}^1+Los{s}_{GE}^2\approx \frac{1}{N_{i,j,k}}\sum _{i,j,k}|{\partial }_t{f}^{nn}(t_i,x_j,v_k;m,w,b)+v{\partial }_x{f}^{nn}(t_i,x_j,v_k;m,w,b)-(\sigma {\partial }_v{h}^{nn}(t_i,x_j,v_k;m,w,b)+\beta {\partial }_v(v{f}^{nn})(t_i,x_j,v_k;m,w,b)){|}^2+|h^{nn}(t_i,x_j,v_k;m,w,b)-{\partial }_v{f}^{nn}(t_i,x_j,v_k;m,w,b){|}^2,$$ (2.1)

where $N_{i,j,k}$ is the number of grid points.

We now define the loss function for the initial condition via the use of the initial grid points as

$$Los{s}_{IC}=\underset{(-1,1)}{\int }dx\underset{V}{\int }dv{|f^{nn}(0,x,v)-f_0(x,v)|}^2\approx \frac{1}{N_{j,k}}\sum _{j,k}{|f^{nn}(0,x_j,v_k)-f_0(x_j,v_k)|}^2.$$ (2.2)

The loss function for the inflow boundary condition in Section 1.4 is defined as follows:

$$Los{s}_{BC}=\sum _{x\in \{-1,1\}}\underset{(0,T)}{\int }dt\underset{V}{\int }dv{|f^{nn}(t,x,v;m,w,b)-g(t,x,v;m,w,b)|}^2\approx \frac{1}{2N_{i,k}}\sum _{x\in \{-1,1\},i,k}{|f^{nn}(t_i,x,v_k;m,w,b)-g(t_i,x,v_k;m,w,b)|}^2.$$ (2.3)

For the other types of the boundary conditions from Section 1.4, we define the loss term similarly via altering $g(t,x,v;m,w,b)$ and $g(t_i,x,v_k;m,w,b)$.

One of the well-known a priori conservation law for the Fokker-Planck equation (1.2) for the specular, the periodic and the diffusive boundary conditions is the conservation of mass, in which we have that the $L_{x,v}^1$ moment of ${\Vert f^{nn}(t,\cdot ,\cdot ;m,w,b)\Vert }_{L_{x,v}^1}$ propagates in time. Therefore, the reduction of the loss term (2.1) would result in the reduction of the loss term that is defined for the conservation of mass. Therefore, without loss of generality, we add one more loss term with respect to the conservation of mass for more accurate analysis when impose the three boundary conditions for each $t_i$:

$$Los{s}_{\text{Mass}}^i={|\frac{d}{dt}\underset{(-1,1)}{\int }dx\underset{V}{\int }dv{f}^{nn}(t,x,v;m,w,b)|}_{t=t_i}^2.$$

Then we define the whole $Los{s}_{\text{Mass}}$ as

$$Los{s}_{\text{Mass}}=\sum _{i}Los{s}_{\text{Mass}}^i\approx \frac{1}{N_i}\sum _i{|\frac{1}{N_{j,k}}\frac{d}{dt}\sum _{j,k}{f}^{nn}(t,x_j,v_k;m,w,b)|}_{t=t_i}^2.$$ (2.4)

Finally, we define the total loss as

$$Los{s}_{Total}=Los{s}_{GE}+Los{s}_{IC}+Los{s}_{BC}$$ (2.5)

for the absorbing boundary (1.10) and the inflow boundary (1.11), and

$$Los{s}_{Total}=Los{s}_{GE}+Los{s}_{IC}+Los{s}_{BC}+Los{s}_{\text{Mass}}$$

for the rest of the boundary conditions of Section 1.4.

3. Theoretical discussions

3.1. On the convergence of DNN solutions to analytic solutions

In this section, we prove that there exists a sequence of weights such that the total sum of loss functions, defined later than (2.5), converges to 0. Sequentially, we also prove that neural networks equipped with such weights converge to an analytic solution. Throughout the section, we assume that the existence and the uniqueness of solutions for (1.2) and (1.3) with either (1.11) or (1.7) are a priori given. We first introduce the following definition and the theorem from [50] on the existence of the approximated neural network solution:

Definition 3.1 Li, [50] —

For a compact set K of ${\mathbb{R}}^n$ , we say $f\in {\hat{C}}^m(K)$ , $m\in {\mathbb{Z}}_{+}^n$ if there is an open Ω (depending on f) such that $K\subset \mathit{\Omega }$ and $f\in C^m(\mathit{\Omega })$ .

Theorem 3.2 Li, Theorem 2.1, [50] —

Let K be a compact subset of ${\mathbb{R}}^n$ , $n\ge 1$ , and $f\in {\hat{C}}^{m_1}(K)\cap {\hat{C}}^{m_2}(K)\cap \cdots {\hat{C}}^{m_q}(K)$ , where $m_i\in {\mathbb{Z}}_{+}^n$ for $1\le i\le q$ . Also, let $\overline{\sigma }$ be any non-polynomial function in $C^l(\mathbb{R})$ , where $l=\max \{|m_i|:1\le i\le q\}$ . Then for any $\epsilon >0$ , there is a network

$$f^{nn}(x)=\sum _{i=0}^{\nu }{c}_i\overline{\sigma }(〈w_i,x〉+b_i),x\in {\mathbb{R}}^n,$$

where $c_i\in \mathbb{R}$ , $w_i\in {\mathbb{R}}^n$ , and $b_i\in \mathbb{R}$ , $0\le i\le \nu$ such that

$${\Vert D^{k}f-D^k{f}^{nn}\Vert }_{L^{\infty }(K)}<\epsilon ,$$

for $k\in {\mathbb{Z}}_{+}^n$ , $k\le m_i$ , for some i, $1\le i\le q$ .

Remark 3.3

One may generalize the result above to the one with several hidden layers (see, [38] ). Also, we may assume that the architecture is assumed to have only one hidden layer; i.e., $L=2$ .

Now we introduce our first main theorem which states that a sequence of neural network solutions that makes the total loss term converge to zero exists if a ${\hat{C}}^{(1,1,2)}$ solution to the equation exists:

Theorem 3.4

Assume that the number of layers $L=2$ and that the solution f to (1.2) and (1.3) with either (1.11) or (1.7) belongs to ${\hat{C}}^{(1,1,2)}([0,T]\times [-1,1]\times V)$ , and the activation function $\overline{\sigma }(x)\in C^{(2,2,3)}([0,T]\times [-1,1]\times V)$ is non-polynomial. Then, there exists ${\{m_{[j]},w_{[j]},b_{[j]}\}}_{j=1}^{\infty }$ such that a sequence of the DNN solutions with $m_{[j]}$ nodes, denoted by ${\{f_j(t,x,v)=f^{nn}(t,x,v;m_{[j]},w_{[j]},b_{[j]})\}}_{j=1}^{\infty }$ satisfies 1

$$Los{s}_{Total}(f_j)\to 0\mathit{\text{as}}j\to \infty .$$ (3.1)

Proof

Let $ϵ>0$ be given. By Theorem 3.2, there exists a neural network

$$f_j(t,x,v)=\sum _{i=1}^{m_{[j],1}}{w}_{[j],1i}^{(2)}\overline{\sigma }((w_{[j],i1}^{(1)},w_{[j],i2}^{(1)},w_{[j],i3}^{(1)})\cdot (t,x,v)+b_{[j],i}^{(1)})+b_{[j],i}^{(2)},$$

such that ${\Vert D^{\underset{\_}{k}}f-D^{\underset{\_}{k}}{f}_j\Vert }_{L^{\infty }([0,T]\times [-1,1]\times V)}<ϵ$, where $\underset{\_}{k}\le (1,1,2)$ is a multi-index. By integrating

$$|{\partial }_t{f}_j+v{\partial }_x{f}_j-{\partial }_v(\sigma {\partial }_v+\beta v)f_j{|}^2$$

and

$$|f_j(0,\cdot ,\cdot )-f_0{|}^2$$

over $[0,T]\times [-1,1]\times V\text{, }[-1,1]\times V$, respectively, we obtain that the loss terms (2.1) and (2.2) are bounded by $(1+\mu (V)+\sigma +\beta \mu (V))T^2\mu {(V)}^2\epsilon$ and $2\mu {(V)}^2\epsilon$, where $\mu (\cdot )$ is the Lebesgue measure on $\mathbb{R}$. Now we assume that the boundary condition satisfies (1.11). Then we note that all the boundary values are bounded by a supremum of interior values, since f and $f_j$ are sufficiently smooth. That is,

$${\Vert f_j-g\Vert }_{L^2({\gamma }_{T,V}^{-})}^2={\Vert f_j-f\Vert }_{L^2({\gamma }_{T,V}^{-})}^2\le 2T\mu (V){\Vert f_j-f\Vert }_{L^{\infty }({\gamma }_{T,V}^{-})}^2\le 2T\mu (V){\Vert f_j-f\Vert }_{L^{\infty }([0,T]\times [-1,1]\times V)}^2\le 2T\mu (V){\epsilon }^2,$$

where ${\gamma }_{T,V}^{+}$ and ${\gamma }_{T,V}^{-}$ are defined as

$${\gamma }_{T,V}^{\pm }\stackrel{\text{def}}{=}[0,T]\times {\gamma }_{\pm ,V},$$ (3.2)

where ${\gamma }_{\pm ,V}$ denote the boundaries sets ${\gamma }_{+}$ and ${\gamma }_{-}$ with the v domain truncated to V. This completes the proof by setting $\epsilon ={\epsilon }_j=\frac{1}{j}$. □

Remark 3.5

The assumption $f\in {\hat{C}}^{(1,1,2)}([0,T]\times [-1,1]\times V)$ can be replaced by a general Sobolev space, since the functions in a Sobolev space can be approximated by the continuous functions on a compact set.

We also remark that Theorem 3.4 provides us that we can find the weight of the neural network that reduces the error function as much as we want. However, this does not guarantee that the neural network could converge to the solution of the original equation when the loss function converges to zero, though we will also discuss how to find the weights in the forthcoming sections. Due to the reason, we introduce our second theorem, Theorem 3.6, which shows that the neural network architecture converges to an analytic solution in a suitable function space when the weights of the neural networks minimize $Los{s}_{Total}$. We prove it under the following additional condition at the boundary that is consistent to the truncation of the domain in v variable introduced in Section 2.2:

$$|{\partial }_v^{k}f(t,x,v)-{\partial }_v^k{f}_j(t,x,v)|\le \epsilon \text{ if }v\in \partial V,$$ (3.3)

for some sufficiently small $\epsilon >0$ and $k=0,1$.

Theorem 3.6

Let ${\{m_{[j]},w_{[j]},b_{[j]}\}}_{j=1}^{\infty }$ be a sequence minimizing the $Los{s}_{Total}(f_j)$ that also satisfies $Los{s}_{Total}(f_j)<{\epsilon }_j$ as in Theorem 3.4 and let $f_j$ and f satisfy (3.3) . Then, $Los{s}_{Total}(f_j)\to 0$ implies

$${\Vert f_j(\cdot ,\cdot ,\cdot ;m_{[j]},w_{[j]},b_{[j]})-f\Vert }_{L^{\infty }([0,T];L^2([-1,1]\times V))}\le C(\sigma ,\beta ){\epsilon }^2,$$ (3.4)

where f is a solution to (1.2) and (1.3) with either (1.11) or (1.7) , and $C(\sigma ,\beta )$ is a non-negative constant depending only on σ and β.

Proof

Motivated by [17], we define a transform $\overline{u}(t,x,v)$ of a function $u(t,x,v)$ as follows:

$$\overline{u}(t,x,v)=e^{-(\lambda +\beta )t}u(t,x,e^{-\beta t}v)$$

where λ is any non-negative constant that could control the convergence rate. Then the transformed function $\overline{f}$ satisfies

$${\partial }_t\overline{f}+e^{-\beta t}(v\cdot {\partial }_x)\overline{f}-\sigma e^{2\beta t}{\partial }_v^2\overline{f}+\lambda \overline{f}=0.$$

We now consider the following set of equations on the difference between $\overline{f}$ and $\overline{f_j}$ for each fixed j as

$$[{\partial }_t+e^{-\beta t}(v\cdot {\partial }_x)-\sigma e^{2\beta t}{\partial }_v^2+\lambda ]\{\overline{f}-\overline{f_j}\}=d_{ge,j}(t,x,v),\text{ for }(t,x,v)\in [0,T]\times [-1,1]\times e^{\beta t}V,$$ (3.5)

$$\overline{f}(0,x,v)-\overline{f_j}(0,x,v)=d_{ic,j}(x,v),\text{ for }(x,v)\in [-1,1]\times e^{\beta t}V,$$ (3.6)

$$\overline{f}(t,x,v)-\overline{f_j}(t,x,v)=d_{bc,j}(t,x,v),\text{ for }(t,x,v)\in {\gamma }_{T,e^{\beta t}V}^{-},$$ (3.7)

where the interval $e^{\beta t}V$ is defined as

$$e^{\beta t}V\stackrel{\text{def}}{=}[-10e^{\beta t},10e^{\beta t}],$$

and ${\gamma }_{T,e^{\beta t}V}^{+}$ and ${\gamma }_{T,e^{\beta t}V}^{-}$ are defined as in (3.2). Here, we define

$$d_{ge,j}(t,x,v)\stackrel{\text{def}}{=}-[{\partial }_t+e^{-\beta t}(v\cdot {\partial }_x)-\sigma e^{2\beta t}{\partial }_v^2+\lambda ]\overline{f_j},$$

$$d_{ic,j}(x,v)\stackrel{\text{def}}{=}{\overline{f}}_0(x,v)-\overline{f_j}(0,x,v),$$

and

$$d_{bc,j}\stackrel{\text{def}}{=}\overline{g}(t,x,v)-\overline{f_j}(t,x,v),$$

for the inflow boundary condition (1.11) and

$$d_{bc,j}=\overline{f_j}(t,x,-v)-\overline{f_j}(t,x,v),$$

for the specular boundary condition (1.7) instead. Then we derive the inequality below by multiplying $2(\overline{f}-\overline{f_j})$ onto (3.5) and integrating it over $[-1,1]\times e^{\beta t}V$ as let2

$$\underset{-1}{\overset{1}{\int }}\underset{-10e^{\beta t}}{\overset{10e^{\beta t}}{\int }}\frac{\partial }{\partial t}{(\overline{f}-\overline{f_j})}^2(t,x,v)dvdx-2\sigma e^{2\beta t}〈{\partial }_v^2(\overline{f}-\overline{f_j}),(\overline{f}-\overline{f_j})〉+\underset{{\gamma }_{+,e^{\beta t}V}}{\int }{(\overline{f}-{\overline{f}}_j)}^{2}d\gamma +2\lambda {\Vert \overline{f}-\overline{f_j}\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2=\underset{{\gamma }_{-,e^{\beta t}V}}{\int }{d}_{bc,j}^{2}d\gamma +2〈d_{ge,j},(\overline{f}-\overline{f_j})〉,$$ (3.8)

where $〈\cdot ,\cdot 〉$ denotes the standard inner product on $L^2([-1,1]\times e^{\beta t}V)$. On the left-hand side of (3.8), we note that

$$\underset{{\gamma }_{+,e^{\beta t}V}}{\int }|\overline{f}-\overline{f_j}{|}^{2}d\gamma \ge 0,$$

$$\frac{d}{dt}{\Vert (\overline{f}-\overline{f_j})(t,\cdot ,\cdot )\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2=\underset{-1}{\overset{1}{\int }}\underset{-10e^{\beta t}}{\overset{10e^{\beta t}}{\int }}\frac{\partial }{\partial t}{(\overline{f}-\overline{f_j})}^2(t,x,v)dvdx+\underset{\stackrel{\text{def}}{=}{B}_1(t)}{\underbrace{10\beta e^{\beta t}({\Vert (\overline{f}-\overline{f_j})(t,\cdot ,10e^{\beta t})\Vert }_{L^2([-1,1])}^2+{\Vert (\overline{f}-\overline{f_j})(t,\cdot ,-10e^{\beta t})\Vert }_{L^2([-1,1])}^2)}},$$

by the Leibniz rule and

$$2\sigma e^{2\beta t}〈{\partial }_v^2(\overline{f}-\overline{f_j}),(\overline{f}-\overline{f_j})〉=-2\sigma e^{2\beta t}{\Vert {\partial }_v(\overline{f}-\overline{f_j})\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2+2\sigma e^{2\beta t}\underset{-1}{\overset{1}{\int }}{\partial }_v(\overline{f}-\overline{f_j})(\overline{f}-\overline{f_j})(t,\cdot ,10e^{\beta t})dx-2\sigma e^{2\beta t}\underset{-1}{\overset{1}{\int }}{\partial }_v(\overline{f}-\overline{f_j})(\overline{f}-\overline{f_j})(t,\cdot ,-10e^{\beta t})dx\stackrel{\text{def}}{=}-2\sigma e^{2\beta t}{\Vert {\partial }_v(\overline{f}-\overline{f_j})\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2+B_2(t).$$

Therefore, we reduce (3.8) to

$$\frac{d}{dt}\stackrel{Y(t)\stackrel{\text{def}}{=}}{\overbrace{{\Vert \overline{f}-\overline{f_j}\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2}}+2\lambda {\Vert \overline{f}-\overline{f_j}\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2\le {\Vert \overline{f}-\overline{f_j}\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2+\underset{\stackrel{\text{def}}{=}L(t)}{\underbrace{2\underset{{\gamma }_{-,e^{\beta t}V}}{\int }{d}_{bc,j}^{2}d\gamma +{\Vert d_{ge,j}\Vert }_{L^2([-1,1]\times e^{\beta t}V)}^2}}+B_1(t)+B_2(t).$$ (3.9)

Multiplying (3.9) by $e^{(2\lambda -1)t}$ and integrating it over $[0,t]$ for $t<T$, we have

$$Y(t)\le Y(0)e^{-(2\lambda -1)t}+e^{-(2\lambda -1)t}\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}(L(s)+B_1(s)+B_2(s))ds$$ (3.10)

Finally, we recall that $Y(t)=e^{-(2\lambda +\beta )t}{\Vert f-f_j\Vert }_{L^2([-1,1]\times V)}^2$, $Y(0)=Los{s}_{IC}$, and

$$\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}L(s)ds=\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}[2e^{-(2\lambda +\beta )s}\underset{{\gamma }_{-},V}{\int }{(g-f_j)}^{2}d\gamma +e^{-(2\lambda +\beta )s}{\Vert L{f}_j\Vert }_{L^2((-1,1)\times V)}^2]ds=\underset{0}{\overset{t}{\int }}[2e^{-(\beta +1)s}\underset{{\gamma }_{-},V}{\int }{(g-f_j)}^{2}d\gamma +e^{-(\beta +1)s}{\Vert L{f}_j\Vert }_{L^2((-1,1)\times V)}^2]ds\le 2(Los{s}_{BC}+Los{s}_{GE}),$$

where $L{f}_j=[{\partial }_t+v{\partial }_x-{\partial }_v(\sigma {\partial }_v+\beta v)]f_j$ is the Fokker-Planck operator. Moreover, under the assumption on (3.3), we have

$$\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}{B}_1(s)ds\le 40\beta {\epsilon }^2\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}{e}^{-(2\lambda +\beta )s}ds\le C_1(\beta ){\epsilon }^2,$$

$$\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}{B}_2(s)ds\le 4\sigma {\epsilon }^2\underset{0}{\overset{t}{\int }}{e}^{(2\lambda -1)s}{e}^{-(2\lambda +\beta )s}ds\le C_2(\sigma ,\beta ){\epsilon }^2.$$

Therefore, (3.10) and the inverse transform from $\overline{f}$ to f imply that

$${\Vert f-f_j\Vert }_{L^2([-1,1]\times V)}^2\le 2e^{(2\beta +1)t}(Los{s}_{IC}+Los{s}_{BC}+Los{s}_{GE}+C(\sigma ,\beta ){\epsilon }^2),$$

where $C(\sigma ,\beta )=C_1(\beta )+C_2(\sigma ,\beta )$. Since $t\in [0,T]$ and $Los{s}_{Total}(f_j)\to 0$, this completes the proof of Theorem 3.6. □

3.2. On the convergence rate of the kinetic energy

In this section, we would like to record the theoretical background on the convergence rate of the kinetic energy of the system, which will be observed via the neural network simulations as well in Section 4. This will be done only for the specular boundary condition (1.7) in this section. We define the kinetic energy functional of the solution f to the Fokker-Planck equation as follows:

$$\text{KE}(t)\stackrel{\text{def}}{=}\frac{1}{2}\underset{(-1,1)\times \mathbb{R}}{\int }dxdv|v{|}^{2}f(t,x,v).$$

Then we can rewrite the balance of kinetic energy (1.15) as follows:

$$\frac{d}{dt}\text{KE}(t)=-\frac{1}{2}\underset{\{-1,1\}\times \mathbb{R}}{\int }dxdv(v\cdot n_x)|v{|}^{2}f(t,x,v)-2\beta \text{KE}(t)+\sigma M,$$ (3.11)

where $M={\Vert f_0(\cdot ,\cdot )\Vert }_{L_{x,v}^1}$. Then the first term on the right-hand side of (3.11) is equal to

$$-\frac{1}{2}\underset{\{-1,1\}\times \mathbb{R}}{\int }dxdv(v\cdot n_x)|v{|}^{2}f(t,x,v)=-\frac{1}{2}(\underset{-\infty }{\overset{\infty }{\int }}dv(-v)|v{|}^{2}f(t,x=-1,v)+\underset{-\infty }{\overset{\infty }{\int }}dvv|v{|}^{2}f(t,x=1,v)).$$ (3.12)

Under the specular boundary condition (1.7), both two terms on the right-hand side of (3.12) is 0, since

$$\underset{-\infty }{\overset{\infty }{\int }}dv(-v)|v{|}^{2}f(t,x=-1,v)=\underset{-\infty }{\overset{0}{\int }}dv(-v)|v{|}^{2}f(t,x=-1,v)+\underset{0}{\overset{\infty }{\int }}dv(-v)|v{|}^{2}f(t,x=-1,v)=\underset{0}{\overset{\infty }{\int }}dvv|v{|}^{2}f(t,x=-1,-v)+\underset{0}{\overset{\infty }{\int }}dv(-v)|v{|}^{2}f(t,x=-1,v)=\underset{0}{\overset{\infty }{\int }}dvv|v{|}^{2}f(t,x=-1,v)-\underset{0}{\overset{\infty }{\int }}dvv|v{|}^{2}f(t,x=-1,v)=0,$$

and ${\int }_{-\infty }^{\infty }dvv|v{|}^{2}f(t,x=1,v)$ is also 0 in a similar manner. Therefore, we can rewrite (3.11) as

$$\frac{d}{dt}\text{KE}(t)+2\beta \text{KE}(t)=\sigma M.$$ (3.13)

Therefore, we obtain

$$\text{KE}(t)e^{2\beta t}=\frac{\sigma M}{2\beta }{e}^{2\beta t}+C$$

by multiplying (3.13) by $e^{2\beta t}$ and integrating over $[0,t]$, where C is a constant. We can get the closed-form of the kinetic energy under the specular boundary conditions for the kinetic Fokker-Planck equation as

$$\text{KE}(t)=\frac{\sigma M}{2\beta }+C{e}^{-2\beta t}.$$

Now, for the total kinetic energy of the approximated solution $f^{nn}(t,x,v;m,w,b)$ which is defined as

$${\text{KE}}^{nn}(t;m,w,b)\stackrel{\text{def}}{=}\frac{1}{2}\underset{(-1,1)\times V}{\int }dxdv|v{|}^2{f}^{nn}(t,x,v;m,w,b),$$

the kinetic energy ${\text{KE}}^{nn}(t;m,w,b)$ satisfies

$${\text{KE}}^{nn}(t;m,w,b)\le \text{KE}(t)=\frac{\sigma M}{2\beta }+C{e}^{-2\beta t},$$ (3.14)

since the truncated domain of velocity variable $V=[-10,10]$ is contained in $\mathbb{R}$. This shows that the kinetic energy of the neural network solution converges faster to the steady-state value as the value of the coefficient β gets larger.

4. Neural network simulations

In this section, we introduce the results of our neural network simulations for the DNN solution $f^{nn}(t,x,v;m,w,b)$ in three different ways. We first simulate our neural network algorithms for the varied boundary conditions, a given initial condition and the fixed values of σ, β coefficients. In the cases of the specular reflection, the diffusive reflection, and the periodic boundaries, we expect that the Lyapunov functional (also called as the free energy or the relative entropy) satisfies ${\eta }^{\prime }\le 0$, which is a manifestation of the second law of thermodynamics. Then, we alter the initial conditions for the specular boundary condition and for the fixed values of σ, β coefficients. Lastly, we also obtain the results via altering the coefficients σ and β for a given initial condition and the specular boundary condition.

We analyze our neural network solution $f^{nn}(t,x,v;m,w,b)$ via computing the pointwise values at each grid-point, observing the $L^{\infty }$ norm of the solution and via computing the physical quantities of the total mass, the kinetic energy, the entropy, and the free energy. We compute these quantities via the approximations of the integration by the Riemann sum on the grid points, similarly to the previous subsection. For example, $L^1$ norm of $f^{nn}(t,x,v;m,w,b)$ for each time t can be approximated as

$$\underset{(-1,1)}{\int }\underset{\mathbb{R}}{\int }|f(t,x,v)|dxdv\approx \frac{1}{N_{j,k}}\sum _{j,k}|f(t,x_j,v_k)|.$$

We also compare our results with the existing theoretical results (cf. [20]). Especially for the specular (1.7) and the diffusive (1.8) boundary conditions which conserve the total mass, Bonilla-Carrillo-Soler [20] gives the form of the steady-state as given in (1.12). The expected limiting values of kinetic energy, entropy, and free energy based on the global equilibrium [68] are defined as follows:

$${\text{KE}}_{\infty }=\frac{\sigma M}{2\beta },$$ (4.1)

$${\text{Ent}}_{\infty }=-M\log \left(\frac{M}{C{(2\pi \frac{\sigma }{\beta })}^{0.5}}\right)+\frac{1}{2}M$$ (4.2)

$${\text{FE}}_{\infty }={\text{KE}}_{\infty }-\frac{\sigma }{\beta }{\text{Ent}}_{\infty },$$ (4.3)

where $M={\Vert f_0(\cdot ,\cdot )\Vert }_{L_{x,v}^1}$ and $C=|\mathrm{\Omega }|=2$ for our case. Also, we calculate the entropy as the integral of $-f\log (f+{10}^{-10})$ to prevent from the divergence when $f(t,x,v)=0$.

We predict the pointwise values of the distribution functions $f^{nn}$ for each spatial variable x in $[-1,1]$ and the velocity variable v in $[-10,10]$ corresponding to the varied values of time variable t in $[0,T]$. Namely, we observe the long-time behavior of neural network solution $f^{nn}(t,x,v;m,w,b)$ and check if the solution converges to Maxwellian (1.12) for the reflection-type boundaries. Note that minimizing the total loss does not guarantee a positivity of $f^{nn}$. Therefore, we also truncate $f^{nn}$ if $f^{nn}$ is sufficiently small (< 0.005).

4.1. Results by varying the boundary conditions

In this section, we impose different types of the boundary conditions that we introduced in Section 1.4. Throughout this section, we set the values of the coefficients σ and β to be 1, and we consider the following cake-shaped initial condition for the varied boundary conditions:

$$f(0,x,v)=f_0(x,v)=\{\begin{array}{cc}1,\hfill & \text{ if }(x,v)\in (-0.9,0.9)\times (-2,2)\text{,}\hfill \\ 0,\hfill & \text{ otherwise}.\hfill \end{array}$$ (4.4)

We would first like to introduce the behavior of the approximated solutions under the absorbing boundary condition in the following section.

4.1.1. The absorbing boundary condition

By imposing the absorbing boundary condition (1.10), we would like to understand the dynamics of particles whose momenta vanish when they collide against the boundary. Fig. 2 shows the $L^{\infty }$ norm, total mass, total kinetic energy, and the entropy of our neural network solution $f^{nn}(t,x,v;m,w,b)$ under the absorbing boundary condition. It has been shown in [40] that the $L^{\infty }$ norm and the total mass of the solutions $f(t,x,v)$ to the kinetic Fokker-Planck equation decay exponentially as the time variable t goes to infinity when the friction coefficient β is zero. In our case with the diffusion coefficient $\sigma =1$ and the friction coefficient $\beta =1$, the total mass decays in time as in Fig. 2, while the $L^{\infty }$ norm decays after an initial mixing and climbing.

Fig. 2.

The time-asymptotic behavior of the L^∞ norm and the macroscopic quantities of fⁿⁿ(t,x,v;m,w,b) with the absorbing boundary condition. All quantities converge to zero as all the particles vanish once they reach the boundary.

Also, we can observe that our solution $f^{nn}(t,x,v;m,w,b)$ converges pointwisely to 0, as shown in Fig. 3 . We remark that Fig. 3 shows each pointwise value of the neural network solution, and this definitely gives more information than just $L^1$ and $L^{\infty }$ values of the solution.

Fig. 3.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the absorbing boundary case. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

4.1.2. Inflow boundary condition

We now move onto the next boundary condition, the inflow boundary condition, with which we consider the situation that each boundary bounces particles in a given rate of the given function $g(t,x,v)$. We compare the pointwise values of the neural network solutions under three different inflow boundary conditions. The three different inflow boundary conditions are:

$$f(t,x,v){|}_{{\gamma }_{-}}=g(t,x,v)=\frac{1}{2}{\mathbb{1}}_{|v|\le 5},\text{for}x=-1\text{ and }1,$$ (4.5)

$$f(t,x,v){|}_{{\gamma }_{-}}=g(t,x,v)=\{\begin{array}{cc}\frac{1}{10}{\mathbb{1}}_{|v|\le 5},\hfill & \text{ if }x=-1\hfill \\ \frac{9}{10}{\mathbb{1}}_{|v|\le 5},\hfill & \text{ if }x=1,\hfill \end{array}$$ (4.6)

and

$$f(t,x,v){|}_{{\gamma }_{-}}=g(t,x,v)=\frac{1}{2}{e}^{-t}{\mathbb{1}}_{|v|\le 5},\text{for}x=-1\text{ and }1,$$ (4.7)

where ${\mathbb{1}}_{\mathbb{S}}$ is the characteristic function on a set $\mathbb{S}$.

In Fig. 4 , the plot for the third type of inflow boundary condition (4.7) has the similar pointwise values of $f^{nn}(t,x,v;m,w,b)$ to that of the first inflow boundary condition (4.5), but it behaves like or converges to that of the absorbing boundary condition over time, as shown in the cyan-blue line. This converges pointwisely to 0 after 10 time grids. On the other hand, we can observe that the other two types (4.5) and (4.6) converge to some distorted Maxwellian-like shapes at each spatial position. Also, we can observe the smoothing effect of the kinetic Fokker-Planck equation in the interior domain at $x=-0.5,0$, and 0.5. At the boundaries $x=-1$ and $x=1$, we observe the jump discontinuities of the momentum derivatives as expected due to the fixed inflow profiles.

Fig. 4.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the different inflow boundary cases in different colors as shown in the legend. x = ±1 stand for the boundary points, and x = ±0.5 and =0 are the points away from the boundary. (For interpretation of the colors in the figure(s), the reader is referred to the web version of this article.)

4.1.3. The three boundary conditions which conserve the total mass

Now we would like to impose other types of the boundary conditions under which the total mass of the system is conserved. Namely, we will look at the dynamics of the particles under the specular boundary condition (1.7), the periodic boundary condition (1.9), and the diffusive boundary condition (1.8).

We first remark that we obtain the similar behavior of the $L^{\infty }$ and $L^1$ norms of the solution in the three types of the boundary conditions. Though the convergence rates are slightly different, we could possibly say that the kinetic energies and the entropies of the three different cases converge to the same values in the red dotted line in Fig. 5 .

Fig. 5.

The time-asymptotic behaviors of the L^∞ norms and the macroscopic quantities of fⁿⁿ(t,x,v;m,w,b) with the specular, the periodic, and the diffusive boundary conditions, which conserve the total mass. The steady-state values of the kinetic energy (4.1) and the entropy (4.3) are also given via the red-dotted lines. It is notable that the free energy (Lyapunov functional) is monotonically decreasing.

In the perspective of the pointwise values of the neural network solutions, Fig. 6 shows that the three cases converge pointwisely to the same steady-state solution; i.e., they converge to the global Maxwellian solution (1.12). It is remarkable that the shape of the convergence to the global Maxwellian is slightly different in the diffusive boundary case from the other two conditions at the boundary point $x=-1$. At the boundary $x=-1$, the periodic and the specular cases converge to the Maxwellian via the superposition of two waves from the left- and the right-hand sides at the same rate and hence via the creation of M-shaped plots. On the other hand, at the boundary $x=-1$, the solution in the case of the diffusive boundary condition converges to the global Maxwellian from the left to the right, as the diffusive boundary condition at $x=-1$ emits the Maxwellian-shaped values from the left to the right.

Fig. 6.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the specular, the periodic, and the diffusive boundary cases in different colors as shown in the legend. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

4.2. Results by varying the initial conditions

In this section, we vary the initial condition for the fixed coefficients $\sigma =\beta =1$ and the specular boundary condition. We impose the initial conditions as

$$f(0,x,v)=f_0(x,v)=\{\begin{array}{cc}\frac{1}{25}{v}^2,\hfill & \text{ if }x\in (-0.9,0.9)\text{ and }v\in (-5,5)\hfill \\ 0,\hfill & \text{ otherwise},\hfill \end{array}$$ (4.8)

and

$$f(0,x,v)=f_0(x,v)=\{\begin{array}{cc}\sin (\frac{1}{v^2}),\hfill & \text{if }x\in (-0.9,0.9)\text{ and }v\in (-10,10)\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$ (4.9)

In other words, we impose a M-shaped initial condition with the values compiled at large values in v and a highly-oscillating initial condition near $v=0$.

Fig. 7 shows the pointwise values of $f^{nn}(t,x,v;m,w,b)$ with the two initial conditions. We first note that the M-shaped distribution becomes the Maxwellian over time. Though the initial condition (4.9) is highly singular in its derivatives, Fig. 7 shows that $f^{nn}(t,x,v;m,w,b)$ is being regularized and converges to the Maxwellian over time. We remark that the two initial conditions result in two different Maxwellians and this difference comes from the different total masses of the two initial conditions.

Fig. 7.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the specular boundary case with the special conditions drawn in different colors as shown in the legend. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

4.3. Results by varying the diffusion and the friction coefficients

In this section, we alter the two coefficients σ and β of the Fokker-Planck equation; the coefficients σ and β are related to the diffusion and the friction rates, respectively. Throughout the section, we fix the initial condition (4.4) and the specular boundary condition (1.7).

We mention that, throughout the section, we plot the values of the kinetic energy (4.1) and the entropy (4.3) of the global Maxwellian solution for each value of σ and β in the red dotted lines. In the following three sub-sections, we alter the values of β with a fixed σ, alter the values of σ with a fixed β, and alter both σ and β by keeping the ratio $\frac{\sigma }{\beta }$ being the same.

4.3.1. Different values of β

Throughout the section, we set $\sigma =1$ and let $\beta =2,1,0.5$, and 0.25. Fig. 8 shows that the kinetic energy and the entropy converge to the values of the red dotted lines of the expected steady-states for each different value of β. For example, the value of the kinetic energy for the steady-state with the coefficients $\sigma =1$ and $\beta =0.5$ is $K{E}_{\infty }=\frac{\sigma M}{2\beta }=7.2$ by (4.1), and the cyan-blue colored graph in the third plot of Fig. 8 converges to the red dotted line of the value 7.2. In addition, the graph for the $L^{\infty }$ norms shows that the $L^{\infty }$ values after a sufficiently large time become larger as β gets larger and this agrees the expected $L^{\infty }$ values of the steady-state by (1.12).

Fig. 8.

The time-asymptotic behaviors of the L^∞ norms and the macroscopic quantities of fⁿⁿ(t,x,v;m,w,b) for the specular boundary case as β varies with the fixed σ = 1. The steady-state values of the kinetic energy (4.1) and the entropy (4.3) are also given via the red-dotted lines for each different β. It is notable that the free energy (Lyapunov functional) is monotonically decreasing.

We remark that the rate of the convergence of the kinetic energy and the entropy for each different β depends on the values of β; the solution $f^{nn}(t,x,v;m,w,b)$ with a larger value of the friction coefficient β converged rapidly. This is also consistent to our theoretical supports provided in Section 3.2.

Fig. 9 shows that the pointwise values of $f^{nn}(t,x,v;m,w,b)$ approach the different steady-state solutions. Theoretically, the steady-state solution with the smaller friction coefficient β makes the larger variance in the Maxwellian (1.12) with the same mean 0, and Fig. 9 agrees with the theory.

Fig. 9.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the specular boundary case, where σ is fixed to be =1 and β varies as shown in different colors as in the legend. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

4.3.2. Different values of σ

In this section, we show the result with the fixed $\beta =1$ and the varied σ coefficients of $\sigma =2,1,0.5,0.25$. Fig. 10 shows that the physical quantities converge to those of the steady-state. Theoretically, the steady-state solution with the smaller diffusion coefficient σ makes the smaller variance in the Maxwellian (1.12) with the same mean 0, and Fig. 11 agrees with the theory.

Fig. 10.

The time-asymptotic behaviors of the L^∞ norms and the macroscopic quantities of fⁿⁿ(t,x,v;m,w,b) for the specular boundary case as σ varies with the fixed β = 1. The steady-state values of the kinetic energy (4.1) and the entropy (4.3) are also given via the red-dotted lines for each different σ. It is notable that the free energy (Lyapunov functional) is monotonically decreasing.

Fig. 11.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the specular boundary case, where β is fixed to be =1 and σ varies as shown in different colors as in the legend. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

4.3.3. Different values of σ and β in the same ratio $\frac{\sigma }{\beta }$

In this section, we vary both coefficients by keeping the ratio $\frac{\sigma }{\beta }$ the same. We set $\frac{\sigma }{\beta }=\frac{1}{2}$ and vary the coefficients σ and β as $\sigma =1,0.5,0.25,0.05$ and $\beta =2\sigma$. The convergence to the same physical quantities of the same steady-state in Fig. 12 agrees with the theory, as the steady-state (1.12) is determined by the fraction of the coefficient $\frac{\sigma }{\beta }$. The figure shows that the coefficient $\sigma =0.05$ is too small to make the quantities converge within 5 time-grids. The difference between the four cases is how fast the kinetic energy and the entropy of DNN solutions converge to those of the steady-state. A larger value of β results in the faster convergence though the fractions of the coefficients $\frac{\sigma }{\beta }$ are the same. Fig. 13 shows the different rate of the convergence to the steady-state; i.e., the four graphs in each plot converge to the same maxwellian but at different rates.

Fig. 12.

The time-asymptotic behaviors of the L^∞ norms and the macroscopic quantities of fⁿⁿ(t,x,v;m,w,b) for the specular boundary case as both β and σ vary by keeping the ratio $\frac{\sigma }{\beta }=\frac{1}{2}$. The steady-state values of the kinetic energy (4.1) and the entropy (4.3) are also given via the red-dotted lines. It is notable that the free energy (Lyapunov functional) is monotonically decreasing.

Fig. 13.

The pointwise values of fⁿⁿ(t,x,v;m,w,b) as t varies at each x's for the specular boundary case, where both σ and β vary with the ratio fixed as $\frac{\sigma }{\beta }=\frac{1}{2}$. The plots are shown in different colors as in the legend. x = −1 stands for the boundary point, and x = −0.5 or =0 are the points away from the boundary. We omit the graphs for x > 0 as we have obtained the symmetric graphs to x < 0.

5. Conclusion

This paper presents an approximation of the 1D Fokker-Planck equation solution with the inflow-type or the reflection-type boundary conditions. We first provide a proof for the existence of a sequence of DNN solutions such that the total loss, defined in terms of the difference between the ${\hat{C}}^{(1,1,2)}$-type solutions and the neural network solutions, converges to zero. We then provide a proof for the $L_{x,v}^2$ convergence of the sequence of neural network solutions to the actual ${\hat{C}}^{(1,1,2)}$-type solutions to the original Fokker-Planck equation in a bounded interval, in the case that the total sum of the loss terms of our own goes to zero. In other words, it has been shown that we can reduce the predefined loss function via the appropriate selection of DNN model parameters. Also, via choosing the model parameters that reduce the loss function as desired, it has been shown that the neural network solutions converge to the analytic solution by obtaining an $L^2$ energy estimate based on the adaptation of the transformation of the function in the sense of [17].

For the neural network simulations, our DNN algorithm uses the library PyTorch and the hyper-tangent tanh activation function between each layer. Regarding the optimization algorithm, we use Adam optimizer which is an extended algorithm of the stochastic gradient descent in the process of minimizing the actual loss in the case of the reflection-type boundary conditions. The reduction of the order and the weight sharing between $f^{nn}$ and ${\partial }_v{f}^{nn}$ dramatically reduced the learning cost. We remark that we also increased the learning efficiency of the model by adding the mass conservation law to the total loss function. In addition, we have numerically confirmed several theoretical predictions on the asymptotic behaviors of the solutions to the equation and the entropy production; i.e., we were able to provide the numerical simulations on the pointwise convergence of the neural network solutions to the global Maxwellian for varied types of the boundary conditions and the varied coefficients for the diffusion and the friction and observed that the free energy (the relative entropy) is monotonically decreasing.

Our method finds the parameters (weights and biases) which minimize the total loss function and the minimizer converges to an analytic solution. Also, we adopted an optimization method used for the practical exercise (a real-world problem) in the machine-learning community, but it does not provide an explicit way of finding the minimizer. Moreover, in the aspect of the optimization, it is difficult to find the exact global minimum due to the intrinsic limitation of the gradient-descent-based approaches though the number of parameters is small. More precisely, we were able to find that the minimized total-losses starting from different initial parameters have the similar scales of $\simeq {10}^{-3}$, and it is possible that different solutions that produce the minimized total-losses in the similar scale can be obtained.

In general, we think that the numerical schemes are more useful in the sense of the computational efficiency and that they could treat well-known error bound analysis very well. Nevertheless, we would like to remark that the DNN approach is a function-approximation approach rather than a numerical scheme, and hence it is not necessary to require some additional conditions such as the stability conditions. Also, we could directly compute the derivatives of a DNN solution via the AD, since the convergence criterion is a function-norm up to the regularity of the analytic solutions. In this sense, we can say that a DNN solution contains the mesh-free information on the geometries of the analytic solutions as well, and it is useful in computing some physical quantities. However, this method would require some guarantees on the existence and the uniqueness of the analytic solutions so that a DNN solution matches the analytic solution well.

The possible extensions of our work for the future include the generalizations to other various types of the kinetic equations. These include the multi-dimensional (fractional) Fokker-Planck equation, the coupled Vlasov equation such as the Vlasov-Poisson or the Vlasov-Maxwell systems. We expect that our work can further be extended to the generalized situations listed above via the adaptation of the sampling of random grids to our learning algorithm so the total cost for the learning process of the DNN algorithm is being effectively reduced while the total loss function is being properly reduced at the same time. Also, our approach could apply to the parameter-optimization problems that arise from the real world. For example, Jo et al. [45] computes the time-varying system parameters of the SIR model using the COVID-19 data. They considered the system parameters as DNN and added an observation loss term in the loss function. Therefore, the above method can enable us to calculate the model solution and the parameters at the same time.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.