Solving Fredholm Integral Equations Using Deep Learning

Abstract

The aim of this paper is to provide a deep learning based method that can solve high-dimensional Fredholm integral equations. A deep residual neural network is constructed at a fixed number of collocation points selected randomly in the integration domain. The loss function of the deep residual neural network is defined as a linear least-square problem using the integral equation at the collocation points in the training set. The training iteration is done for the same set of parameters for different training sets. The numerical experiments show that the deep learning method is efficient with a moderate generalization error at all points. And the computational cost does not suffer from “curse of dimensionality” problem.

Introduction

Integral equations have wide applications in electrical engineering [1], optics [2], mathematical biology [3] and other fields. The most popular integral equations are the Fredhom integral equations and the Volterra integral equations. The Fredholm integral equation can be considered as a reformulation of the elliptic partial differential equation and the Volterra integral equation is a reformulation of the fractional-order differential equation, which has wide applications in modeling the real problems, for instance, the chaotic system [4], the dynamics of COVID-19 [5], the motion of beam on nanowire [6], the capacitor microphone dynamical system [7], etc. Since these integral equations usually can not be solved explicitly, numerical methods are necessary to be considered.

We consider the linear Fredholm integral equation of the second kind

$$\begin{array}{c}\hfill f(\mathbf{x})-\underset{\mathrm{\Omega }}{\int }k(\mathbf{x},\mathbf{y})f(\mathbf{y})d\mathbf{y}=g(\mathbf{x}),\end{array}$$ 1

where $\mathbf{x},\mathbf{y}\in \mathrm{\Omega }\subset {\mathbb{R}}^m$, the function $g(\mathbf{x})$ and the kernel $k(\mathbf{x},\mathbf{y})$ are given, and $f(\mathbf{x})$ is the unknown that we want to find.

So far, many numerical methods have been proposed to solve the Fredholm integral equations, for example, the Nyström method [3, 8, 9], the Galerkin method [10], the wavelet analysis method [11], the neural network [12, 13], the collocation method [14], the maximum entropy method [15], etc. However, most of these traditional methods can only solve low-dimensional Fredholm integral equations and suffer from “curse of dimensionality".

The neural network has been successful in solving partial differential equations in mathematical modelling and the applied science, such as medical smoking model [16], nonlinear high order singular models [17], food chain system [18–20], Liénard differential model [21], etc. The neural network was also used to solve the Fredholm integral equations in [12, 13], where the authors only evaluated the approximation at some fixed points without generalization. And the integral was evaluated using numerical integral method whose cost depends on the dimension exponentially.

In recent years, deep learning method has been successfully used in artificial intelligence solving high-dimensional problems, such as image recognition [22, 23], speech recognition [24, 25], natural language processing [26], and also in mathematical problems [27–29] and physical problems [30].

E and his collaborators have done a series of works on solving high-dimensional differential equations based on deep learning method. In [28], a deep learning-based algorithm was proposed for solving high-dimensional semilinear parabolic partial differential equations and reverse stochastic differential equations from a relation between BSDE (backward stochastic differential equations) and reinforcement learning. In [29], the deep Ritz method for elliptic differential equations was given by numerically solving variational problems. In [27], a machine learning approximation algorithm was raised to solve high-dimensional fully nonlinear second-order partial differential equations. These works show that deep learning method provides a new idea to solve high-dimensional mathematical problems.

In this paper, a deep residual neural network method is proposed to approximate the solution of the high-dimensional linear Fredholm integral equations of the second kind. Few novel highlights of this deep learning method are briefly provided as follows:

• A deep residual neural network is constructed to solve numerically the linear Fredholm integral equations of the second kind.

• The proposed method can solve high-dimensional Fredholm integral equations and does not suffer from “curse of dimensionality” problem, that is the cost depends on the dimension linearly.

• The reasonable absolute error values validate the reliability of the deep learning method.

• The proposed method has a small generalization error in the domain.

This paper is organized as follows. In Sect. 2 we construct a deep residual neutral network for solving the Fredholm integral equations. In Sect. 3, some numerical experiments are given to show the efficiency of the numerical method. The conclusion is given in Sect. 4.

Proposed Deep Learning Method for Solving Fredhom Integral Equations

The output $F(\mathbf{x},\mathit{\theta })$ of the neural network is a composite function of the input $\mathbf{x}$, where $\mathit{\theta }$ denotes the parameters of the neural network including the weighs and bias. Let $\mathbf{x}$ be any point in the domain $\mathrm{\Omega }$. Now we want to train a deep neural network whose output $F(\mathbf{x},\mathit{\theta })$ is the solution of the Fredholm integral Eq. (1), that is

$$\begin{array}{c}\hfill F(\mathbf{x},\mathit{\theta })-\underset{\mathrm{\Omega }}{\int }k(\mathbf{x},\mathbf{y})F(\mathbf{y},\mathit{\theta })d\mathbf{y}=g(\mathbf{x}),\mathbf{x}\in \mathrm{\Omega }.\end{array}$$

To learn the parameters $\mathit{\theta }$, and so the function $F(\mathbf{x},\mathit{\theta })$, take randomly n points $\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$ with a uniform distribution as the training set. Initializing the parameter vector $\mathit{\theta }$, the prediction values $F({\mathbf{x}}_i,\mathit{\theta })$ for $i=1,2,\cdots ,n$, can be obtained by forward propagation neural network.

Define the loss function as

$$\begin{array}{c}\hfill los{s}_{\mathbf{x}}(\mathit{\theta })=\frac{1}{n}\sum _{i=1}^n{\left(F,({\mathbf{x}}_i,\mathit{\theta }),-,\underset{\mathrm{\Omega }}{\int },k,({\mathbf{x}}_i,\mathbf{y}),F,(\mathbf{y},\mathit{\theta }),d,\mathbf{y},-,g,({\mathbf{x}}_i)\right)}^2.\end{array}$$ 2

The training of the neural network is to minimize the loss function (2) by the backward propagation neural network, which is a least-square problem

$$\begin{array}{c}\hfill \underset{\mathit{\theta }}{min}los{s}_{\mathbf{x}}(\mathit{\theta }).\end{array}$$ 3

In Eq. (2) the integral term ${\displaystyle \underset{\mathrm{\Omega }}{\int }k({\mathbf{x}}_i,\mathbf{y})F(\mathbf{y},\mathit{\theta })d\mathbf{y}}$ can be evaluated using the Monte Carlo method, leading to

$$\begin{array}{c}\hfill \underset{\mathrm{\Omega }}{\int }k({\mathbf{x}}_i,\mathbf{y})F(\mathbf{y},\mathit{\theta })d\mathbf{y}\approx \frac{\beta }{n}\sum _{j=1}^{n}k({\mathbf{x}}_i,{\mathbf{x}}_j)F({\mathbf{x}}_j,\mathit{\theta }),\end{array}$$ 4

where ${\displaystyle \beta =\underset{\mathrm{\Omega }}{\int }d\mathbf{x}}$ is the volume of $\mathrm{\Omega }$.

The training can be done repeatedly for different training set until we get a stationary loss function.

As the network deepens, minimizing the loss function has great difficulties, such as vanishing gradient problem, gradient explosion, and degradation problem. The residual neural network can avoid the vanishing gradient problem and may greatly improve the solution. It also can reduce the risk of over-adapting the parameters to a specific dataset [22]. A residual block is shown in Fig. 1, where an identity shortcut connection is added to a shallow neural network, whose output is $H(\mathbf{x})=\phi (\mathbf{x})+\mathbf{x}$, where $\phi (\mathbf{x})$ is the output of the shallow neural network. Then the output of the residual block is taken as the input of the next residual block.

Fig. 1.

Residual neural network block

Our algorithm of deep residual neural network for solving Fredholm integral equations is shown in algorithm 1.

Numerical Experiments

In this section, several Fredholm integral equations are numerically solved using algorithm 1. In the numerical experiments, $n=1000$ points in $\mathrm{\Omega }$ are randomly sampled uniformly as the training set to train the deep residual neural network, and the number of training iterations is $M=2000$. The neural network consists of one input layer, two blocks of residual neural network shown in Fig. 1, and one output layer. There are 30 neurons in the second layer and the forth layer and 10 neurons in the other layers. The ReLU function is used as the active function in the neural network. Minimization is realized by “AdamOptimizer” [31] built in TensorFlow (version 1.13.1 ) with a learning rate 0.001.

To measure the efficiency of the deep learning method for solving the Fredholm integral equations, we consider several examples whose exact solutions are known. Denote $f^{\ast }({\mathbf{x}}_1),f^{\ast }({\mathbf{x}}_2),\cdots ,f^{\ast }({\mathbf{x}}_n)$ as the exact solutions at n points ${\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n$ in the test set. Define the generalization error between the exact solution $f^{\ast }(\mathbf{x})$ of the integral equation and the approximate solution $F(\mathbf{x},\mathit{\theta })$ obtained by using the deep residual neural network as

$$\begin{array}{c}\hfill error=\frac{1}{n}\sum _{i=1}^n{\left(f^{\ast },({\mathbf{x}}_i),-,F,({\mathbf{x}}_i,\mathit{\theta })\right)}^2.\end{array}$$

The generalization error is evaluated for each example in the following numerical experiments.

Example 1

Consider the three-dimensional Fredholm integral equation

$$\begin{array}{c}\hfill f(x,y,z)+{\int }_1^2{\int }_1^2{\int }_1^{2}k(x,y,z,s,t,v)f(s,t,v)dsdtdv=g(x,y,z),\end{array}$$ 5

where $\mathrm{\Omega }={[1,2]}^3$,

$$\begin{array}{cc}& k(x,y,z,s,t,v)=e^{-xs-yt-zv},\hfill \end{array}$$

and

$$\begin{array}{cc}& g(x,y,z)=1-\frac{1}{\mathit{xyz}}(e^{-2x}-e^{-x})(e^{-2y}-e^{-y})(e^{-2z}-e^{-z}).\hfill \end{array}$$

The exact solution of Eq. (5) is $f^{\ast }(x,y,z)=1$.

For Example 1, the convergence of the loss function and the generalization error are shown in Fig. 2. Some typical iteration data of the loss function (loss) and the generalization error (error) are given in Table 1. The loss function converges to ${10}^{-5}$, and the generalization error converges to ${10}^{-3}$.

Fig. 2.

Convergence of the loss function (left) and the generalization error (right) of Example 1

Table 1.

Partial iterative results of the loss function and the generalization error for Example 1

Number of training	loss	error	Number of training	loss	error
1	0.8958	0.9446	800	3.498e-4	0.0161
60	0.0428	0.2040	1200	9.472e-5	0.0080
80	0.008	0.0804	1600	3.852e-5	0.0051
409	9.937e-4	0.0270	2000	1.917e-5	0.0035

Example 2

Consider a high-dimensional version of the three-dimensional Fredholm integral Eq. (5), that is

$$\begin{array}{c}\hfill f(\mathbf{x})+\underset{\mathrm{\Omega }}{\int }k(\mathbf{x},\mathbf{y})f(\mathbf{y})d\mathbf{y}=g(\mathbf{x}),\end{array}$$

where $\mathrm{\Omega }={[1,2]}^m$, $\mathbf{x}=\{x_1,x_2,\cdots ,x_m\}$, $\mathbf{y}=\{y_1,y_2,\cdots ,y_m\}$,

$$\begin{array}{cc}& {\displaystyle k(\mathbf{x},\mathbf{y})=e^{-x_1{y}_1-\cdots -x_m{y}_m}}\hfill \end{array}$$

and

$$\begin{array}{cc}& {\displaystyle g(\mathbf{x})=1+{(-1)}^m\frac{1}{x_1{x}_2\cdots x_m}(e^{-2x_1}-e^{-x_1})(e^{-2x_2}-e^{-x_2})\cdots (e^{-2x_m}-e^{-x_m}).}\hfill \end{array}$$

The exact solution is $f^{\ast }(\mathbf{x})=1$.

For Example 2, when the dimension $m=100$, the convergence of the loss function and the generalization error are shown in Fig. 3. Some typical iteration data of the loss function (loss) and the generalization error (error) are given in Table 2. The loss function converges to ${10}^{-4}$, and the generalization error converges to ${10}^{-3}$.

Fig. 3.

Convergence of the loss function (left) and the generalization error (right) of Example 2

Table 2.

Partial iterative results of the loss function and the generalization error for Example 2 when the dimension $m=100$

Number of training	loss	error	Number of training	loss	error
1	1.264	1.123	800	5.814e-4	0.0195
32	0.0131	0.1055	1100	4.195e-4	0.0162
300	0.0012	0.0277	1600	2.239e-4	0.0119
400	9.805e-4	0.0254	2000	1.417e-4	0.0094

Example 3

Consider the four-dimensional Fredholm integral equation

$$\begin{array}{c}\hfill f(x,y,z,w)-xyzw{\int }_0^1{\int }_0^1{\int }_0^1{\int }_0^{1}f(s,t,v,r)dsdtdvdr=\frac{15}{16}xyzw\end{array}$$ 6

where $\mathrm{\Omega }={[0,1]}^4$,

$$\begin{array}{cc}& k(x,y,z,w,s,t,v,r)=xyzw,\hfill \end{array}$$

and

$$\begin{array}{cc}& {\displaystyle g(x,y,z,w)=\frac{15}{16}xyzw.}\hfill \end{array}$$

The exact solution is $f^{\ast }(x,y,z,w)=xyzw$.

For Example 3, the convergence of the loss function and the generalization error are shown in Fig. 4, and some typical iteration data of the loss function (loss) and the generalization error (error) are given in Table 3. The loss function and the generalized error function synchronously converge very fast to a stable state. The loss function converges to ${10}^{-3}$, and the generalization error converges to ${10}^{-2}$.

Fig. 4.

Convergence of the loss function (left) and the generalization error (right) of Example 3

Table 3.

Partial iterative results of the loss function and the generalization error for Example 3

Number of training	loss	error	Number of training	loss	error
1	0.0167	0.1190	1100	0.0074	0.0621
100	0.0070	0.0602	1400	0.0079	0.0603
500	0.0075	0.0632	1700	0.0064	0.0585
800	0.0064	0.0614	2000	0.0066	0.0606

Example 4

Consider a high-dimensional version of the four-dimensional Fredholm integral Eq. (6), that is

$$\begin{array}{c}\hfill f(\mathbf{x})-x_1{x}_2\cdots x_m\underset{\mathrm{\Omega }}{\int }f(\mathbf{y})d\mathbf{y}=\left(1,-,\frac{1}{2^m}\right)x_1{x}_2\cdots x_m\end{array}$$

where $\mathbf{x}\in {[0,1]}^m$, $\mathbf{x}=\{x_1,x_2,\cdots ,x_m\}$, $\mathbf{y}=\{y_1,y_2,\cdots ,y_m\}$,

$$\begin{array}{cc}& k(\mathbf{x},\mathbf{y})=x_1{x}_2\cdots x_m,\hfill \end{array}$$

and

$$\begin{array}{cc}& {\displaystyle g(\mathbf{x})=\left(1,-,\frac{1}{2^m}\right)x_1{x}_2\cdots x_m.}\hfill \end{array}$$

The exact solution of the equation is ${\displaystyle f^{\ast }(\mathbf{x})=x_1{x}_2\cdots x_m}$.

For Example 4, when the dimension $m=100$, the convergence of the loss function and the generalization error are shown in Fig. 5, and some typical iteration data of the loss function (loss) and the generalization error (error) are given in Table 4. The loss function converges to ${10}^{-7}$, and the generalization error converges to ${10}^{-4}$.

Fig. 5.

Convergence of the loss function (left) and the generalization error (right) of Example 4

Table 4.

Partial iterative results of the loss function and the generalization error for Example 4 when the dimension $m=100$

Number of training	loss	error	Number of training	loss	error
1	0.6974	0.8341	1114	9.910e-6	0.0023
500	0.0010	0.0256	1243	3.728e-6	0.0010
800	2.731e-4	0.0129	1337	7.648e-7	4.484e-4
1000	4.119e-5	0.0050	2000	2.826e-7	1.718e-4

Discussion and Conclusions

In this paper, we propose a deep learning method based on the residual neural network to solve numerically the linear Fredholm integral equations of the second kind. The output of the deep residual network is used as the numerical solution. The loss function is defined using the Fredholm integral equation. The loss function is optimized by Adam method built in TensorFlow. Then the numerical results, including high-dimensional problems, confirm the efficiency of the method. The main advantage of this method is that it can solve high-dimensional Fredholm integral equations with a cost less sensitive to the dimensionality of the problem.

The accuracy of the residual neural network is not as good as that of the traditional method, such as the Galerkin method. Some error analysis of the neural network has been discussed in [32–34]. But so far rigorous error analysis for neural network can not be given yet. The error of the neural network consists of three parts, that is the error between the space of the output of the neural network and the exact solution of the Fredholm integral equation, the optimization error in Eq. (3), and the approximation error in Eq. (4). The error in our numerical experiments has a good accuracy compared to the error of the Monte Carlo method $1/\sqrt{n}$ in Eq. (4). In the future we will explore more techniques or theory to improve the convergent accuracy. Additionally, we will try to construct a deep residual neural network to solve the Volterra integral equations.

Funding

The authors have not disclosed any funding.

Availability of data and materials

All data generated or analyzed during this study are included in this manuscript.

Declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.