Statistical analysis of differential equations: introducing probability measures on numerical solutions

Abstract

In this paper, we present a formal quantification of uncertainty induced by numerical solutions of ordinary and partial differential equation models. Numerical solutions of differential equations contain inherent uncertainties due to the finite-dimensional approximation of an unknown and implicitly defined function. When statistically analysing models based on differential equations describing physical, or other naturally occurring, phenomena, it can be important to explicitly account for the uncertainty introduced by the numerical method. Doing so enables objective determination of this source of uncertainty, relative to other uncertainties, such as those caused by data contaminated with noise or model error induced by missing physical or inadequate descriptors. As ever larger scale mathematical models are being used in the sciences, often sacrificing complete resolution of the differential equation on the grids used, formally accounting for the uncertainty in the numerical method is becoming increasingly more important. This paper provides the formal means to incorporate this uncertainty in a statistical model and its subsequent analysis. We show that a wide variety of existing solvers can be randomised, inducing a probability measure over the solutions of such differential equations. These measures exhibit contraction to a Dirac measure around the true unknown solution, where the rates of convergence are consistent with the underlying deterministic numerical method. Furthermore, we employ the method of modified equations to demonstrate enhanced rates of convergence to stochastic perturbations of the original deterministic problem. Ordinary differential equations and elliptic partial differential equations are used to illustrate the approach to quantify uncertainty in both the statistical analysis of the forward and inverse problems.

Electronic supplementary material

The online version of this article (doi:10.1007/s11222-016-9671-0) contains supplementary material, which is available to authorized users.

Introduction

Motivation

The numerical analysis literature has developed a large range of efficient algorithms for solving ordinary and partial differential equations, which are typically designed to solve a single problem as efficiently as possible (Hairer et al. 1993; Eriksson 1996). When classical numerical methods are placed within statistical analysis, however, we argue that significant difficulties can arise as a result of errors in the computed approximate solutions. While the distributions of interest commonly do converge asymptotically as the solver mesh becomes dense [e.g. in statistical inverse problems (Dashti and Stuart 2016)], we argue that at a finite resolution, the statistical analyses may be vastly overconfident as a result of these unmodelled errors.

The purpose of this paper is to address these issues by the construction and rigorous analysis of novel probabilistic integration methods for both ordinary and partial differential equations. The approach in both cases is similar: we identify the key discretisation assumptions and introduce a local random field, in particular a Gaussian field, to reflect our uncertainty in those assumptions. The probabilistic solver may then be sampled repeatedly to interrogate the uncertainty in the solution. For a wide variety of commonly used numerical methods, our construction is straightforward to apply and provably preserves the order of convergence of the original method.

Furthermore, we demonstrate the value of these probabilistic solvers in statistical inference settings. Analytic and numerical examples show that using a classical non-probabilistic solver with inadequate discretisation when performing inference can lead to inappropriate and misleading posterior concentration in a Bayesian setting. In contrast, the probabilistic solver reveals the structure of uncertainty in the solution, naturally limiting posterior concentration as appropriate.

As a motivating example, consider the solution of the Lorenz’63 system. Since the problem is chaotic, any typical fixed-step numerical methods will become increasingly inaccurate for long integration times. Figure 1 depicts a deterministic solution for this problem, computed with a fixed-step, fourth-order, Runge–Kutta integrator. Although the solver becomes completely inaccurate by the end of the depicted interval given the step-size selected, the solver provides no obvious characterisation of its error at late times. Compare this with a sample of randomised solutions based on the same integrator and the same step-size; it is obvious that early-time solutions are accurate and that they diverge at late times, reflecting instability of the solver. Every curve drawn has the same theoretical accuracy as the original classical method, but the randomised integrator provides a detailed and practical approach for revealing the sensitivity of the solution to numerical errors. The method used requires only a straightforward modification of the standard Runge–Kutta integrator and is explained in Sect. 2.3.

Fig. 1.

A comparison of solutions to the Lorenz’63 system using deterministic (red) and randomised (blue) integrators based on a fourth-order Runge–Kutta integrator

We summarise the contributions of this work as follows:

• Construct randomised solvers of ODEs and PDEs using natural modification of popular, existing solvers.

• Prove the convergence of the randomised methods and study their behaviour by showing a close link between randomised ODE solvers and stochastic differential equations (SDEs).

• Demonstrate that these randomised solvers can be used to perform statistical analyses that appropriately consider solver uncertainty.

Review of existing work

The statistical analysis of models based on ordinary and partial differential equations is growing in importance and a number of recent papers in the statistics literature have sought to address certain aspects specific to such models, e.g. parameter estimation (Liang and Wu 2008; Xue et al. 2010; Xun et al. 2013; Brunel et al. 2014) and surrogate construction (Chakraborty et al. 2013). However, the statistical implications of the reliance on a numerical approximation to the actual solution of the differential equation have not been addressed in the statistics literature to date and this is the open problem comprehensively addressed in this paper. Earlier work in the literature including randomisation in the approximate integration of ordinary differential equations (ODEs) includes (Coulibaly and Lécot 1999; Stengle 1995). Our strategy fits within the emerging field known as Probabilistic Numerics (Hennig et al. 2015), a perspective on computational methods pioneered by Diaconis (1988), and subsequently (Skilling 1992). This framework recasts solving differential equations as a statistical inference problem, yielding a probability measure over functions that satisfy the constraints imposed by the specific differential equation. This measure formally quantifies the uncertainty in candidate solution(s) of the differential equation, allowing its use in uncertainty quantification (Sullivan 2016) or Bayesian inverse problems (Dashti and Stuart 2016).

A recent Probabilistic Numerics methodology for ODEs (Chkrebtii et al. 2013) [explored in parallel in Hennig and Hauberg (2014)] has two important shortcomings. First, it is impractical, only supporting first-order accurate schemes with a rapidly growing computational cost caused by the growing difference stencil [although Schober et al. (2014) extends to Runge–Kutta methods]. Secondly, this method does not clearly articulate the relationship between their probabilistic structure and the problem being solved. These methods construct a Gaussian process whose mean coincides with an existing deterministic integrator. While they claim that the posterior variance is useful, by the conjugacy inherent in linear Gaussian models, it is actually just an a priori estimate of the rate of convergence of the integrator, independent of the actual forcing or initial condition of the problem being solved. These works also describe a procedure for randomising the construction of the mean process, which bears similarity to our approach, but it is not formally studied. In contrast, we formally link each draw from our measure to the analytic solution.

Our motivation for enhancing inference problems with models of discretisation error is similar to the more general concept of model error, as developed by Kennedy and O’Hagan (2001). Although more general types of model error, including uncertainty in the underlying physics, are important in many applications, our focus on errors arising from the discretisation of differential equations leads to more specialised methods. Future work may be able to translate insights from our study of the restricted problem to the more general case. Existing strategies for discretisation error include empirically fitted Gaussian models for PDE errors (Kaipio and Somersalo 2007) and randomly perturbed ODEs (Arnold et al. 2013); the latter partially coincides with our construction, but our motivation and analysis are distinct. Recent work (Capistrán et al. 2013) uses Bayes factors to analyse the impact of discretisation error on posterior approximation quality. Probabilistic models have also been used to study error propagation due to rounding error; see Hairer et al. (2008).

Organisation

The remainder of the paper has the following structure: Sect. 2 introduces and formally analyses the proposed probabilistic solvers for ODEs. Section 3 explores the characteristics of random solvers employed in the statistical analysis of both forward and inverse problems. Then, we turn to elliptic PDEs in Sect. 4, where several key steps of the construction of probabilistic solvers and their analysis have intuitive analogues in the ODE context. Finally, an illustrative example of an elliptic PDE inference problem is presented in Sect. 5.1

Probability measures via probabilistic time integrators

Consider the following ordinary differential equation (ODE):

$$\begin{array}{c}\hfill \frac{\mathrm{d}u}{\mathrm{d}t}=f(u),u(0)=u_0,\end{array}$$ 1

where $u(·)$ is a continuous function taking values in ${\mathbb{R}}^n$.2 We let ${\mathit{\Phi }}_t$ denote the flow map for Eq. (1), so that $u(t)={\mathit{\Phi }}_t(u(0))$. The conditions ensuring that this solution exists will be formalised in Assumption 2, below.

Deterministic numerical methods for the integration of this equation on time interval [0, T] will produce an approximation to the equation on a mesh of points ${\{t_k=kh\}}_{k=0}^K$, with $Kh=T$, (for simplicity we assume a fixed mesh). Let $u_k=u(t_k)$ denote the exact solution of (1) on the mesh and $U_k\approx u_k$ denote the approximation computed using finite evaluations of f. Typically, these methods output a single discrete solution ${\{U_k\}}_{k=0}^K$, often augmented with some type of error indicator, but do not statistically quantify the uncertainty remaining in the path.

Let $X_{a,b}$ denote the Banach space $C([a,b];{\mathbb{R}}^n)$. The exact solution of (1) on the time interval [0, T] may be viewed as a Dirac measure ${\mathit{\delta }}_u$ on $X_{0,T}$ at the element u that solves the ODE. We will construct a probability measure ${\mathit{\mu }}^h$ on $X_{0,T}$, that is straightforward to sample from both on and off the mesh, for which h quantifies the size of the discretisation step employed, and whose distribution reflects the uncertainty resulting from the solution of the ODE. Convergence of the numerical method is then related to the contraction of ${\mathit{\mu }}^h$ to ${\mathit{\delta }}_u$.

We briefly summarise the construction of the numerical method. Let ${\mathit{\Psi }}_h:{\mathbb{R}}^n\to {\mathbb{R}}^n$ denote a classical deterministic one-step numerical integrator over time-step h, a class including all Runge–Kutta methods and Taylor methods for ODE numerical integration (Hairer et al. 1993). Our numerical methods will have the property that, on the mesh, they take the form

$$\begin{array}{c}\hfill U_{k+1}={\mathit{\Psi }}_h(U_k)+{\mathit{\xi }}_k(h),\end{array}$$ 2

where ${\mathit{\xi }}_k(h)$ are suitably scaled, i.i.d. Gaussian random variables. That is, the random solution iteratively takes the standard step, ${\mathit{\Psi }}_h$, followed by perturbation with a random draw, ${\mathit{\xi }}_k(h)$, modelling uncertainty that accumulates between mesh points. The discrete path ${\{U_k\}}_{k=0}^K$ is straightforward to sample and in general is not a Gaussian process. Furthermore, the discrete trajectory can be extended into a continuous time approximation of the ODE, which we define as a draw from the measure ${\mathit{\mu }}^h$.

The remainder of this section develops these solvers in detail and proves strong convergence of the random solutions to the exact solution, implying that ${\mathit{\mu }}^h\to {\mathit{\delta }}_u$ in an appropriate sense. Finally, we establish a close relationship between our random solver and a stochastic differential equation (SDE) with small mesh-dependent noise. Intuitively, adding Gaussian noise to an ODE suggests a link to SDEs. Additionally, note that the mesh-restricted version of our algorithm, given by (2), has the same structure as a first-order Ito–Taylor expansion of the SDE

$$\begin{array}{c}\hfill \mathrm{d}u=f(u)\mathrm{d}t+\mathit{\sigma }\mathrm{d}W,\end{array}$$ 3

for some choice of $\mathit{\sigma }$. We make this link precise by performing a backwards error analysis, which connects the behaviour of our solver to an associated SDE.

Probabilistic time integrators: general formulation

The integral form of Eq. (1) is

$$\begin{array}{c}\hfill u(t)=u_0+{\int }_0^{t}f(u(s))\mathrm{d}s.\end{array}$$ 4

The solutions on the mesh satisfy

$$\begin{array}{c}\hfill u_{k+1}=u_k+{\int }_{t_k}^{t_{k+1}}f(u(s))\mathrm{d}s,\end{array}$$ 5

and may be interpolated between mesh points by means of the expression

$$\begin{array}{c}\hfill u(t)=u_k+{\int }_{t_k}^{t}f(u(s))\mathrm{d}s,t\in [t_k,t_{k+1}).\end{array}$$ 6

We may then write

$$\begin{array}{c}\hfill u(t)=u_k+{\int }_{t_k}^{t}g(s)\mathrm{d}s,t\in [t_k,t_{k+1}),\end{array}$$ 7

where $g(s)=f(u(s))$ is an unknown function of time. In the algorithmic setting, we have approximate knowledge about g(s) through an underlying numerical method. A variety of traditional numerical algorithms may be derived based on approximation of g(s) by various simple deterministic functions $g^h(s)$. The simplest such numerical method arises from invoking the Euler approximation that

$$\begin{array}{c}\hfill g^h(s)=f(U_k),s\in [t_k,t_{k+1}).\end{array}$$ 8

In particular, if we take $t=t_{k+1}$ and apply this method inductively the corresponding numerical scheme arising from making such an approximation to g(s) in (7) is $U_{k+1}=U_k+hf(U_k).$ Now consider the more general one-step numerical method $U_{k+1}={\mathit{\Psi }}_h(U_k).$ This may be derived by approximating g(s) in (7) by

9

We note that all consistent (in the sense of numerical analysis) one-step methods will satisfy

$$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}\mathit{\tau }}({\mathit{\Psi }}_{\mathit{\tau }}(u){)}_{\mathit{\tau }=0}=f(u).\end{array}$$

The approach based on the approximation (9) leads to a deterministic numerical method which is defined as a continuous function of time. Specifically, we have $U(s)={\mathit{\Psi }}_{s-t_k}(U_k),s\in [t_k,t_{k+1}).$ Consider again the Euler approximation, for which ${\mathit{\Psi }}_{\mathit{\tau }}(U)=U+\mathit{\tau }f(U)$, and whose continuous time interpolant is then given by $U(s)=U_k+(s-t_k)f(U_k),s\in [t_k,t_{k+1}).$ Note that this produces a continuous function, namely an element of $X_{0,T}$, when extended to $s\in [0,T].$ The preceding development of a numerical integrator does not acknowledge the uncertainty that arises from lack of knowledge about g(s) in the interval $s\in [t_k,t_{k+1}).$ We propose to approximate g stochastically in order to represent this uncertainty, taking

$$\begin{array}{c}\hfill g^h(s)=\frac{\mathrm{d}}{\mathrm{d}\mathit{\tau }}({\mathit{\Psi }}_{\mathit{\tau }}(U_k){)}_{\mathit{\tau }=s-t_k}+{\mathit{\chi }}_k(s-t_k),s\in [t_k,t_{k+1})\end{array}$$

where the $\{{\mathit{\chi }}_k\}$ form an i.i.d. sequence of Gaussian random functions defined on [0, h] with ${\mathit{\chi }}_k\sim N(0,C^h)$.3

We will choose $C^h$ to shrink to zero with h at a prescribed rate (see Assumption 1), and also to ensure that ${\mathit{\chi }}_k\in X_{0,h}$ almost surely. The functions $\{{\mathit{\chi }}_k\}$ represent our uncertainty about the function g. The corresponding numerical scheme arising from such an approximation is given by

$$\begin{array}{c}\hfill U_{k+1}={\mathit{\Psi }}_h(U_k)+{\mathit{\xi }}_k(h),\end{array}$$ 10

where the i.i.d. sequence of functions $\{{\mathit{\xi }}_k\}$ lies in $X_{0,h}$ and is given by

$$\begin{array}{c}\hfill {\mathit{\xi }}_k(t)={\int }_0^t{\mathit{\chi }}_k(\mathit{\tau })\mathrm{d}\mathit{\tau }.\end{array}$$ 11

Note that the numerical solution is now naturally defined between grid points, via the expression

$$\begin{array}{c}\hfill U(s)={\mathit{\Psi }}_{s-t_k}(U_k)+{\mathit{\xi }}_k(s-t_k),s\in [t_k,t_{k+1}).\end{array}$$ 12

When it is necessary to evaluate a solution at multiple points in an interval, $s\in (t_k,t_{k+1}]$, the perturbations ${\mathit{\xi }}_k(s-t_k)$ must be drawn jointly, which is facilitated by their Gaussian structure. Although most users will only need the formulation on mesh points, we must consider off-mesh behaviour to rigorously analyse higher order methods, as is also required for the deterministic variants of these methods.

In the case of the Euler method, for example, we have

$$\begin{array}{c}\hfill U_{k+1}=U_k+hf(U_k)+{\mathit{\xi }}_k(h)\end{array}$$ 13

and, between grid points,

$$\begin{array}{c}\hfill U(s)=U_k+(s-t_k)f(U_k)+{\mathit{\xi }}_k(s-t_k),s\in [t_k,t_{k+1}).\end{array}$$ 14

This method is illustrated in Fig. 2. Observe that Eq. (13) has the same form as an Euler–Maryama method for an associated SDE (3) where $\mathit{\sigma }$ depends on the step-size h. In particular, in the simple one-dimensional case, $\mathit{\sigma }$ would be given by $\sqrt{C^h/h}$. Section 2.4 develops a more sophisticated connection that extends to higher order methods and off the mesh.

Fig. 2.

An illustration of deterministic Euler steps and randomised variations. The random integrator in (b) outputs the path in red; we overlay the standard Euler step constructed at each step, before it is perturbed (blue)

While we argue that the choice of modelling local uncertainty in the flow map as a Gaussian process is natural and analytically favourable, it is not unique. It is possible to construct examples where the Gaussian assumption is invalid; for example, when a highly inadequate time-step is used, a systemic bias may be introduced. However, in regimes where the underlying deterministic method performs well, the centred Gaussian assumption is a reasonable prior.

Strong convergence result

To prove the strong convergence of our probabilistic numerical solver, we first need two assumptions quantifying properties of the random noise and of the underlying deterministic integrator, respectively. In what follows we use $⟨·,·⟩$ and $|·|$ to denote the Euclidean inner product and norm on ${\mathbb{R}}^n$. We denote the Frobenius norm on ${\mathbb{R}}^{n\times n}$ by ${|·|}_{\mathrm{F}}$, and ${\mathbb{E}}^h$ denotes expectation with respect to the i.i.d. sequence $\{{\mathit{\chi }}_k\}$.

Assumption 1

Let ${\mathit{\xi }}_k(t):={\int }_0^t{\mathit{\chi }}_k(s)ds$ with ${\mathit{\chi }}_k\sim N(0,C^h).$ Then there exists $K>0,p\ge 1$ such that, for all $t\in [0,h]$, ${\mathbb{E}}^h{|{\mathit{\xi }}_k(t){\mathit{\xi }}_k{(t)}^T|}_{\mathrm{F}}^2\le K{t}^{2p+1};$ in particular ${\mathbb{E}}^h{|{\mathit{\xi }}_k(t)|}^2\le K{t}^{2p+1}.$ Furthermore, we assume the existence of matrix Q, independent of h, such that ${\mathbb{E}}^h[{\mathit{\xi }}_k(h){\mathit{\xi }}_k{(h)}^T]=Q{h}^{2p+1}.$

Here, and in the sequel, K is a constant independent of h, but possibly changing from line to line. Note that the covariance kernel $C^h$ is constrained, but not uniquely defined. We will assume the form of the constant matrix is $Q=\mathit{\sigma }I$, and we discuss one possible strategy for choosing $\mathit{\sigma }$ in Sect. 3.1. Section 2.4 uses a weak convergence analysis to argue that once Q is selected, the exact choice of $C^h$ has little practical impact.

Assumption 2

The function f and a sufficient number of its derivatives are bounded uniformly in ${\mathbb{R}}^n$ in order to ensure that f is globally Lipschitz and that the numerical flow map ${\mathit{\Psi }}_h$ has uniform local truncation error of order $q+1$:

$$\begin{array}{c}\hfill \underset{u\in {\mathbb{R}}^n}{sup}|{\mathit{\Psi }}_t(u)-{\mathit{\Phi }}_t(u)|\le K{t}^{q+1}.\end{array}$$

Remark 2.1

We assume globally Lipschitz f, and bounded derivatives, in order to highlight the key probabilistic ideas, whilst simplifying the numerical analysis. Future work will address the non-trivial issue of extending of analyses to weaken these assumptions. In this paper, we provide numerical results indicating that a weakening of the assumptions is indeed possible.

Theorem 2.2

Under Assumptions 1, 2 it follows that there is $K>0$ such that

$$\begin{array}{c}\hfill \underset{0\le kh\le T}{sup}{\mathbb{E}}^h{|u_k-U_k|}^2\le K{h}^{2min\{p,q\}}.\end{array}$$

Furthermore,

$$\begin{array}{c}\hfill \underset{0\le t\le T}{sup}{\mathbb{E}}^h|u(t)-U(t)|\le K{h}^{min\{p,q\}}.\end{array}$$

This theorem implies that every probabilistic solution is a good approximation of the exact solution in both a discrete and continuous sense. Choosing $p\ge q$ is natural if we want to preserve the strong order of accuracy of the underlying deterministic integrator; we proceed with the choice $p=q$, introducing the maximum amount of noise consistent with this constraint.

Examples of probabilistic time integrators

The canonical illustration of a probabilistic time integrator is the probabilistic Euler method already described.4 Another useful example is the classical Runge–Kutta method which defines a one-step numerical integrator as follows:

$$\begin{array}{c}\hfill {\mathit{\Psi }}_h(u)=u+\frac{h}{6}(k_1(u)+2k_2(u,h)+2k_3(u,h)+k_4(u,h)),\end{array}$$

where

$$\begin{array}{cc}& k_1(u)=f(u),k_2(u,h)=f(u+\frac{1}{2}h{k}_1(u))\hfill \\ \hfill & k_3(u,h)=f(u+\frac{1}{2}h{k}_2(u)),k_4(u,h)=f(u+h{k}_3(u)).\hfill \end{array}$$

The method has local truncation error in the form of Assumption 2 with $q=4.$ It may be used as the basis of a probabilistic numerical method (12), and hence (10) at the grid points. Thus, provided that we choose to perturb this integrator with a random process ${\mathit{\chi }}_k$ satisfying Assumption 1 with $p\ge 4$,5 Theorem 2.2 shows that the error between the probabilistic integrator based on the classical Runge–Kutta method is, in the mean square sense, of the same order of accuracy as the deterministic classical Runge–Kutta integrator.

Backward error analysis

Backwards error analyses are useful tool for numerical analysis; the idea is to characterise the method by identifying a modified equation (dependent upon h) which is solved by the numerical method either exactly, or at least to a higher degree of accuracy than the numerical method solves the original equation. For our random ODE solvers, we will show that the modified equation is a stochastic differential equation (SDE) in which only the matrix Q from Assumption 1 enters; the details of the random processes used in our construction do not enter the modified equation. This universality property underpins the methodology we introduce as it shows that many different choices of random processes all lead to the same effective behaviour of the numerical method.

We introduce the operators $\mathcal{L}$ and ${\mathcal{L}}^h$ defined so that, for all $\mathit{\varphi }\in C^{\infty }({\mathbb{R}}^n,\mathbb{R})$,

$$\begin{array}{c}\hfill \mathit{\varphi }({\mathit{\Phi }}_h(u))=(e^{h\mathcal{L}}\mathit{\varphi })(u),\mathbb{E}\mathit{\varphi }(U_1|U_0=u)=(e^{h{\mathcal{L}}^h}\mathit{\varphi })(u).\end{array}$$ 15

Thus $\mathcal{L}:=f·\mathrm{\nabla }$ and $e^{h{\mathcal{L}}^h}$ is the kernel for the Markov chain generated by the probabilistic integrator (2). In fact we never need to work with ${\mathcal{L}}^h$ itself in what follows, only with $e^{h{\mathcal{L}}^h}$, so that questions involving the operator logarithm do not need to be discussed.

We now introduce a modified ODE and a modified SDE which will be needed in the analysis that follows. The modified ODE is

$$\begin{array}{c}\hfill \frac{\mathrm{d}\widehat{u}}{\mathrm{d}t}=f^h(\widehat{u})\end{array}$$ 16

whilst the modified SDE has the form

$$\begin{array}{c}\hfill \mathrm{d}\tilde{u}=f^h(\tilde{u})\mathrm{d}t+\sqrt{h^{2p}Q}\mathrm{d}W.\end{array}$$ 17

The precise choice of $f^h$ is detailed below. Letting $\mathbb{E}$ denote expectation with respect to W, we introduce the operators ${\widehat{\mathcal{L}}}^h$ and ${\tilde{\mathcal{L}}}^h$ so that, for all $\mathit{\varphi }\in C^{\infty }({\mathbb{R}}^n,\mathbb{R})$,

$$\begin{array}{cc}& \mathit{\varphi }(\widehat{u}(h)|\widehat{u}(0)=u)=(e^{h{\widehat{\mathcal{L}}}^h}\mathit{\varphi })(u),\hfill \end{array}$$ 18

$$\begin{array}{cc}& \mathbb{E}\mathit{\varphi }(\tilde{u}(h)|\tilde{u}(0)=0)=(e^{h{\tilde{\mathcal{L}}}^h}\mathit{\varphi })(u).\hfill \end{array}$$ 19

Thus,

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}^h:=f^h·\mathrm{\nabla },{\tilde{\mathcal{L}}}^h=f^h·\mathrm{\nabla }+\frac{1}{2}{h}^{2p}Q:\mathrm{\nabla }\mathrm{\nabla },\end{array}$$ 20

where  :  denotes the inner product on ${\mathbb{R}}^{n\times n}$ which induces the Frobenius norm, that is, A:B $=\text{trace}(A^{T}B)$.

The fact that the deterministic numerical integrator has uniform local truncation error of order $q+1$ (Assumption 2) implies that, since $\mathit{\varphi }\in C^{\infty }$,

$$\begin{array}{c}\hfill e^{h\mathcal{L}}\mathit{\varphi }(u)-\mathit{\varphi }({\mathit{\Psi }}_h(u))=\mathcal{O}(h^{q+1}).\end{array}$$ 21

The theory of modified equations for classical one-step numerical integration schemes for ODEs (Hairer et al. 1993) establishes that it is possible to find $f^h$ in the form

$$\begin{array}{c}\hfill f^h:=f+\sum _{i=q}^{q+l}{h}^i{f}_i,\end{array}$$ 22

such that

$$\begin{array}{c}\hfill e^{h{\widehat{\mathcal{L}}}^h}\mathit{\varphi }(u)-\mathit{\varphi }({\mathit{\Psi }}_h(u))=\mathcal{O}(h^{q+2+l}).\end{array}$$ 23

We work with this choice of $f^h$ in what follows.

Now for our stochastic numerical method we have

$$\begin{array}{cc}\hfill \mathit{\varphi }(U_{k+1})=& \mathit{\varphi }({\mathit{\Psi }}_h(U_k))+{\mathit{\xi }}_k(h)·\mathrm{\nabla }\mathit{\varphi }({\mathit{\Psi }}_h(U_k))\hfill \\ \hfill & +\frac{1}{2}{\mathit{\xi }}_k(h){\mathit{\xi }}_k^T(h):\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }({\mathit{\Psi }}_h(U_k))+\mathcal{O}(|{\mathit{\xi }}_k(h){|}^3).\hfill \end{array}$$

Furthermore, the last term has mean of size $\mathcal{O}(|{\mathit{\xi }}_k(h){|}^4)$. From Assumption 1 we know that ${\mathbb{E}}^h\left({\mathit{\xi }}_k(h){\mathit{\xi }}_k^T(h)\right)=Q{h}^{2p+1}.$ Thus

$$\begin{array}{cc}& e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)-\mathit{\varphi }({\mathit{\Psi }}_h(u))\hfill \\ \hfill & =\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }({\mathit{\Psi }}_h(u))+\mathcal{O}(h^{4p+2}).\hfill \end{array}$$ 24

From this it follows that

$$\begin{array}{cc}& e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)-\mathit{\varphi }({\mathit{\Psi }}_h(u))\hfill \\ \hfill & =\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }(u)+\mathcal{O}(h^{2p+2}).\hfill \end{array}$$ 25

Finally we note that (20) implies that

$$\begin{array}{cc}& e^{h{\tilde{\mathcal{L}}}^h}\mathit{\varphi }(u)-e^{h{\widehat{\mathcal{L}}}^h}\mathit{\varphi }(u)\hfill \\ \hfill & =e^{h{\widehat{\mathcal{L}}}^h}(e^{\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }}-I)\mathit{\varphi }(u)\hfill \\ \hfill & =e^{h{\widehat{\mathcal{L}}}^h}(\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }(u)+\mathcal{O}(h^{4p+2}))\hfill \\ \hfill & =(I+\mathcal{O}(h))(\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }(u)\hfill \\ \hfill & +\mathcal{O}(h^{4p+2})).\hfill \end{array}$$

Thus we have

$$\begin{array}{c}\hfill e^{h{\tilde{\mathcal{L}}}^h}\mathit{\varphi }(u)-e^{h{\widehat{\mathcal{L}}}^h}\mathit{\varphi }(u)=\frac{1}{2}{h}^{2p+1}Q:\mathrm{\nabla }\mathrm{\nabla }\mathit{\varphi }(u)+\mathcal{O}(h^{2p+2}).\end{array}$$ 26

Now using (23), (25), and (26) we obtain

$$\begin{array}{c}\hfill e^{h{\tilde{\mathcal{L}}}^h}\mathit{\varphi }(u)-e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)=\mathcal{O}(h^{2p+2})+\mathcal{O}(h^{q+2+l}).\end{array}$$ 27

Balancing these terms, in what follows we make the choice $l=2p-q$. If $l<0$ we adopt the convention that the drift $f^h$ is simply f. With this choice of q we obtain

$$\begin{array}{c}\hfill e^{h{\tilde{\mathcal{L}}}^h}\mathit{\varphi }(u)-e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)=\mathcal{O}(h^{2p+2}).\end{array}$$ 28

This demonstrates that the error between the Markov kernel of one-step of the SDE (17) and the Markov kernel of the numerical method (2) is of order $\mathcal{O}(h^{2p+2})$. Some straightforward stability considerations show that the weak error over an $\mathcal{O}(1)$ time interval is $\mathcal{O}(h^{2p+1})$. We make assumptions giving this stability and then state a theorem comparing the weak error with respect to the modified Eq. (17), and the original Eq. (1).

Assumption 3

The function f is in $C^{\infty }$ and all its derivatives are uniformly bounded on ${\mathbb{R}}^n$. Furthermore, f is such that the operators $e^{h\mathcal{L}}$ and $e^{h{\mathcal{L}}^h}$ satisfy, for all $\mathit{\psi }\in C^{\infty }({\mathbb{R}}^n,\mathbb{R})$ and some $L>0$,

$$\begin{array}{cc}\hfill \underset{u\in {\mathbb{R}}^n}{sup}|e^{h\mathcal{L}}\mathit{\psi }(u)|& \le (1+Lh)\underset{u\in {\mathbb{R}}^n}{sup}|\mathit{\psi }(u)|,\hfill \\ \hfill \underset{u\in {\mathbb{R}}^n}{sup}|e^{h{\mathcal{L}}^h}\mathit{\psi }(u)|& \le (1+Lh)\underset{u\in {\mathbb{R}}^n}{sup}|\mathit{\psi }(u)|.\hfill \end{array}$$

Remark 2.3

If $p=q$ in what follows (our recommended choice) then the weak order of the method coincides with the strong order; however, measured relative to the modified equation, the weak order is then one plus twice the strong order. In this case, the second part of Theorem 2.2 gives us the first weak order result in Theorem 2.4. Additionally, Assumption 3 is stronger than we need, but allows us to highlight probabilistic ideas whilst keeping overly technical aspects of the numerical analysis to a minimum. More sophisticated, but structurally similar, analysis would be required for weaker assumptions on f. Similar considerations apply to the assumptions on $\mathit{\varphi }$.

Theorem 2.4

Consider the numerical method (10) and assume that Assumptions 1 and 3 are satisfied. Then, for $\mathit{\varphi }\in C^{\infty }$ function with all derivatives bounded uniformly on ${\mathbb{R}}^n$, we have that

$$\begin{array}{c}\hfill |\mathit{\varphi }(u(T))-{\mathbb{E}}^h(\mathit{\varphi }(U_k))|\le K{h}^{min\{2p,q\}},kh=T,\end{array}$$

and

$$\begin{array}{c}\hfill |\mathbb{E}(\mathit{\varphi }(\tilde{u}(T)))-{\mathbb{E}}^h(\mathit{\varphi }(U_k))|\le K{h}^{2p+1},kh=T,\end{array}$$

where u and $\tilde{u}$ solve (1) and (17), respectively.

Example 2.5

Consider the probabilistic integrator derived from the Euler method in dimension $n=1$. We thus have $q=1$, and we hence set $p=1$. The results in Hairer et al. (2006) allow us to calculate $f^h$ with $l=1$. The preceding theory then leads to strong order of convergence 1, measured relative to the true ODE (1), and weak order 3 relative to the SDE

$$\begin{array}{cc}\hfill \mathrm{d}\widehat{u}& =(f(\widehat{u})-\frac{h}{2}{f}^{\prime }(\widehat{u})f(\widehat{u})+\frac{h^2}{12}(f^{\prime \prime }(\widehat{u})f^2(\widehat{u})\hfill \\ \hfill & +4{(f^{\prime }(\widehat{u}))}^{2}f(\widehat{u})))\mathrm{d}t+\sqrt{C}h\mathrm{d}W.\hfill \end{array}$$

These results allow us to constrain the behaviour of the randomised method using limited information about the covariance structure, $C^h$. The randomised solution converges weakly, at a high rate, to a solution that only depends on Q. Hence, we conclude that the practical behaviour of the solution is only dependent upon Q, and otherwise, $C^h$ may be any convenient kernel. With these results now available, the following section provides an empirical study of our probabilistic integrators.

Statistical inference and numerics

This section explores applications of the randomised ODE solvers developed in Sect. 2 to forward and inverse problems. Throughout this section, we use the FitzHugh–Nagumo model to illustrate ideas (Ramsay et al. 2007). This is a two-state non-linear oscillator, with states (V, R) and parameters (a, b, c), governed by the equations

$$\begin{array}{c}\hfill \frac{\mathrm{d}V}{\mathrm{d}t}=c\left(V-\frac{V^3}{3}+R\right),\frac{\mathrm{d}R}{\mathrm{d}t}=-\frac{1}{c}\left(V-a+bR\right).\end{array}$$ 29

This particular example does not satisfy the stringent Assumptions 2 and 3 and the numerical results shown demonstrate that, as indicated in Remarks 2.1 and 2.3, our theory will extend to weaker assumptions on f, something we will address in future work.

Calibrating forward uncertainty propagation

Consider Eq. (29) with fixed initial conditions $V(0)=-1,R(0)=1$, and parameter values (.2, .2, 3). Figure 3 shows draws of the V species trajectories from the measure associated with the probabilistic Euler solver with $p=q=1$, for various values of the step-size and fixed $\mathit{\sigma }=0.1$. The random draws exhibit non-Gaussian structure at large step-size and clearly contract towards the true solution.

Fig. 3.

The true trajectory of the V species of the FitzHugh–Nagumo model (red) and one hundred realisations from a probabilistic Euler ODE solver with various step-sizes and noise scale $\mathit{\sigma }=.1$ (blue)

Although the rate of contraction is governed by the underlying deterministic method, the scale parameter, $\mathit{\sigma }$, completely controls the apparent uncertainty in the solver.6 This tuning problem exists in general, since $\mathit{\sigma }$ is problem dependent and cannot obviously be computed analytically.

Therefore, we propose to calibrate $\mathit{\sigma }$ to replicate the amount of error suggested by classical error indicators. In the following discussion, we often explicitly denote the dependence on h and $\mathit{\sigma }$ with superscripts, hence the probabilistic solver is $U^{h,\mathit{\sigma }}$ and the corresponding deterministic solver is $U^{h,0}$. Define the deterministic error as $e(t)=u(t)-U^{h,0}(t)$. Then we assume there is some computable error indicator $E(t)\approx e(t)$, defining $E_k=E(t_k)$. The simplest error indicators might compare differing step-sizes, $E(t)=U^{h,0}(t)-U^{2h,0}(t)$, or differing order methods, as in a Runge–Kutta 4–5 scheme.

We proceed by constructing a probability distribution $\mathit{\pi }(\mathit{\sigma })$ that is maximised when the desired matching occurs. We estimate this scale matching by comparing: (i) a Gaussian approximation of our random solver at each step k, ${\tilde{\mathit{\mu }}}_k^{h,\mathit{\sigma }}=\mathcal{N}(\mathbb{E}(U_k^{h,\mathit{\sigma }}),\mathbb{V}(U_k^{h,\mathit{\sigma }}));$ and (ii) the natural Gaussian measure from the deterministic solver, $U_k^{h,0}$, and the available error indicator, $E_k$, ${\mathit{\nu }}_k^{\mathit{\sigma }}=\mathcal{N}(U_k^{h,0},{(E_k)}^2).$ We construct $\mathit{\pi }(\mathit{\sigma })$ by penalising the distance between these two normal distributions at every step: $\mathit{\pi }(\mathit{\sigma })\propto {\prod }_{k}exp\left(-d({\tilde{\mathit{\mu }}}_k^{h,\mathit{\sigma }},{\mathit{\nu }}_k^{\mathit{\sigma }})\right)$. We find that the Bhattacharyya distance (closely related to the Hellinger metric) works well (Kailath 1967), since it diverges quickly if either the mean or variance differs. The density can be easily estimated using Monte Carlo. If the ODE state is a vector, we take the product of the univariate Bhattacharyya distances. Note that this calibration depends on the initial conditions and any parameters of the ODE.

Returning to the FitzHugh–Nagumo model, sampling from $\mathit{\pi }(\mathit{\sigma })$ yields strongly peaked, uni-modal posteriors, hence we proceed using ${\mathit{\sigma }}^{\ast }=\text{arg max}\mathit{\pi }(\mathit{\sigma })$. We examine the quality of the scale matching by plotting the magnitudes of the random variation against the error indicator in Fig. 4, observing good agreement of the marginal variances. Note that our measure still reveals non-Gaussian structure and correlations in time not revealed by the deterministic analysis. As described, this procedure requires fixed inputs to the ODE, but it is straightforward to marginalise out a prior distribution over input parameters.

Fig. 4.

A comparison of the error indicator for the V species of the FitzHugh–Nagumo model (blue) and the observed variation in the calibrated probabilistic solver. The red curves depict 50 samples of the magnitude of the difference between a standard Euler solver for several step-sizes and the equivalent randomised variant, using ${\mathit{\sigma }}^{\ast }$, maximising $\mathit{\pi }(\mathit{\sigma })$

Bayesian posterior inference problems

Given the calibrated probabilistic ODE solvers described above, let us consider how to incorporate them into inference problems.

Assume we are interested in inferring parameters of the ODE given noisy observations of the state. Specifically, we wish to infer parameters $\mathit{\theta }\in {\mathbb{R}}^d$ for the differential equation $\dot{u}=f(u,\mathit{\theta })$, with fixed initial conditions $u(t=0)=u_0$ (a straightforward modification may include inference on initial conditions). Assume we are provided with data $d\in {\mathbb{R}}^m$, $d_j=u({\mathit{\tau }}_j)+{\mathit{\eta }}_j$ at some collection of times ${\mathit{\tau }}_j$, corrupted by i.i.d. noise, ${\mathit{\eta }}_j\sim \mathcal{N}(0,\mathit{\Gamma })$. If we have prior $\mathbb{Q}(\mathit{\theta })$, the posterior we wish to explore is, $\mathbb{P}(\mathit{\theta }\mid d)\propto \mathbb{Q}(\mathit{\theta })\mathcal{L}(d,u(\mathit{\theta })),$ where density $\mathcal{L}$ compactly summarises this likelihood model.

The standard computational strategy is to simply replace the unavailable trajectory u with a numerical approximation, inducing approximate posterior ${\mathbb{P}}^{h,0}(\mathit{\theta }\mid d)\propto \mathbb{Q}(\mathit{\theta })\mathcal{L}(d,U^{h,0}(\mathit{\theta })).$ Informally, this approximation will be accurate when the error in the numerical solver is small compared to $\mathit{\Gamma }$ and often converges formally to $\mathbb{P}(\mathit{\theta }\mid d)$ as $h\to 0$ (Dashti and Stuart 2016). However, highly correlated errors at finite h can have substantial impact.

In this work, we are concerned about the undue optimism in the predicted variance, that is, when the posterior concentrates around an arbitrary parameter value even though the deterministic solver is inaccurate and is merely able to reproduce the data by coincidence. The conventional concern is that any error in the solver will be transferred into posterior bias. Practitioners commonly alleviate both concerns by tuning the solver to be nearly perfect, however, we note that this may be computationally prohibitive in many contemporary statistical applications.

We can construct a different posterior that includes the uncertainty in the solver by taking an expectation over random solutions to the ODE

$$\begin{array}{c}\hfill {\mathbb{P}}^{h,\mathit{\sigma }}(\mathit{\theta }\mid d)\propto \mathbb{Q}(\mathit{\theta })\int \mathcal{L}(d,U^{h,\mathit{\sigma }}(\mathit{\theta },\mathit{\xi }))d\mathit{\xi },\end{array}$$ 30

where $U^{h,\mathit{\sigma }}(\mathit{\theta },\mathit{\xi })$ is a draw from the randomised solver given parameters $\mathit{\theta }$ and random draw $\mathit{\xi }$. Intuitively, this construction favours parameters that exhibit agreement with the entire family of uncertain trajectories. The typical effect of this expectation is to increase the posterior uncertainty on $\mathit{\theta }$, preventing the inappropriate posterior collapse we are concerned about. Indeed, if the integrator cannot resolve the underlying dynamics, $h^{p+1/2}\mathit{\sigma }$ will be large. Then $U^{h,\mathit{\sigma }}(\mathit{\theta },\mathit{\xi })$ is independent of $\mathit{\theta }$, hence the prior is recovered, ${\mathbb{P}}^{h,\mathit{\sigma }}(\mathit{\theta }\mid d)\approx \mathbb{Q}(\mathit{\theta })$.

Notice that as $h\to 0$, both the measures ${\mathbb{P}}^{h,0}$ and ${\mathbb{P}}^{h,\mathit{\sigma }}$ typically collapse to the analytic posterior, $\mathbb{P}$, hence both methods are correct. We do not expect the bias of ${\mathbb{P}}^{h,\mathit{\sigma }}$ to be improved, since all of the averaged trajectories are of the same quality as the deterministic solver in ${\mathbb{P}}^{h,0}$. We now construct an analytic inference problem demonstrating these behaviours.

Example 3.1

Consider inferring the initial condition, $u_0\in \mathbb{R}$, of the scalar linear differential equation, $\dot{u}=\mathit{\lambda }u,$ with $\mathit{\lambda }>0.$ We apply a numerical method to produce the approximation $U_k\approx u(kh)$. We observe the state at some times $t=kh$, with additive noise ${\mathit{\eta }}_k\sim \mathcal{N}(0,{\mathit{\gamma }}^2)$: $d_k=U_k+{\mathit{\eta }}_k$. If we use a deterministic Euler solver, the model predicts $U_k={(1+h\mathit{\lambda })}^k{u}_0.$ These model predictions coincide with the slightly perturbed problem

$$\begin{array}{c}\hfill \frac{\mathrm{d}u}{\mathrm{d}t}=h^{-1}log(1+\mathit{\lambda }h)u,\end{array}$$

hence error increases with time. However, the assumed observational model does not allow for this, as the observation variance is ${\mathit{\gamma }}^2$ at all times.

In contrast, our proposed probabilistic Euler solver predicts

$$\begin{array}{c}\hfill U_k={(1+h\mathit{\lambda })}^k{u}_0+\mathit{\sigma }{h}^{3/2}\sum _{j=0}^{k-1}{\mathit{\xi }}_j{(1+\mathit{\lambda }h)}^{k-j-1},\end{array}$$

where we have made the natural choice $p=q$, where $\mathit{\sigma }$ is the problem-dependent scaling of the noise and the ${\mathit{\xi }}_k$ are i.i.d. $\mathcal{N}(0,1).$ For a single observation, ${\mathit{\eta }}_k$ and every ${\mathit{\xi }}_k$ are independent, so we may rearrange the equation to consider the perturbation as part of the observation operator. Hence, a single observation at k has effective variance

$$\begin{array}{cc}\hfill {\mathit{\gamma }}_h^2:=& {\mathit{\gamma }}^2+{\mathit{\sigma }}^2{h}^3\sum _{j=0}^{k-1}{(1+\mathit{\lambda }h)}^{2(k-j-1)}\hfill \\ \hfill =& {\mathit{\gamma }}^2+{\mathit{\sigma }}^2{h}^3\frac{{(1+\mathit{\lambda }h)}^{2k}-1}{{(1+\mathit{\lambda }h)}^2-1}.\hfill \end{array}$$

Thus, late-time observations are modelled as being increasingly inaccurate.

Consider inferring $u_0$, given a single observation $d_k$ at time k. If a Gaussian prior $\mathcal{N}(m_0,{\mathit{\zeta }}_0^2)$ is specified for $u_0$, then the posterior is $\mathcal{N}(m,{\mathit{\zeta }}^2)$, where

$$\begin{array}{cc}& {\mathit{\zeta }}^{-2}=\frac{{(1+h\mathit{\lambda })}^{2k}}{{\mathit{\gamma }}_h^2}+{\mathit{\zeta }}_0^{-2},\hfill \\ \hfill & {\mathit{\zeta }}^{-2}m=\frac{{(1+h\mathit{\lambda })}^k{d}_k}{{\mathit{\gamma }}_h^2}+{\mathit{\zeta }}_0^{-2}{m}_0.\hfill \end{array}$$

The observation precision is scaled by ${(1+h\mathit{\lambda })}^{2k}$ because late-time data contain increasing information. Assume that the data are $d_k=e^{\mathit{\lambda }kh}{u}_0^{†}+\mathit{\gamma }{\mathit{\eta }}^{†}$, for some given true initial condition $u_0^{†}$ and noise realisation ${\mathit{\eta }}^{†}.$ Consider now the asymptotic regime, where h is fixed and $k\to \infty$. For the standard Euler method, where ${\mathit{\gamma }}_h=\mathit{\gamma }$, we see that ${\mathit{\zeta }}^2\to 0$, whilst $m\asymp ({(1+h\mathit{\lambda })}^{-1}{e}^{h\mathit{\lambda }}{)}^k{u}_0^{†}$. Thus the inference scheme becomes increasingly certain of the wrong answer: the variance tends to zero and the mean tends to infinity.

In contrast, with a randomised integrator, the fixed h, large k asymptotics are

$$\begin{array}{cc}& {\mathit{\zeta }}^2\asymp \frac{1}{{\mathit{\zeta }}_0^{-2}+\mathit{\lambda }(2+\mathit{\lambda }h){\mathit{\sigma }}^{-2}{h}^{-2}},\hfill \\ \hfill & m\asymp \frac{{\left({(1+h\mathit{\lambda })}^{-1}{e}^{h\mathit{\lambda }}\right)}^k{u}_0^{†}}{1+{\mathit{\zeta }}_0^{-2}{\mathit{\sigma }}^2{h}^2{\mathit{\lambda }}^{-1}{(2+\mathit{\lambda }h)}^{-1}}.\hfill \end{array}$$

Thus, the mean blows up at a modified rate, but the variance remains positive.

We take an empirical Bayes approach to choosing $\mathit{\sigma }$, that is, using a constant, fixed value ${\mathit{\sigma }}^{\ast }=\text{arg max}\mathit{\pi }(\mathit{\sigma })$, chosen before the data are observed. Joint inference of the parameters and the noise scale suffer from well-known MCMC mixing issues in Bayesian hierarchic models. To handle the unknown parameter $\mathit{\theta }$, we can marginalise it out using the prior distribution, or in simple problems, it may be reasonable to choose a fixed representative value.

We now return to the FitzHugh–Nagumo model; given fixed initial conditions, we attempt to recover parameters $\mathit{\theta }=(a,b,c)$ from observations of both species at times $\mathit{\tau }=1,2,\dots ,40$. The priors are log-normal, centred on the true value with unit variance, and with observational noise $\mathit{\Gamma }=0.001$. The data are generated from a high-quality solution, and we perform inference using Euler integrators with various step-sizes, $h\in \{0.005,0.01,0.02,0.05,0.1\}$, spanning a range of accurate and inaccurate integrators.

We first perform the inferences with naive use of deterministic Euler integrators. We simulate from each posterior using delayed rejection MCMC (Haario et al. 2006), shown in Fig. 5. Observe the undesirable concentration of every posterior, even those with poor solvers; the posteriors are almost mutually singular, hence clearly the posterior widths are meaningless.

Fig. 5.

The posterior marginals of the FitzHugh–Nagumo inference problem using deterministic integrators with various step-sizes

Secondly, we repeat the experiment using our probabilistic Euler integrators, with results shown in Fig. 6. We use a noisy pseudomarginal MCMC method, whose fast mixing is helpful for these initial experiments (Medina-Aguayo et al. 2015). These posteriors are significantly improved, exhibiting greater mutual agreement and obvious increasing concentration with improving solver quality. The posteriors are not perfectly nested, possible evidence that our choice of scale parameter is imperfect, or that the assumption of locally Gaussian error deteriorates for large step-sizes. Note that the bias of ${\mathit{\theta }}_3$ is essentially unchanged with the randomised integrator, but the posterior for ${\mathit{\theta }}_2$ broadens and is correlated to ${\mathit{\theta }}_3$, hence introduces a bias in the posterior mode; without randomisation, only the inappropriate certainty about ${\mathit{\theta }}_3$ allowed the marginal for ${\mathit{\theta }}_2$ to exhibit little bias.

Fig. 6.

The posterior marginals of the FitzHugh–Nagumo inference problem using probabilistic integrators with various step-sizes

Probabilistic solvers for partial differential equations

We now turn to present a framework for probabilistic solutions to partial differential equations, working within the finite element setting. Our discussion closely resembles the ODE case, except that now we randomly perturb the finite element basis functions.

Probabilistic finite element method for variational problems

Let $\mathcal{V}$ be a Hilbert space of real-valued functions defined on a bounded polygonal domain $D\subset {\mathbb{R}}^d$. Consider a weak formulation of a linear PDE specified via a symmetric bilinear form $a:\mathcal{V}\times \mathcal{V}⟶\mathbb{R}$, and a linear form $r:\mathcal{V}⟶\mathbb{R}$ to give the problem of finding $u\in \mathcal{V}:a(u,v)=r(v),\forall v\in \mathcal{V}.$ This problem can be approximated by specifying a finite-dimensional subspace ${\mathcal{V}}^h\subset \mathcal{V}$ and seeking a solution in ${\mathcal{V}}^h$ instead. This leads to a finite-dimensional problem to be solved for the approximation U:

$$\begin{array}{c}\hfill U\in {\mathcal{V}}^h:a(U,v)=r(v),\forall v\in {\mathcal{V}}^h.\end{array}$$ 31

This is known as the Galerkin method.

We will work in the setting of finite element methods, assuming that ${\mathcal{V}}^h=\mathrm{span}{\{{\mathit{\varphi }}_j\}}_{j=1}^J$, where ${\mathit{\varphi }}_j$ is locally supported on a grid of points ${\{x_j\}}_{j=1}^J.$ The parameter h is introduced to measure the diameter of the finite elements. We will also assume that

$$\begin{array}{c}\hfill {\mathit{\varphi }}_j(x_k)={\mathit{\delta }}_{jk}.\end{array}$$ 32

Any element $U\in {\mathcal{V}}^h$ can then be written as

$$\begin{array}{c}\hfill U(x)=\sum _{j=1}^J{U}_j{\mathit{\varphi }}_j(x)\end{array}$$ 33

from which it follows that $U(x_k)=U_k.$ The Galerkin method then gives $A\mathsf{U}=\mathsf{r}$, for $\mathsf{U}={(U_1,\dots ,U_J)}^T$, $A_{jk}=a({\mathit{\varphi }}_j,{\mathit{\varphi }}_k)$, and ${\mathsf{r}}_k=r({\mathit{\varphi }}_k).$

In order to account for uncertainty introduced by the numerical method, we will assume that each basis function ${\mathit{\varphi }}_j$ can be split into the sum of a systematic part ${\mathit{\varphi }}_j^{\mathsf{s}}$ and random part ${\mathit{\varphi }}_j^{\mathsf{r}}$, where both ${\mathit{\varphi }}_j$ and ${\mathit{\varphi }}_j^{\mathsf{s}}$ satisfy the nodal property (32), hence ${\mathit{\varphi }}_j^{\mathsf{r}}(x_k)=0$. Furthermore, we assume that each ${\mathit{\varphi }}_j^{\mathsf{r}}$ shares the same compact support as the corresponding ${\mathit{\varphi }}_j^{\mathsf{s}}$, preserving the sparsity structure of the underlying deterministic method.

Strong convergence result

As in the ODE case, we begin our convergence analysis with assumptions constraining the random perturbations and the underlying deterministic approximation. The bilinear form $a(·,·)$ is assumed to induce an inner product, and then norm via ${\Vert ·\Vert }_a^2=a(·,·);$ furthermore, we assume that this norm is equivalent to the norm on $\mathcal{V}$. Throughout, ${\mathbb{E}}^h$ denotes expectation with respect to the random basis functions.

Assumption 4

The collection of random basis functions ${\{{\mathit{\varphi }}_j^{\mathsf{r}}\}}_{j=1}^J$ are independent, zero-mean, Gaussian random fields, each of which satisfies ${\mathit{\varphi }}_j^{\mathsf{r}}(x_k)=0$ and shares the same support as the corresponding systematic basis function ${\mathit{\varphi }}_j^{\mathsf{s}}.$ For all j, the number of basis functions with index k which share the support of the basis functions with index j is bounded independently of J, the total number of basis functions. Furthermore, the basis functions are scaled so that ${\sum }_{j=1}^J{\mathbb{E}}^h{\Vert {\mathit{\varphi }}_j^{\mathsf{r}}\Vert }_a^2\le C{h}^{2p}.$

Assumption 5

The true solution u of problem (4.1) is in $L^{\infty }(D).$ Furthermore, the standard deterministic interpolant of the true solution, defined by $v^{\mathsf{s}}:={\sum }_{j=1}^{J}u(x_j){\mathit{\varphi }}_j^{\mathsf{s}},$ satisfies $\Vert u-v^{\mathsf{s}}{\Vert }_a\le C{h}^q.$

Theorem 4.1

Under Assumptions 4 and 5 it follows that the approximation U, given by (31), satisfies

$$\begin{array}{c}\hfill {\mathbb{E}}^h{\Vert u-U\Vert }_a^2\le C{h}^{2min\{p,q\}}.\end{array}$$

As for ODEs, the solver accuracy is limited by either the amount of noise injected or the convergence rate of the underlying deterministic method, making $p=q$ the natural choice.

Poisson solver in two dimensions

Consider a Poisson equation with Dirichlet boundary conditions in dimension $d=2$, namely

$$\begin{array}{cc}& -▵u=f,x\in D,\hfill \\ \hfill & u=0,x\in \mathit{\partial }D.\hfill \end{array}$$

We set $\mathcal{V}=H_0^1(D)$ and H to be the space $L^2(D)$ with inner product $⟨·,·⟩$ and resulting norm ${|·|}^2=⟨·,·⟩.$ The weak formulation of the problem has the form (4.1) with

$$\begin{array}{c}\hfill a(u,v)={\int }_D\mathrm{\nabla }u(x)\mathrm{\nabla }v(x)\mathrm{d}x,r(v)=⟨f,v⟩.\end{array}$$

Now consider piecewise linear finite elements satisfying the assumptions of Sect. 4.2 in Johnson (2012) and take these to comprise the set ${\{{\mathit{\varphi }}_j^{\mathsf{s}}\}}_{j=1}^J.$ Then h measures the width of the triangulation of the finite element mesh. Assuming that $f\in H$ it follows that $u\in H^2(D)$ and that

$$\begin{array}{c}\hfill \Vert u-v^{\mathsf{s}}{\Vert }_a\le Ch{\Vert u\Vert }_{H^2}.\end{array}$$ 34

Thus $q=1$. We choose random basis members ${\{{\mathit{\varphi }}_j^{\mathsf{r}}\}}_{j=1}^J$ so that Assumption 4 hold with $p=1$. Theorem 4.1 then shows that, for $e=u-U$, ${\mathbb{E}}^h{\Vert e\Vert }_a^2\le C{h}^2.$ We note that that in the deterministic case, we expect an improved rate of convergence in the function space H. Such a result can be shown to hold in our setting, following the usual arguments for the Aubin–Nitsche trick Johnson (2012), which is available in the supplementary materials.

PDE inference and numerics

We now perform numerical experiments using probabilistic solvers for elliptic PDEs. Specifically, we perform inference in a 1D elliptic PDE, $\mathrm{\nabla }·(\mathit{\kappa }(x)\mathrm{\nabla }u(x))=4x$ for $x\in [0,1]$, given boundary conditions $u(0)=0,u(1)=2$. We represent $log\mathit{\kappa }$ as piecewise constant over ten equal-sized intervals; the first, on $x\in [0,.1)$ is fixed to be one to avoid non-identifiability issues, and the other nine are given a prior ${\mathit{\theta }}_i=log{\mathit{\kappa }}_i\sim \mathcal{N}(0,1)$. Observations of the field u are provided at $x=(0.1,0.2,\dots 0.9)$, with i.i.d. Gaussian error, $\mathcal{N}(0,{10}^{-5})$; the simulated observations were generated using a fine grid and quadratic finite elements, then perturbed with error from this distribution.

Again we investigate the posterior produced at various grid sizes, using both deterministic and randomised solvers. The randomised basis functions are draws from a Brownian bridge conditioned to be zero at the nodal points, implemented in practice with a truncated Karhunen–Loève expansion. The covariance operator may be viewed as a fractional Laplacian, as discussed in Lindgren et al. (2011). The scaling $\mathit{\sigma }$ is again determined by maximising the distribution described in Sect. 3.1, where the error indicator compares linear to quadratic basis functions, and we marginalise out the prior over the ${\mathit{\kappa }}_i$ values.

The posteriors are depicted in Figs. 7 and 8. As in the ODE examples, the deterministic solvers lead to incompatible posteriors for varying grid sizes. In contrast, the randomised solvers suggest increasing confidence as the grid is refined, as desired. The coarsest grid size uses an obviously inadequate ten elements, but this is only apparent in the randomised posterior.

Fig. 7.

The marginal posterior distributions for the first four coefficients in 1D elliptic inverse problem using a classic deterministic solver with various grid sizes

Fig. 8.

The marginal posterior distributions for the first four coefficients in 1D elliptic inverse problem using a randomised solver with various grid sizes

Conclusions

We have presented a computational methodology, backed by rigorous analysis, which enables quantification of the uncertainty arising from the finite-dimensional approximation of solutions of differential equations. These methods play a natural role in statistical inference problems as they allow for the uncertainty from discretisation to be incorporated alongside other sources of uncertainty such as observational noise. We provide theoretical analyses of the probabilistic integrators which form the backbone of our methodology. Furthermore we demonstrate empirically that they induce more coherent inference in a number of illustrative examples. There are a variety of areas in the sciences and engineering which have the potential to draw on the methodology introduced including climatology, computational chemistry, and systems biology.

Our key strategy is to make assumptions about the local behaviour of solver error, which we have assumed to be Gaussian, and to draw samples from the global distribution of uncertainty over solutions that results. Section 2.4 describes a universality result, simplifying task of choosing covariance kernels in practice, within the family of Gaussian processes. However, assumptions of Gaussian error, even locally, may not be appropriate in some cases, or may neglect important domain knowledge. Our framework can be extended in future work to consider alternate priors on the error, for example, multiplicative or non-negative errors.

Our study highlights difficult decisions practitioners face, regarding how to expend computational resources. While standard techniques perform well when the solver is highly converged, our results show standard techniques can be disastrously wrong when the solver is not converged. As the measure of convergence is not a standard numerical analysis one, but a statistical one, we have argued that it can be surprisingly difficult to determine in advance which regime a particular problem resides in. Therefore, our practical recommendation is that the lower cost of the standard approach makes it preferable when it is certain that the numerical method is strongly converged with respect to the statistical measure of interest. Otherwise, the randomised method we propose provides a robust and consistent approach to address the error introduced into the statistical task by numerical solver error. In difficult problem domains, such as numerical weather prediction, the focus has typically been on reducing the numerical error in each solver run; techniques such as these may allow a difference balance between numerical and statistical computing effort in the future.

The prevailing approach to model error described in Kennedy and O’Hagan (2001) is based on a non-intrusive methodology where the effect of model discrepancy is allowed for in observation space. Our intrusive randomisation of deterministic methods for differential equations can be viewed as a highly specialised discrepancy model, designed using our intimate knowledge of the structure and properties of numerical methods. In this vein, we intend to extend this work to other types of model error, where modifying the internal structure of the models can produce computationally and analytically tractable measures of uncertainty which perform better than non-intrusive methods. Our future work will continue to study the computational challenges and opportunities presented by these techniques.

Electronic supplementary material

Below is the link to the electronic supplementary material.

A Numerical analysis details and proofs

Proof

(Theorem 2.2) We first derive the convergence result on the grid, and then in continuous time. From (10) we have

$$\begin{array}{c}\hfill U_{k+1}={\mathit{\Psi }}_h(U_k)+{\mathit{\xi }}_k(h)\end{array}$$ 35

whilst we know that

$$\begin{array}{c}\hfill u_{k+1}={\mathit{\Phi }}_h(u_k).\end{array}$$ 36

Define the truncation error ${\mathit{ϵ}}_k={\mathit{\Psi }}_h(U_k)-{\mathit{\Phi }}_h(U_k)$ and note that

$$\begin{array}{c}\hfill U_{k+1}={\mathit{\Phi }}_h(U_k)+{\mathit{ϵ}}_k+{\mathit{\xi }}_k(h).\end{array}$$ 37

Subtracting Eq. (37) from (36) and defining $e_k=u_k-U_k$, we get

$$\begin{array}{c}\hfill e_{k+1}={\mathit{\Phi }}_h(u_k)-{\mathit{\Phi }}_h(u_k-e_k)-{\mathit{ϵ}}_k-{\mathit{\xi }}_k(h).\end{array}$$

Taking the Euclidean norm and expectations give, using Assumption 1 and the independence of the ${\mathit{\xi }}_k$,

$$\begin{array}{c}\hfill {\mathbb{E}}^h{|e_{k+1}|}^2={\mathbb{E}}^h|{\mathit{\Phi }}_h(u_k)-{\mathit{\Phi }}_h(u_k-e_k)-{\mathit{ϵ}}_k{|}^2+\mathcal{O}(h^{2p+1}),\end{array}$$

where the constant in the $\mathcal{O}(h^{2p+1})$ term is uniform in $k:0\le kh\le T.$ Assumption 2 implies that ${\mathit{ϵ}}_k=\mathcal{O}(h^{q+1}),$ again uniformly in $k:0\le kh\le T.$ Noting that ${\mathit{\Phi }}_h$ is globally Lipschitz with constant bounded by $1+Lh$ under Assumption 2, we then obtain

$$\begin{array}{cc}\hfill {\mathbb{E}}^h{|e_{k+1}|}^2\le & {(1+Lh)}^2{\mathbb{E}}^h{|e_k|}^2\hfill \\ \hfill & +{\mathbb{E}}^h|\langle h^{\frac{1}{2}}({\mathit{\Phi }}_h(u_k)-{\mathit{\Phi }}_h(u_k-e_k)),h^{-\frac{1}{2}}{\mathit{ϵ}}_k\rangle |\hfill \\ \hfill & +\mathcal{O}(h^{2q+2})+\mathcal{O}(h^{2p+1}).\hfill \end{array}$$

Using Cauchy–Schwarz on the inner product, and the fact that ${\mathit{\Phi }}_h$ is Lipschitz with constant bounded independently of h, we get

$$\begin{array}{c}\hfill {\mathbb{E}}^h|e_{k+1}{|}^2\le (1+\mathcal{O}(h)){\mathbb{E}}^h{|e_k|}^2+\mathcal{O}(h^{2q+1})+\mathcal{O}(h^{2p+1}).\end{array}$$

Application of the Gronwall inequality gives the desired result.

Now we turn to continuous time. We note that, for $s\in [t_k,t_{k+1})$,

$$\begin{array}{cc}\hfill U(s)& ={\mathit{\Psi }}_{s-t_k}(U_k)+{\mathit{\xi }}_k(s-t_k),\hfill \\ \hfill u(s)& ={\mathit{\Phi }}_{s-t_k}(u_k).\hfill \end{array}$$

Let ${\mathcal{F}}_t$ denote the $\mathit{\sigma }$-algebra of events generated by the $\{{\mathit{\xi }}_k\}$ up to time t. Subtracting we obtain, using Assumptions 1 and 2 and the fact that ${\mathit{\Phi }}_{s-t_k}$ has Lipschitz constant of the form $1+\mathcal{O}(h)$,

$$\begin{array}{cc}& {\mathbb{E}}^h(|U(s)-u(s)||{\mathcal{F}}_{t_k})\hfill \\ \hfill & \le |{\mathit{\Phi }}_{s-t_k}(U_k)-{\mathit{\Phi }}_{s-t_k}(u_k)|\hfill \\ \hfill & +|{\mathit{\Psi }}_{s-t_k}(U_k)-{\mathit{\Phi }}_{s-t_k}(U_k)|\hfill \\ \hfill & +{\mathbb{E}}^h(|{\mathit{\xi }}_k(s-t_k)||{\mathcal{F}}_{t_k})\hfill \\ \hfill & \le (1+Lh)|e_k|+\mathcal{O}(h^{q+1})+{\mathbb{E}}^h|{\mathit{\xi }}_k(s-t_k)|\hfill \\ \hfill & \le (1+Lh)|e_k|+\mathcal{O}(h^{q+1})\hfill \\ \hfill & +({\mathbb{E}}^h{|{\mathit{\xi }}_k(s-t_k)|}^2{)}^{\frac{1}{2}}\hfill \\ \hfill & \le (1+Lh)|e_k|+\mathcal{O}(h^{q+1})+\mathcal{O}\left(h^{p+\frac{1}{2}}\right).\hfill \end{array}$$

Now taking expectations we obtain

$$\begin{array}{cc}\hfill {\mathbb{E}}^h|U(s)-u(s)|\le & (1+Lh)({\mathbb{E}}^h{|e_k|}^2{)}^{\frac{1}{2}}+\mathcal{O}(h^{q+1})\hfill \\ \hfill & +\mathcal{O}\left(h^{p+\frac{1}{2}}\right).\hfill \end{array}$$

Using the on-grid error bound gives the desired result, after noting that the constants appearing are uniform in $0\le kh\le T.$ $\square$

Proof

(Theorem 2.4) We prove the second bound first. Let $w_k=\mathbb{E}(\mathit{\varphi }(\tilde{u}(t_k))|\tilde{u}(0)=u)$ and $W_k={\mathbb{E}}^h(\mathit{\varphi }(U_k)|U_0=u).$ Then let ${\mathit{\delta }}_k={sup}_{u\in {\mathbb{R}}^n}|W_k-w_k|.$ It follows from the Markov property that

$$\begin{array}{cc}\hfill W_{k+1}-w_{k+1}& =e^{h{\mathcal{L}}^h}{W}_k-e^{h{\tilde{\mathcal{L}}}^h}{w}_k\hfill \\ \hfill & =e^{h{\mathcal{L}}^h}{W}_k-e^{h{\mathcal{L}}^h}{w}_k\hfill \\ \hfill & +(e^{h{\mathcal{L}}^h}{w}_k-e^{h{\tilde{\mathcal{L}}}^h}{w}_k).\hfill \end{array}$$

Using (28) and Assumption 3 we obtain

$$\begin{array}{c}\hfill {\mathit{\delta }}_{k+1}\le (1+Lh){\mathit{\delta }}_k+\mathcal{O}(h^{2p+2}).\end{array}$$

Iterating and employing the Gronwall inequality gives the second error bound.

Now we turn to the first error bound, comparing with the solution u of the original Eq. (1). From (25) and then (21) we see that

$$\begin{array}{cc}\hfill e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)-\mathit{\varphi }({\mathit{\Psi }}_h(u))& =\mathcal{O}(h^{2p+1}),\hfill \\ \hfill e^{h\mathcal{L}}\mathit{\varphi }(u)-e^{h{\mathcal{L}}^h}\mathit{\varphi }(u)& =\mathcal{O}(h^{min\{2p+1,q+1\}}).\hfill \end{array}$$

This gives the first weak error estimate, after using the stability estimate on $e^{h\mathcal{L}}$ from Assumption 3. $\square$

Proof (Theorem 4.1) Recall the Galerkin orthogonality property which follows from subtracting the approximate variational principle from the true variational principle: it states that, for $e=u-U$,

$$\begin{array}{c}\hfill a(e,v)=0,\forall v\in {\mathcal{V}}^h.\end{array}$$ 38

From this it follows that

$$\begin{array}{c}\hfill {\Vert e\Vert }_a\le {\Vert u-v\Vert }_a,\forall v\in {\mathcal{V}}^h.\end{array}$$ 39

To see this note that, for any $v\in {\mathcal{V}}^h$, the orthogonality property (38) gives

$$\begin{array}{c}\hfill a(e,e)=a(e,e+U-v)=a(e,u-v).\end{array}$$ 40

Thus, by Cauchy–Schwarz, ${\Vert e\Vert }_a^2\le {\Vert e\Vert }_a{\Vert u-v\Vert }_a,\forall v\in {\mathcal{V}}^h$ implying (39). We now set, for $v\in \mathcal{V}$,

$$\begin{array}{cc}\hfill v(x)=& \sum _{j=1}^{J}u(x_j){\mathit{\varphi }}_j(x)\hfill \\ \hfill =& \sum _{j=1}^{J}u(x_j){\mathit{\varphi }}_j^{\mathsf{s}}(x)+\sum _{j=1}^{J}u(x_j){\mathit{\varphi }}_j^{\mathsf{r}}(x)\hfill \\ \hfill =:& v^{\mathsf{s}}(x)+v^{\mathsf{r}}(x).\hfill \end{array}$$

By the mean-zero and independence properties of the random basis functions we deduce that

$$\begin{array}{cc}\hfill {\mathbb{E}}^h{\Vert u-v\Vert }_a^2=& {\mathbb{E}}^{h}a(u-v,u-v)\hfill \\ \hfill =& {\mathbb{E}}^{h}a(u-v^{\mathsf{s}},u-v^{\mathsf{s}})+{\mathbb{E}}^{h}a(v^{\mathsf{r}},v^{\mathsf{r}})\hfill \\ \hfill =& \Vert u-v^{\mathsf{s}}{\Vert }_a^2+\sum _{j=1}^{J}u{(x_j)}^2{\mathbb{E}}^h{\Vert {\mathit{\varphi }}_j^{\mathsf{r}}\Vert }_a^2.\hfill \end{array}$$

The result follows from Assumptions 4 and 5. $\square$

Ornstein–Uhlenbeck integrator

An additional example of a randomised integrator is an integrated Ornstein–Uhlenbeck process, derived as follows. Define, on the interval $s\in [t_k,t_{k+1})$, the pair of equations

$$\begin{array}{cc}\hfill \mathrm{d}U& =V\mathrm{d}t,U(t_k)=U_k,\hfill \end{array}$$ 41a

$$\begin{array}{cc}\hfill \mathrm{d}V& =-\mathit{\Lambda }V\mathrm{d}t+\sqrt{2\mathit{\Sigma }}\mathrm{d}W,V(t_k)=f(U_k).\hfill \end{array}$$ 41b

Here W is a standard Brownian motion and $\mathit{\Lambda }$ and $\mathit{\Sigma }$ are invertible matrices, possibly depending on h. The approximating function $g^h(s)$ is thus defined by V(s), an Ornstein–Uhlenbeck process.

Integrating (41b) we obtain

$$\begin{array}{c}\hfill V(s)=exp(-\mathit{\Lambda }(s-t_k))f(U_k)+{\mathit{\chi }}_k(s-t_k),\end{array}$$ 42

where $s\in [t_k,t_{k+1})$ and the $\{{\mathit{\chi }}_k\}$ form an i.i.d. sequence of Gaussian random functions defined on [0, h] with

$$\begin{array}{c}\hfill {\mathit{\chi }}_k(s)=\sqrt{2\mathit{\Sigma }}{\int }_0^{s}exp(\mathit{\Lambda }(\mathit{\tau }-s))\mathrm{d}W(\mathit{\tau }).\end{array}$$

Note that the h-dependence of $C^h$ comes through the time interval on which ${\mathit{\chi }}_k$ is defined, and through $\mathit{\Lambda }$ and $\mathit{\Sigma }$.

Integrating (41a), using (42), we obtain

$$\begin{array}{cc}\hfill U(s)=& U_k+{\mathit{\Lambda }}^{-1}\left(I-exp(-\mathit{\Lambda }(s-t_k))\right)f(U_k)\hfill \\ \hfill & +{\mathit{\xi }}_k(s-t_k),\hfill \end{array}$$ 43

where $s\in [t_k,t_{k+1}]$, and, for $t\in [0,h]$,

$$\begin{array}{c}\hfill {\mathit{\xi }}_k(t)={\int }_0^t{\mathit{\chi }}_k(\mathit{\tau })\mathrm{d}\mathit{\tau }.\end{array}$$ 44

The numerical method (43) may be written in the form (12), and hence (10) at the grid points, with the definition

$$\begin{array}{c}\hfill {\mathit{\Psi }}_h(u)=u+{\mathit{\Lambda }}^{-1}(I-exp(-\mathit{\Lambda }h))f(u).\end{array}$$

This integrator is first-order accurate and satisfies Assumption 2 with $p=1$. Choosing to scale $\mathit{\Sigma }$ with h so that $q\ge 1$ in Assumption 1 leads to convergence of the numerical method with order 1.

Had we carried out the above analysis in the case $\mathit{\Lambda }=0$ we would have obtained the probabilistic Euler method (14), and hence (13) at grid points, used as our canonical example in the earlier developments.

Convergence rate of $L_2$ convergence

When considering the Poisson problem in two dimensions, as discussed in Sect. 4, we expect an improved rate of convergence in the function space H. We now show that such a result also holds in our random setting.

Note that, under Assumption 4, if we introduce ${\mathit{\gamma }}_{jk}$ that is 1 when two basis functions have overlapping support, and 0 otherwise, then ${\mathit{\gamma }}_{jk}$ is symmetric and there is constant C, independent of j and J, such that ${\sum }_{k=1}^J{\mathit{\gamma }}_{jk}\le C.$ Now let $\mathit{\phi }$ solve the equation $a(\mathit{\phi },v)=⟨e,v⟩,\forall v\in \mathcal{V}.$ Then ${\Vert \mathit{\phi }\Vert }_{H^2}\le C|e|.$ We define ${\mathit{\phi }}^{\mathsf{s}}$ and ${\mathit{\phi }}^{\mathsf{r}}$ in analogy with the definitions of $v^{\mathsf{s}}$ and $v^{\mathsf{r}}$. Following the usual arguments for application of the Aubin–Nitsche trick Johnson (2012), we have ${|e|}^2=a(e,\mathit{\phi })=a(e,\mathit{\phi }-{\mathit{\phi }}^{\mathsf{s}}-{\mathit{\phi }}^{\mathsf{r}}).$ Thus

$$\begin{array}{cc}\hfill {|e|}^2\le & {\Vert e\Vert }_a{\Vert \mathit{\phi }-{\mathit{\phi }}^{\mathsf{s}}-{\mathit{\phi }}^{\mathsf{r}}\Vert }_a\hfill \\ \hfill \le & \sqrt{2}{\Vert e\Vert }_a(\Vert \mathit{\phi }-{\mathit{\phi }}^{\mathsf{s}}{\Vert }_a^2+{\Vert {\mathit{\phi }}^{\mathsf{r}}\Vert }_a^2{)}^{\frac{1}{2}}.\hfill \end{array}$$ 45

We note that ${\mathit{\phi }}^{\mathsf{r}}(x)={\sum }_{j=1}^J\mathit{\phi }(x_j){\mathit{\varphi }}_j^{\mathsf{r}}(x)={\Vert \mathit{\phi }\Vert }_{H^2}{\sum }_{j=1}^J{a}_j{\mathit{\varphi }}_j^{\mathsf{r}}(x)$ where, by Sobolev embedding ($d=2$ here), $a_j:=\mathit{\phi }(x_j)/{\Vert \mathit{\phi }\Vert }_{H^2}$ satisfies ${max}_{1\le j\le J}|a_j|\le C.$ Note, however, that the $a_j$ are random and correlated with all of the random basis functions. Using this, together with (34), in (45), we obtain

$$\begin{array}{c}\hfill {|e|}^2\le {C\Vert e\Vert }_a(h^2+\Vert \sum _{j=1}^J{a}_j{\mathit{\varphi }}_j^{\mathsf{r}}(x){\Vert }_a^2{)}^{\frac{1}{2}}{\Vert \mathit{\phi }\Vert }_{H^2}.\end{array}$$

We see that

$$\begin{array}{c}\hfill |e|\le {C\Vert e\Vert }_a(h^2+\sum _{j=1}^J\sum _{k=1}^J{a}_j{a}_{k}a({\mathit{\varphi }}_j^{\mathsf{r}},{\mathit{\varphi }}_k^{\mathsf{r}}){)}^{\frac{1}{2}}.\end{array}$$

From this and the symmetry of ${\mathit{\gamma }}_{jk}$, we obtain

$$\begin{array}{cc}\hfill |e|& \le {C\Vert e\Vert }_a(h^2+\sum _{j=1}^J\sum _{k=1}^J{\mathit{\gamma }}_{jk}(\Vert {\mathit{\varphi }}_j^{\mathsf{r}}{\Vert }_a^2+{\Vert {\mathit{\varphi }}_k^{\mathsf{r}}\Vert }_a^2){)}^{\frac{1}{2}}\hfill \\ \hfill & \le {C\Vert e\Vert }_a(h^2+2C\sum _{j=1}^J{\Vert {\mathit{\varphi }}_j^{\mathsf{r}}\Vert }_a^2{)}^{\frac{1}{2}}.\hfill \end{array}$$

Taking expectations, using that $p=q=1$, we find, using Assumption 4, that ${\mathbb{E}}^h|e|\le Ch({\mathbb{E}}^h{\Vert e\Vert }_a^2{)}^{\frac{1}{2}}\le C{h}^2$ as desired. Thus we recover the extra order of convergence over the rate 1 in the ${\Vert ·\Vert }_a$ norm (although the improved rate is in $L^1(\mathit{\Omega };H)$ whilst the lower rate of convergence is in $L^2(\mathit{\Omega };\mathcal{V}).$).