Blended particle filters for large-dimensional chaotic dynamical systems

Significance

Combining large uncertain computational models with big noisy datasets is a formidable problem throughout science and engineering. These are especially difficult issues when real-time state estimation and prediction are needed such as, for example, in weather forecasting. Thus, a major challenge in contemporary data science is the development of statistically accurate particle filters to capture non-Gaussian features in large-dimensional chaotic dynamical systems. New blended particle filters are developed in this paper. These algorithms exploit the physical structure of turbulent dynamical systems and capture non-Gaussian features in an adaptively evolving low-dimensional subspace through particles interacting with evolving Gaussian statistics on the remaining portion of the phase space.

Abstract

A major challenge in contemporary data science is the development of statistically accurate particle filters to capture non-Gaussian features in large-dimensional chaotic dynamical systems. Blended particle filters that capture non-Gaussian features in an adaptively evolving low-dimensional subspace through particles interacting with evolving Gaussian statistics on the remaining portion of phase space are introduced here. These blended particle filters are constructed in this paper through a mathematical formalism involving conditional Gaussian mixtures combined with statistically nonlinear forecast models compatible with this structure developed recently with high skill for uncertainty quantification. Stringent test cases for filtering involving the 40-dimensional Lorenz 96 model with a 5-dimensional adaptive subspace for nonlinear blended filtering in various turbulent regimes with at least nine positive Lyapunov exponents are used here. These cases demonstrate the high skill of the blended particle filter algorithms in capturing both highly non-Gaussian dynamical features as well as crucial nonlinear statistics for accurate filtering in extreme filtering regimes with sparse infrequent high-quality observations. The formalism developed here is also useful for multiscale filtering of turbulent systems and a simple application is sketched below.

Many contemporary problems in science ranging from protein folding in molecular dynamics to scaling up of small-scale effects in nanotechnology to making accurate predictions of the coupled atmosphere–ocean system involve partial observations of extremely complicated large-dimensional chaotic dynamical systems. Filtering is the process of obtaining the best statistical estimate of a natural system from partial observations of the true signal from nature. In many contemporary applications in science and engineering, real-time filtering or data assimilation of a turbulent signal from nature involving many degrees of freedom is needed to make accurate predictions of the future state.

Particle filtering of low-dimensional dynamical systems is an established discipline (1). When the system is low dimensional, Monte Carlo approaches such as the particle filter with its various up-to-date resampling strategies (2) provide better estimates than the Kalman filter in the presence of strong nonlinearity and highly non-Gaussian distributions. However, these accurate nonlinear particle filtering strategies are not feasible for large-dimensional chaotic dynamical systems because sampling a high-dimensional variable is computationally impossible for the foreseeable future. Recent mathematical theory strongly supports this curse of dimensionality for particle filters (3, 4). In the second direction, Bayesian hierarchical modeling (5) and reduced-order filtering strategies (6–8) based on the Kalman filter (9) have been developed with some success in these extremely complex high-dimensional nonlinear systems. There is an inherently difficult practical issue of small ensemble size in filtering statistical solutions of these complex problems due to the large computational overload in generating individual ensemble members through the forward dynamical operator. Numerous ensemble-based Kalman filters (10–14) show promising results in addressing this issue for synoptic-scale midlatitude weather dynamics by imposing suitable spatial localization on the covariance updates; however, all these methods are very sensitive to model resolution, observation frequency, and the nature of the turbulent signals when a practical limited ensemble size (typically less than 100) is used.

Recent attempts without a major breakthrough to use particle filters with small ensemble size directly on large-dimensional chaotic dynamical systems are surveyed in chap. 15 of ref. 15. The goal here is to develop blended particle filters for large-dimensional chaotic dynamical systems. For the blended particle filters developed below for a state vector $\mathit{u}\in {\mathrm{ℝ}}^N$, there are two subspaces that typically evolve adaptively in time, where $\mathit{u}=\left({\mathit{u}}_1,{\mathit{u}}_2\right)$, ${\mathit{u}}_j\in {\mathrm{ℝ}}^{N_j}$, and $N_1+N_2=N$, with the property that $N_1$ is low dimensional enough so that the non-Gaussian statistics of ${\mathit{u}}_1$ can be calculated from a particle filter whereas the evolving statistics of ${\mathit{u}}_2$ are conditionally Gaussian given ${\mathit{u}}_1$. Statistically nonlinear forecast models with this structure with high skill for uncertainty quantification have been developed recently by Sapsis and Majda (16–19) and are used below in the blended filters.

The mathematical foundation for implementing the analysis step where observations are used in the conditional Gaussian mixture framework is developed in the next section, followed by a summary of the nonlinear forecast models as well as crucial issues for practical implementation. The skill of the blended filters in capturing significant non-Gaussian features as well as crucial nonlinear dynamics is tested below in a 40-dimensional chaotic dynamical system with at least nine positive Lyapunov exponents in various turbulent regimes where the adaptive non-Gaussian subspace with a particle filter is only 5 dimensional with excellent performance for the blended filters. The mathematical formalism for filtering with conditionally Gaussian mixtures should be useful as a framework for multiscale data assimilation of turbulent signals and a simple application is sketched below. An earlier strategy related to the approach developed here is simply to filter the solution on an evolving low-dimensional subspace that captures the leading variance adaptively (20) while ignoring the other degrees of freedom; simple examples for nonnormal linear systems (18) demonstrate the poor skill of such an approach for reduced filtering in general; for filtering linear nonnormal systems, the optimal reduced basis instead is defined through balanced truncation (8).

Mathematical Foundations for Blended Particle Filters

Here we consider real-time filtering or data assimilation algorithms for a state vector $\mathit{u}\in {\mathrm{ℝ}}^N$ from a nonlinear turbulent dynamical system where the state $\mathit{u}$ is forecast by an approximate dynamical model between successive observation times, $m\mathrm{\Delta }t$, and the state of the system is updated through the use of the observations at the discrete times $m\mathrm{\Delta }t$ in the analysis step. For the blended particle filters developed below, there are two subspaces that typically evolve adaptively in time (although that dependence is suppressed here), where $\mathit{u}=\left({\mathit{u}}_1,{\mathit{u}}_2\right)$, ${\mathit{u}}_j\in {\mathrm{ℝ}}^{N_j}$, and $N_1+N_2=N$ with the property that $N_1$ is low dimensional enough so that the statistics of ${\mathit{u}}_1$ can be calculated from a particle filter whereas the statistics of ${\mathit{u}}_2$ are conditionally Gaussian given ${\mathit{u}}_1$. Thus, at any analysis time step $m\mathrm{\Delta }t$, we have the prior forecast density given by

$$p_{-}\left(\mathit{u}\right)=p_{-}\left({\mathit{u}}_1\right)p_{-}^G\left({\mathit{u}}_2|{\mathit{u}}_1\right),$$ [1]

where $p_{-}^G\left({\mathit{u}}_2|{\mathit{u}}_1\right)$ is a Gaussian distribution determined by the conditional mean and covariance, $p_{-}^G\left({\mathit{u}}_2|{\mathit{u}}_1\right)=\mathcal{N}\left({\overline{\mathit{u}}}_2^{-}\left({\mathit{u}}_1\right),R_2^{-}\left({\mathit{u}}_1\right)\right)$. We assume that the marginal distribution, $p_{-}\left({\mathit{u}}_1\right)$, is approximated by Q particles,

$$p_{-}\left({\mathit{u}}_1\right)={\displaystyle \sum _{j=1}^Q{p}_{j,-}\delta \left({\mathit{u}}_1-{\mathit{u}}_{1,j}\right)},$$ [2]

with nonnegative particle weights, $p_{j,-}$ with ${\displaystyle {\sum }_j{p}_{j,-}}=1$. Below we sometimes abuse notation as in [2] and refer to the continuous distribution in [1] and the particle distribution in [2] interchangeably. Here it is assumed that the nonlinear observation operator $G\left(\mathit{u}\right)$ maps ${\mathrm{ℝ}}^N$ to ${\mathrm{ℝ}}^M$, with M observations at the analysis time, $m\mathrm{\Delta }t$, and has the form

$$\mathit{v}=G\left(\mathit{u}\right)+{\mathit{\sigma }}_0=G_0\left({\mathit{u}}_1\right)+G_1\left({\mathit{u}}_1\right){\mathit{u}}_2+{\mathit{\sigma }}_0,$$ [3]

where $G_1\left({\mathit{u}}_1\right)$ has rank M and the observational noise, ${\mathit{\sigma }}_0$, is Gaussian ${\mathit{\sigma }}_0=\mathcal{N}\left(\mathbf{0},R_0\right)$. Note that the form in [3] is the leading Taylor expansion around ${\mathit{u}}_1$ of the general nonlinear observation operator. Now, start with the conditional Gaussian particle distribution, $p_{-}\left({\mathit{u}}_1,{\mathit{u}}_2\right)$ given from the forecast distribution

$$p_{-}\left(\mathit{u}\right)={\displaystyle \sum _{j=1}^Q{p}_{j,-}\delta \left({\mathit{u}}_1-{\mathit{u}}_{1,j}\right)\mathcal{N}\left({\overline{\mathit{u}}}_{2,j}^{-},R_{2,j}^{-}\right)},$$ [4]

and compute the posterior distribution in the analysis step through Bayes’ theorem, $p_{+}\left(\mathit{u}|\mathit{v}\right)\sim p\left(\mathit{v}|\mathit{u}\right)p_{-}\left(\mathit{u}\right)$. The key fact is the following.

Proposition 1.

Assume the prior distribution from the forecast is the blended particle filter conditional Gaussian distribution in [4] and assume the observations have the structure in [3]; then the posterior distribution in the analysis step taking into account the observations in [3] is also a blended particle filter conditional Gaussian distribution; i.e., there are explicit formulas for the updated weights, $p_{j,+}$, $1\le j\le Q$, and conditional mean, ${\overline{\mathit{u}}}_{2,j}^{+}$, and covariance, $R_{2,j}^{+}$, so that

$$p_{+}\left(\mathit{u}\right)={\displaystyle \sum _{j=1}^Q{p}_{j,+}\delta \left({\mathit{u}}_1-{\mathit{u}}_{1,j}\right)\mathcal{N}\left({\overline{\mathit{u}}}_{2,j}^{+},R_{2,j}^{+}\right)}.$$ [5]

In fact, the distributions $\mathcal{N}\left({\overline{\mathit{u}}}_{2,j}^{+},R_{2,j}^{+}\right)$ are updated by suitable Kalman filter formulas with the mean update for ${\overline{\mathit{u}}}_{2,j}^{+}$ depending nonlinearly on ${\mathit{u}}_{1,j}$ in general.

The proof of Proposition 1 is a direct calculation similar to those in refs. 21 and 22 for other formulations of Gaussian mixtures over the entire state space and the explicit formulas can be found in SI Text. However, there is a crucial difference that here conditional Gaussian mixtures are applied in the reduced subspace ${\mathit{u}}_2$ blended with particle filter approximations only in the lower-dimensional subspace, ${\mathit{u}}_1$, unlike the previous work.

Here we consider algorithms for filtering turbulent dynamical systems with the form

$${\mathit{u}}_t=L\mathit{u}+\mathit{B}\left(\mathit{u},\mathit{u}\right)+\mathit{F},$$ [6]

where $\mathit{B}\left(\mathit{u},\mathit{u}\right)$ involves energy-conserving nonlinear interactions with $\mathit{u}\cdot \mathit{B}\left(\mathit{u},\mathit{u}\right)=0$ whereas L is a linear operator including damping as well as anisotropic physical effects; many turbulent dynamical systems in geosciences and engineering have the structure shown in [6] (23, 24).

Application to Multiscale Filtering of a Slow–Fast System.

A typical direct use of the above formalism is briefly sketched. In many applications in the geosciences, there is a fixed subspace ${\mathit{u}}_1$ representing the slow (vortical) waves and a fixed subspace ${\mathit{u}}_2$ representing the fast (gravity) waves with observations of pressure and velocity for example, which naturally mix the slow and fast waves at each analysis step (15, 25–27). A small parameter $\mathit{ϵ}\ll 1$ characterizes the ratio of the fast timescale to the slow timescale. A well-known formalism for stochastic mode reduction has been developed for such multiscale systems (28, 29) with a simplified forecast model valid in the limit $\mathit{ϵ}\to 0$ with the form

$$\begin{array}{c}p\left({\mathit{u}}_1,{\mathit{u}}_2\right)\left(t\right)=p\left({\mathit{u}}_1\right)\left(t\right)p\left({\mathit{u}}_2|{\mathit{u}}_1\right)\\ =p\left({\mathit{u}}_1\right)\left(t\right)\mathcal{N}\left(0,R_2\right),\end{array}$$ [7]

where $p\left({\mathit{u}}_1\right)\left(t\right)$ satisfies a reduced Fokker–Planck equation for the slow variables alone and $R_2$ is a suitable background covariance matrix for the fast variables. A trivial application of Proposition 1 guarantees that there is a simplified algorithm consisting of a particle filter for the slow variables alone updated at each analysis step through Proposition 1, which mixes the slow and fast components through the observations. More sophisticated multiscale filtering algorithms with this flavor designed to capture unresolved features of turbulence have been developed recently in ref. 30.

Blended Statistical Nonlinear Forecast Models.

Whereas the above formulation can be applied to hybrid particle filters with conditional Kalman filters on fixed subspaces, defined by ${\mathit{u}}_1$ and ${\mathit{u}}_2$ as sketched above, a more attractive idea is to use statistical forecast models that adaptively change these subspaces as time evolves in response to the uncertainty without a separation of timescales. Recently Sapsis and Majda (16–19) developed nonlinear statistical forecast models of this type, the quasilinear Gaussian dynamical orthogonality (QG-DO) method and the more sophisticated modified quasilinear Gaussian dynamical orthogonality (MQG-DO) method for turbulent dynamical systems with the structure shown in [6]. It is shown in refs. 16 and 17, respectively, that both QG-DO and MQG-DO have significant skill for uncertainty quantification for turbulent dynamical systems with MQG-DO superior to QG-DO although more calibration is needed for MQG-DO in the statistical steady state.

The starting point for these nonlinear forecast models is a quasilinear Gaussian (QG) statistical closure (16, 18) for [6] or a more statistically accurate modified quasilinear Gaussian (MQG) closure (17, 18); the QG and MQG forecast models incorporate only Gaussian features of the dynamics given by mean and covariance. The more sophisticated QG-DO and MQG-DO methods have an adaptively evolving lower-dimensional subspace where non-Gaussian features are tracked accurately and allow for the exchange of statistical information between the evolving subspace with non-Gaussian statistics and the evolving Gaussian statistical background. We illustrate the simpler QG-DO scheme below and refer to refs. 17 and 18 for the details of the more sophisticated MQG-DO scheme. The QG-DO statistical forecast model is the following algorithm.

The subspace is represented as

$$\mathit{u}\left(t\right)=\overline{\mathit{u}}\left(t\right)+{\displaystyle \sum _{j=1}^s{Y}_j\left(t;\omega \right){\mathit{e}}_j\left(t\right)},$$ [8]

where ${\mathit{e}}_j\left(t\right)$, $j=1,\cdots ,s$ are time-dependent orthonormal modes and $s\ll N$ is the reduction order. The modes and the stochastic coefficients $Y_j\left(t;\omega \right)$ evolve according to the dynamical orthogonality (DO) condition (31). In particular, the equations for the QG-DO scheme are as follows.

Equation for the mean.

The equation for the mean is obtained by averaging the original system equation in [6],

$$\frac{d\overline{\mathit{u}}}{dt}=\left(L+D\right)\overline{\mathit{u}}+\mathit{B}\left(\overline{\mathit{u}},\overline{\mathit{u}}\right)+R_{ij}\mathit{B}\left({\mathit{v}}_i,{\mathit{v}}_j\right)+\mathit{F}.$$ [9]

Equation for the stochastic coefficients and the modes.

Both the stochastic coefficients and the modes evolve according to the DO equations (31). The coefficient equations are obtained by a direct Galerkin projection and the DO condition

$$\begin{array}{c}\frac{d{Y}_i}{dt}=Y_m\left[\left(L+D\right){\mathit{e}}_m+\mathit{B}\left(\overline{\mathit{u}},{\mathit{e}}_m\right)+\mathit{B}\left({\mathit{e}}_m,\overline{\mathit{u}}\right)\right]\cdot {\mathit{e}}_i\\ +\left(Y_m{Y}_n-C_{mn}\right)\mathit{B}\left({\mathit{e}}_m,{\mathit{e}}_n\right)\cdot {\mathit{e}}_i\end{array}$$ [10]

with $C_{mn}=\langle Y_m{Y}_n^{\mathrm{\ast }}\rangle$. Moreover, the modes evolve according to the equation obtained by stochastic projection of the original equation to the DO coefficients

$$\frac{\partial {\mathit{e}}_i}{\partial t}={\mathit{M}}_i-{\mathit{e}}_j\left({\mathit{M}}_i\cdot {\mathit{e}}_j\right),$$ [11]

with

$$\begin{array}{c}{\mathit{M}}_i=\left(L+D\right){\mathit{e}}_i+\mathit{B}\left(\overline{\mathit{u}},{\mathit{e}}_i\right)+\mathit{B}\left({\mathit{e}}_i,\overline{\mathit{u}}\right)\\ +\mathit{B}\left({\mathit{e}}_m,{\mathit{e}}_n\right)\langle Y_m{Y}_n{Y}_k\rangle C_{ik}^{-1}.\end{array}$$

Equation for the covariance.

The equation for the covariance starts with the exact equation involving third-order moments with approximated nonlinear fluxes

$$\frac{dR}{dt}=L_{v}R+R{L}_v^{\mathrm{\ast }}+Q_{F,s},$$ [12]

where the nonlinear fluxes are computed using reduced-order information from the DO subspace

$$\begin{array}{c}{Q}_{F,s}=\langle Y_m{Y}_n{Y}_k\rangle \left(\mathit{B}\left({\mathit{e}}_m,{\mathit{e}}_n\right)\cdot {\mathit{v}}_i\right)\left({\mathit{v}}_j\cdot {\mathit{e}}_k\right)\\ +\langle Y_m{Y}_n{Y}_k\rangle \left(\mathit{B}\left({\mathit{e}}_m,{\mathit{e}}_n\right)\cdot {\mathit{v}}_j\right)\left({\mathit{v}}_i\cdot {\mathit{e}}_k\right).\end{array}$$ [13]

The last expression is obtained by computing the nonlinear fluxes inside the subspace and projecting those back to the full N-dimensional space.

The QG statistical forecast model is the special case with $s=0$ so that [9] and [12] with $Q_{F,s}\equiv 0$ are approximate statistical dynamical equations for the mean and covariance alone. The MQG-DO algorithm is a more sophisticated variant based on MQG with significantly improved statistical accuracy (16–18).

Blended Particle Filter Algorithms

The QG-DO and MQG-DO statistical forecast models are solved by a particle filter or Monte Carlo simulation of the stochastic coefficients $Y_j\left(t\right)$, $1\le j\le s$ from [8] through the equations in [10] coupled to the deterministic equations in [9] and [12] for the statistical mean and covariance and the DO basis equations in [11]. Let $E\left(t\right)=\left\{{\mathit{e}}_1\left(t\right),\cdots ,{\mathit{e}}_s\left(t\right)\right\}$ denote the s-dimensional stochastic subspace in the forecast; at any analysis time, $t=m\mathrm{\Delta }t$, ${\mathit{u}}_1$ denotes the projection of $\mathit{u}\in {\mathrm{ℝ}}^N$ to $E\left(t\right)$. Complete the dynamical basis E with an orthonormal basis $E^{\perp }$ and define ${\mathit{u}}_2$ at any analysis time as the projection on $E^{\perp }$ (the flexibility in choosing $E^{\perp }$ can be exploited eventually). Thus, forecasts by QG-DO or MQG-DO lead to the following data from the forecast statistics at each analysis time: (i) a particle approximation for the marginal distribution

$$p_{-}\left({\mathit{u}}_1\right)={\displaystyle \sum _{j=1}^Q{p}_{j,-}\delta \left({\mathit{u}}_1-{\mathit{u}}_{1,j}\right)}$$ [14]

and (ii) the mean ${\overline{\mathit{u}}}_2^{-}$ and the covariance matrix

$$R=\left(\begin{array}{cc}{R}_1& R_{12}\\ R_{12}^{\mathrm{T}}& R_2\end{array}\right),$$ [15]

where R is the covariance matrix in the basis $\left\{E,E^{\perp }\right\}$.

To apply Proposition 1 in the analysis step, we need to find a probability density $p_{-}\left(\mathit{u}\right)$ with the form in [1] recovering the statistics in [14] and [15]. Below, for simplicity in exposition, we assume linear observations in [3]. We seek this probability density in the form

$$p_{-}\left({\mathit{u}}_1,{\mathit{u}}_2\right)={\displaystyle \sum _{j=1}^Q{p}_{j,-}\delta \left({\mathit{u}}_1-{\mathit{u}}_{1,j}\right)\mathcal{N}\left({\overline{\mathit{u}}}_{2,j}^{-},R_2^{-}\right)}$$ [16]

so that [14] is automatically satisfied by [16] whereas ${\overline{\mathit{u}}}_{2,j}^{-}$ and $R_2^{-}$ need to be chosen to satisfy [15]; note that $R_2^{-}$ is a constant matrix independent of j. Let $\langle g\left(\mathit{u}\right)\rangle$ denote the expected value of g with respect to $p_{-}$ so that, for example, $\langle {\mathit{u}}_1\rangle ={\overline{\mathit{u}}}_1^{-}$, $\langle {\mathit{u}}_2\rangle ={\overline{\mathit{u}}}_2^{-}$, and let ${\mathit{u}}_1^{\prime }={\mathit{u}}_1-\langle {\mathit{u}}_1\rangle$, ${\mathit{u}}_2^{\prime }={\mathit{u}}_2-\langle {\mathit{u}}_2\rangle$ denote fluctuations about this mean. Part I of the blended filter algorithm consists of two steps:

• Solve the following linear system to find the conditional mean ${\overline{\mathit{u}}}_2\left({\mathit{u}}_{1,j}\right)={\overline{\mathit{u}}}_{2,j}^{-}$:

  $$\left[\begin{array}{ccc}{p}_1{u}_{1,1}^{\prime 1}& \cdots & p_Q{u}_{1,1}^{\prime Q}\\ ⋮& \ddots & ⋮\\ p_1{u}_{1,N_1}^{\prime 1}& \cdots & p_Q{u}_{1,N_1}^{\prime Q}\\ p_1& \cdots & p_Q\end{array}\right]\left[\begin{array}{ccc}{u}_{2,1}^{\prime 1}& \cdots & u_{2,N_2}^{\prime 1}\\ u_{2,1}^{\prime 2}& \cdots & u_{2,N_2}^{\prime 2}\\ ⋮& \ddots & ⋮\\ u_{2,1}^{\prime Q}& \cdots & u_{2,N_2}^{\prime Q}\end{array}\right]=\left[\begin{array}{c}{R}_{12}\\ 0\end{array}\right].$$ [17]

  Note that this is an underdetermined system for a sufficiently large number of particles, Q, and “–“ notation is suppressed here.
• Calculate the covariance $R_2^{-}$ in the ${\mathit{u}}_2$ subspace by requiring from [15] and [16]

  $$\begin{array}{c}{R}_2^{-}=R_2+\langle {\mathit{u}}_2\rangle \otimes \langle {\mathit{u}}_2\rangle -{\displaystyle \int {\overline{\mathit{u}}}_2\left({\mathit{u}}_1\right)\otimes {\overline{\mathit{u}}}_2\left({\mathit{u}}_1\right)p\left({\mathit{u}}_1\right)d{\mathit{u}}_1}\\ =R_2-{\displaystyle \int {\overline{\mathit{u}}}_2^{\prime }\left({\mathit{u}}_1\right)\otimes {\overline{\mathit{u}}}_2^{\prime }\left({\mathit{u}}_1\right)p\left({\mathit{u}}_1\right)d{\mathit{u}}_1}\\ =R_2-{\displaystyle \sum _j{\overline{\mathit{u}}}_{2,j}^{\prime }\otimes {\overline{\mathit{u}}}_{2,j}^{\prime }{p}_{j,-}}.\end{array}$$ [18]

Any solution of [17] and [18] with $R_2^{-}\ge 0$ automatically guarantees that [14] and [15] are satisfied by the probability density in [16]. Note that $R_2^{-}$ is a constant matrix that does not depend on the individual particle weights. This crucial fact makes part II inexpensive.

Part II of the analysis step for the blended particle filter algorithm is an application of Proposition 1 to [16] (details in SI Text).

• Use Kalman filter updates in the ${\mathit{u}}_2$ subspace to get posterior mean and covariance ${\overline{\mathit{u}}}_{2,j}^{+}$, ${\tilde{R}}_2^{+}$.

• Update the particle weights in the u₁ subspace by $p_{j,+}\propto p_{j,-}{I}_j$, with

  $$I_j=\exp \left[\frac{1}{2}\left({\overline{\mathit{u}}}_{2,j}^{+\mathrm{T}}{\left({\tilde{R}}_2^{+}\right)}^{-1}{\overline{\mathit{u}}}_{2,j}^{+}-{\overline{\mathit{u}}}_{2,j}^{-\mathrm{T}}{\left(R_2^{-}\right)}^{-1}{\overline{\mathit{u}}}_{2,j}^{-}-{\left(\mathit{v}-GE{\mathit{u}}_{1,j}\right)}^{\mathrm{T}}{R}_0^{-1}\left(\mathit{v}-GE{\mathit{u}}_{1,j}\right)\right)\right].$$ [19]

• Normalize the weights $p_{j,+}=p_{j,-}{I}_j/{\displaystyle {\sum }_j{p}_{j,-}{I}_j}$ and resample.

• Get the posterior mean and covariance matrix from the posterior particle statistics $\left\{{\mathit{u}}_{1,j},{\overline{\mathit{u}}}_{2,j}^{+},p_{j,+}\right\}$.

This completes the description of the blended particle filter algorithms.

Realizability in the Blended Filter Algorithms.

A subtle issue in implementing the blended filter algorithms occurs in [17] and [18] from part I of the analysis step; a particular solution of the linear system in [17], denoted here as $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$, may yield a candidate covariance matrix, $R_2^{-}$, defined in [18] that is not positive definite; i.e., realizability is violated. Here we exploit the fact that the subspace for particle filtering is low dimensional so that in general, the number of particles satisfies $Q\ge N_1+1$ so the linear system $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$ is a strongly underdetermined linear system. The empirical approach that we use here is to seek the least-squares solution of $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$ that minimizes the weighted $L^2$ norm,

$${\displaystyle \sum _{j=1}^Q{\left|{\overline{\mathit{u}}}_{2,j}^{\prime }\right|}^2{p}_j}.$$ [20]

This solution is given as the standard least-squares solution through the pseudoinverse for the auxiliary variable ${\overline{\mathit{v}}}_{2,j}^{\prime }=p_j^{1/2}{\overline{\mathit{u}}}_{2,j}^{\prime }$. Such a least-squares solution guarantees that the trace of $R_2^{-}$, defined in [18], is maximized; however, this criterion still does not guarantee that $R_2^{-}=R_2-{\displaystyle {\sum }_j{\overline{\mathit{u}}}_{2,j}^{\prime }\otimes {\overline{\mathit{u}}}_{2,j}^{\prime }{p}_j}$ is realizable; to help guarantee this, we add extra inflation terms ${\alpha }_j\in \left[0,1\right]$ such that

$$R_2^{-}=R_2-{\displaystyle \sum _j{\alpha }_j{p}_j{\overline{\mathit{u}}}_{2,j}^{\prime }\otimes {\overline{\mathit{u}}}_{2,j}^{\prime }}.$$

Here the inflation coefficients ${\alpha }_j$ can be chosen according to

$${\alpha }_j=1\hspace{1em}\mathrm{if}\hspace{1em}{\overline{\mathit{u}}}_{2,j}^{\prime \mathrm{T}}\left(R_2-{\displaystyle \sum _k{\overline{\mathit{u}}}_{2,k}^{\prime }\otimes {\overline{\mathit{u}}}_{2,k}^{\prime }{p}_k}\right){\overline{\mathit{u}}}_{2,j}^{\prime }>{\mathit{ϵ}}_0,$$

$${\alpha }_j=1-\frac{{\mathit{ϵ}}_0-{\overline{\mathit{u}}}_{2,j}^{\prime \mathrm{T}}\left(R_2-{\displaystyle {\sum }_k{\overline{\mathit{u}}}_{2,k}^{\prime }\otimes {\overline{\mathit{u}}}_{2,k}^{\prime }{p}_k}\right){\overline{\mathit{u}}}_{2,j}^{\prime }}{p_j{\left|{\overline{\mathit{u}}}_{2,j}^{\prime }\right|}^4},\hspace{1em}\mathrm{otherwise},$$

where ${\mathit{ϵ}}_0\ll 1$ is a small number chosen to avoid numerical errors, and $1\le j\le Q$.

We find that this empirical approach works very well in practice as shown in subsequent sections. An even simpler but cruder variance inflation algorithm is to set $R_2^{-}=R_2\ge 0$. Further motivation for the constrained least-squares solution of $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$ minimizing [20] comes from the maximum entropy principle (24); the least biased probability density satisfying $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$ in [17] formally maximizes the entropy of $R_2^{-}$; i.e.,

$$\begin{array}{c}{\overline{\mathit{u}}}_{2,j}^{\prime }=\mathrm{argmax}\hspace{0.17em}\log \hspace{0.17em}\det \left(R_2-{\displaystyle {\sum }_j{p}_j{\overline{\mathit{u}}}_{2,j}^{\prime }\otimes {\overline{\mathit{u}}}_{2,j}^{\prime }}\right)\\ =\mathrm{argmax}\hspace{0.17em}\log \hspace{0.17em}\det \left(I-{\displaystyle {\sum }_j\left(p_j^{1/2}{R}_2^{-\left(1/2\right)}{\overline{\mathit{u}}}_{2,j}^{\prime }\right)}\otimes \left(p_j^{1/2}{R}_2^{-\left(1/2\right)}{\overline{\mathit{u}}}_{2,j}^{\prime }\right)\right).\end{array}$$ [21]

The high-dimensional nonlinear optimization problem in [21] is too expensive to solve directly but the small amplitude expansion, $\det \left(I-\mathit{ϵ}R\right)=-\mathit{ϵ}\mathrm{tr} R+O\left({\mathit{ϵ}}^2\right)$ of [21] becomes a weighted least-squares optimization problem for the new variable, ${\overline{\mathit{v}}}_{2,j}^{\prime }=p_j^{1/2}{R}_2^{-\left(1/2\right)}{\overline{\mathit{u}}}_{2,j}^{\prime }$ constrained by $\mathrm{ℒ}{U}_2=\mathrm{ℱ}$. If we choose $R_2=I$, the least-squares solution from [20] is recovered. Whereas the algorithm using ${\overline{\mathit{v}}}_{2,j}^{\prime }$ has a nice theoretical basis, it requires the singular value decomposition of the large covariance matrix, $R_2$, and the solution in [20] avoids this expensive procedure. Incidentally, the max-entropy principle alone does not guarantee realizability.

Numerical Tests of the Blended Particle Filters

Major challenges for particle filters for large-dimensional turbulent dynamical systems involve capturing substantial non-Gaussian features of the partially observed turbulent dynamical system as well as skillful filtering for spatially sparse infrequent high-quality observations of the turbulent signal (chap. 15 of ref. 15). In this last setting, the best ensemble filters require extensive tuning to avoid catastrophic filter divergence and quite often cheap filters based on linear stochastic forecast models are more skillful (15). Here the performance of the blended particle filter is assessed for two stringent test regimes, elucidating the above challenges, for the Lorenz 96 (L-96) model (32, 33); the L-96 model is a 40-dimensional turbulent dynamical system that is a popular test model for filter performance for turbulent dynamical systems (15). The L-96 model is a discrete periodic model given by

$$\frac{d{u}_i}{dt}=u_{i-1}\left(u_{i+1}-u_{i-2}\right)-u_i+F,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}i=0,\cdots ,J-1,$$ [22]

with $J=40$ and F the deterministic forcing parameter. The model is designed to mimic baroclinic turbulence in the midlatitude atmosphere with the effects of energy conserving nonlinear advection and dissipation represented by the first two terms in [22]. For sufficiently strong constant forcing values such as $F=$ 5, 8, or 16, the L-96 model is a prototype turbulent dynamical system that exhibits features of weakly chaotic turbulence ($F=5$), strongly chaotic turbulence ($F=8$), and strong turbulence ($F=16$) (15, 24, 34). Because the L-96 model is translation invariant, 20 discrete Fourier modes can be used to study its statistical properties. In all filtering experiments described below with the blended particle filters, we use $s=5$ with 10,000 particles. Thus, non-Gaussian effects are captured in a 5-dimensional subspace through a particle filter interacting with a low-order Gaussian statistical forecast model in the remaining 35 dimensions. The numbers of positive Lyapunov exponents on the attractor for the forcing values $F=5,8,16$ considered here are 9, 13, and 16, respectively (34), so the 5-dimensional adaptive subspace with particle filtering can contain at most half of the unstable directions on the attractor; also, non-Gaussian statistics are most prominent in the weakly turbulent regime, $F=5$, with nearly Gaussian statistics for $F=16$, the strongly turbulent regime, and intermediate statistical behavior for $F=8$.

Capturing Non-Gaussian Statistics Through Blended Particle Filters.

As mentioned above, the L-96 model in the weakly turbulent regime with $F=5$ has nine positive Lyapunov exponents on the attractor whereas Fourier modes ${\widehat{u}}_7$ and ${\widehat{u}}_8$ are the two leading empirical orthogonal functions (EOFs) that contain most of the energy. As shown in Fig. 1, where the probability density functions (pdfs) of $\left|{\widehat{u}}_7\right|$, $\left|{\widehat{u}}_8\right|$ are plotted, there is significant non-Gaussian behavior in these modes because the pdfs for $\left|{\widehat{u}}_7\right|$, $\left|{\widehat{u}}_8\right|$ are far from a Rayleigh distribution; see SI Text for the scatter plot of their joint distribution exhibiting strongly non-Gaussian behavior. For the filtering experiments below, sparse spatial observations are used with every fourth grid point observed with moderate observational noise variance $r_0=2$ and moderate observation frequency $\mathrm{\Delta }t=1$ compared with the decorrelation time 4.4 for $F=5$. We tested the QG-DO and MQG-DO blended filters as well as the Gaussian MQG filter and the ensemble adjustment Kalman filter (EAKF) with optimal tuned inflation and localization with $10,000$ ensemble members. All four filters were run for many assimilation steps and forecast pdfs for $\left|{\widehat{u}}_7\right|$, $\left|{\widehat{u}}_8\right|$ as well as forecast error pdfs for the real part of ${\widehat{u}}_7$, ${\widehat{u}}_8$ are plotted in Fig. 1. The blended MQG-DO filter accurately captures the non-Gaussian features and has the tightest forecast error pdf, the Gaussian MQG filter outperforms the blended QG-DO filter with a tighter forecast error distribution, and EAKF yields incorrect Gaussian distributions with the largest forecast error spread. The rms error and pattern correlation plots reported in SI Text confirm the above behavior as well as the scatter plots of the joint pdf for $\left|{\widehat{u}}_7\right|$, $\left|{\widehat{u}}_8\right|$ reported there. This example illustrates that a Gaussian filter like MQG with an accurate statistical forecast operator can have a sufficiently tight forecast error pdf and be a very good filter yet can fail to capture significant non-Gaussian features accurately. On the other hand, for the QG-DO blended algorithm in this example, the larger forecast errors of the QG dynamics compared with MQG swamp the effect of the blended particle filter. However, all three methods significantly improve upon the performance of EAKF with many ensemble members.

Fig. 1.

Comparison of pdfs of the absolute values of the first two leading Fourier modes ${\widehat{u}}_7$, ${\widehat{u}}_8$ (Upper) and pdfs of the forecast error $u_j^{-}-u_{\mathrm{truth}}$ (Lower, only real parts are shown) captured by different filtering methods.

Filter Performance with Sparse Infrequent High-Quality Observations.

Demanding tests for filter performance are the regimes of spatially sparse, infrequent in time, high-quality (low observational noise) observations for a strongly turbulent dynamical system. Here the performances of the blended MQG-DO and QG-DO filters as well as the MQG filter are assessed in this regime. For the strongly chaotic regime, $F=8$, for the L-96 model, observations are taken every fourth grid point with variance $r_0=0.01$ and observation time $\mathrm{\Delta }t=0.25$, which is nearly the decorrelation time, 0.33; the performance of EAKF as well as that of the rank histogram and maximum entropy particle filters has already been assessed for this difficult test problem in figure 15.14 of ref. 15 with large intervals in time with filter divergence (rms errors much larger than 1) for all three methods. A similar test problem for the strongly turbulent regime, $F=16$, with spatial observations every fourth grid point with $r_0=0.01$ and $\mathrm{\Delta }t=0.1$ compared with the decorrelation time, 0.12, is used here. In all examples with L-96 tested here, we find that the MQG-DO algorithm with the approximation described in the paragraph below (Eq. 20) is always realizable and is the most robust accurate filter; on the other hand, for the QG-DO filtering algorithm, the performance of the blended algorithm with crude variance inflation, $R_2^{-}=R_2$, significantly outperforms the basic QG-DO blended algorithm due to incorrect energy transfers in the QG forecast models for the long forecast times used here (SI Text). Fig. 2 reports the filtering performance of the MQG-DO and QG-DO blended filters and the MQG Gaussian filter in these tough regimes for $F=8,16$ through the rms error and pattern correlation. There are no strong filter divergences with the MQG-DO and QG-DO blended filters for both $F=8$ and F = 16 in contrast to other methods as shown in figure 15.14 of ref. 15. The much cheaper MQG filter for $F=16$ exhibits a long initial regime of filter divergence but eventually settles down to comparable filter performance to that of the blended filters. The blended MQG-DO filter is the most skillful robust filter over all these strongly turbulent regimes, $F=8,16$, as the observational noise and observation time are varied; see the examples in SI Text.

Fig. 2.

Comparison of rms errors (Left) and pattern correlations (Right) between different filtering methods in regimes $F=8$ (Top) and $F=16$ (Middle and Bottom) with sparse infrequent high-quality observations.

Concluding Discussion

Blended particle filters that capture non-Gaussian features in an adaptive evolving low-dimensional subspace through particles interacting with evolving Gaussian statistics on the remaining phase space are introduced here. These blended particle filters have been developed here through a mathematical formalism involving conditional Gaussian mixtures (35) combined with statistically nonlinear forecast models developed recently (16–18) with high skill for uncertainty quantification, which are compatible with this structure. Stringent test cases for filtering involving the 40-dimensional L-96 model with a 5-dimensional adaptive subspace for nonlinear filtering in various regimes of chaotic dynamics with at least nine positive Lyapunov exponents are used here. These test cases demonstrate the high skill of these blended filters in capturing both non-Gaussian dynamical features and crucial nonlinear statistics for accurate filtering in extreme regimes with sparse infrequent high-quality observations. The formalism developed is also useful for multiscale filtering of turbulent systems and a simple application has been sketched here.