Information Transfer Among the Components in Multi-Dimensional Complex Dynamical Systems

Abstract

In this paper, a rigorous formalism of information transfer within a multi-dimensional deterministic dynamic system is established for both continuous flows and discrete mappings. The underlying mechanism is derived from entropy change and transfer during the evolutions of multiple components. While this work is mainly focused on three-dimensional systems, the analysis of information transfer among state variables can be generalized to high-dimensional systems. Explicit formulas are given and verified in the classical Lorenz and Chua’s systems. The uncertainty of information transfer is quantified for all variables, with which a dynamic sensitivity analysis could be performed statistically as an additional benefit. The generalized formalisms can be applied to study dynamical behaviors as well as asymptotic dynamics of the system. The simulation results can help to reveal some underlying information for understanding the system better, which can be used for prediction and control in many diverse fields.

1. Introduction

Uncertainty quantification in complex dynamical systems is an important topic in prediction models. By integrating information-theoretic methods to investigate potential physics and measure indices, the uncertainty can be quantified better in ensemble practical predictions of complex dynamical systems. For instance, one of the important motivations is the couplings among variables of dynamical systems generating information at a nonzero rate [1], which produces information exchange [2]. Entropy can be used to quantitatively describe production, gathering, exchange and transfer of information [3]. Information transfer analysis can be used to detect asymmetry in the interactions of subsystems [1,4]. The emergent phenomena cannot be simply derived or solely predicted from the knowledge of the structure or from the interactions among individual elements in complex systems [5]. The dynamics of information transportation plays a critical role in complex systems, resulting in the system prediction [6,7], controls of a system [8,9] and causal analysis [10,11]. It emphasizes further understanding and investigating information transportation in complex dynamical systems. It has been applied to quantify nonlinear interactions based on the information transfer by several underlying efficient estimation strategies in complex dynamical systems [12,13,14]. Simple examples are used to illustrate various complex phenomena. The formalisms about information transfer are mostly based on two time series [1,15,16,17].

Recently, a new approach on information flow between the components of two-dimensional (2D) systems was adapted by Liang and Kleeman [6], which can be used to deal with the change of the uncertainty of one component given by the other component. This idea is based on specific interactions between two components in complex dynamical systems. For a system with dynamics given, a measure of information transfer can be rigorously formulated (referred as LK2005 formalism henceforth in [6]). In the forms of continuous flows and discrete mappings, the information flow has been analyzed using the Liouville equations [18] and the Frobenius–Perron operators [18]. These two equations are the evolution equations of the joint probability distributions, respectively. The present formalism is consistent with the transfer entropy of Schreiber [1] in both transfer asymmetry and quantification. A variety of generalizations and applications of the work in Reference [4] are developed in [19,20,21,22,23,24,25]. Majda and Harlim [26] applied the strategy to study subspaces of complex dynamical systems. For 2D systems, Liang and Kleeman discovered a concise law on the entropy evolution of deterministic autonomous systems and obtained the time rate of information flow from one component to the other [6]. Until now, the 2D formalism has been extended to some dynamical systems in different forms and scales with successful applications between two variables [23,25]. In the light of these applications, by thoroughly describing the statistical behavior of a system, this rigorous LK2005 formalism has yielded remarkable results [3].

However, the uncertainty of many real-world systems needs to be quantified among the variables for revealing the nonlinear relationships, so as to better understand the intrinsic mechanism and predict the forthcoming states of the systems [27]. Besides, many physical systems are affected by the interactions between multiple components in diverse fields [28]. For example, sensitivity analysis of an aircraft system with respect to design variables, parameters and uncertainty factors can be used to estimate the effects on the objective function or constraint function. The uncertainty analysis and sensitivity analysis (UASA) process is one of the key steps for determining the optimal search direction and guiding the design and decision-making, which aims at predicting complex computer models by quantifying the sensitivity information of the coupling variables. It can be offered to quick guide of determining design parameters which lead to high performance aircraft designs. Some preceding tools [29,30] related to sensitivity analysis are applicable for low-dimensional static problems and an urgent problem of high dimensionality arises when outputting variables of numerical models with spatially and temporally need to be solved [31]. The rigorous formalism of information flow has the potential to revolutionize the ability to analyze and measure uncertainty and sensitivity information in dynamical systems.

Hence, considering realistic applications, we generalize the LK2005 formalism to several variables of multi-dimensional dynamical systems in this paper. More precisely, we extend the results in [6,25] to the information flow between groups of components, rather than individual components. We aim to demonstrate that the formalism is feasible among several variables in arbitrary multi-dimensional dynamical systems when dynamics is fully known. In addition, the generalized formalisms can be reduced to two-dimensional formalisms as a special case. We also highlight the relationship between the LK2005 formalism and our generalized formalisms. Two applications are proposed with the classical Lorenz system and Chua’s system as validations of our formalisms. Compared with the LK2005 formalism and the transfer mutual information method [32], the generalized formalisms are beneficial for revealing more information among variables. It can better explore the complexity of evolution and intrinsic regularity of multi-dimensional dynamical systems. Meanwhile, it can provide a simple and versatile method to analyze sensitivity in dynamical models. These generalized formulas enable one to understand the relationship between information transfer and the behavior of a system. It can be used to perform sensitivity analysis as a measure in multi-dimensional complex dynamical systems. Therefore, the generalized formalisms have much wider applications and are significant to investigate real-world problems.

The structure of this paper is as follows: Section 2 recalls a systematic introduction of the theories and the formalisms about information flow in 2D systems; In Section 3, the formalisms are generalized to adapt to multi-dimensional complex dynamical system components based on the LK2005 formalism. Details on the derivations of the formalisms and the related properties are demonstrated; Section 4 gives a description about the formalisms with multi-dimensional applications; the summary of this paper is given in Section 5.

2. Two-Dimensional Formalism of Information Transfer (the LK2005 Formalism [6])

2.1. Continuous Flows

For 2D continuous and deterministic autonomous systems with fully known dynamics,

$$\frac{d\mathbf{x}}{dt}=\mathbf{F}\left(\mathbf{x}\right),$$ (1)

where $\mathbf{F}=(F_1,F_2)$ with $F_i=F_i(x_1,x_2)$ for any $i=1,2$ is known as the flow vector and $\mathbf{x}=(x_1,x_2)\in \mathsf{\Omega }={\mathsf{\Omega }}_1\times {\mathsf{\Omega }}_2.$ A stochastic process $\mathbf{X}=(X_1,X_2)\in \mathsf{\Omega }$ with joint probability density $\rho (x_1,x_2,t)$ at time t is the random variables corresponding to the sample values $(x_1,x_2)$. For convenience, we will use the notation $\rho$ or $\rho (x_1,x_2)$ instead of the notation $\rho (x_1,x_2,t)$ throughout Section 2, including the same expression at multi-dimensional cases in Section 3. In addition, the integral domain is the whole sample space $\mathsf{\Omega }$, except where noted. The probability density $\rho$ associated with Equation (1) satisfies the Liouville equation [18]:

$${\displaystyle \frac{\partial \rho }{\partial t}}+{\displaystyle \frac{\partial \left(F_1\rho \right)}{\partial x_1}}+{\displaystyle \frac{\partial \left(F_2\rho \right)}{\partial x_2}}=0.$$ (2)

The rate of change of joint entropy of $X_1$ and $X_2,$ $H\left(t\right)\stackrel{def}{=}-{\int \int }_{\mathsf{\Omega }}\rho log\rho d{x}_{1}d{x}_2,$ satisfies the relation [6]

$$\frac{dH}{dt}=E(\nabla ·\mathbf{F}),$$ (3)

where E means the mathematical expectation with respect to $\rho$ and $E(\nabla ·\mathbf{F})={\int \int }_{\mathsf{\Omega }}\rho (\nabla ·\mathbf{F})d{x}_{1}d{x}_2.$ That is to say, when a system evolves with time, the change of its joint entropy is totally controlled by the contraction or expansion of the phase space [6]. Later on, Liang and Kleeman showed that this property holds for deterministic systems of arbitrary dimensionality [20].

Liang and Kleeman [6] provided a very efficient heuristic argument to describe the decomposition of the various evolutionary mechanisms of information transfer in terms of the individual and joint time rates of entropy changes of $X_1,X_2$ and $(X_1,X_2).$ Firstly, they computed $\frac{d{H}_1}{dt}$ and $\frac{d{H}_2}{dt}$, where $H_i$ is the entropy of $X_i$ defined according to the marginal density, ${\rho }_i.$ Secondly, they employed the novel idea of frozen variables to analyze the individual time rates of entropy changes. When $X_i$ is fixed and $X_j$ evolves on its own in 2D systems, they found its temporal rate of change of entropy depends only on $E(\frac{\partial F_j}{\partial x_j}),$ denoted by $\frac{d{H}_j^{\ast }}{dt}.$ In the presence of interactions between $X_i$ and $X_j$, they observed that $\frac{d{H}_j}{dt}\ne E\left(\frac{\partial F_j}{\partial x_j}\right)=\frac{d{H}_j^{\ast }}{dt}.$ Therefore, Liang and Kleeman [6] concluded that the difference between $\frac{d{H}_j}{dt}$ and $E\left(\frac{\partial F_j}{\partial x_j}\right)$ should equal to the rate of entropy transfer from $X_i$ to $X_j.$ In the meantime, they denoted the rate of flow from $X_i$ to $X_j$ by $T_{i\to j}$ (T stands for ”transfer”) and defined information flow/transfer as

$$T_{i\to j}={\displaystyle \frac{d{H}_j}{dt}}-{\displaystyle \frac{d{H}_j^{\ast }}{dt}}=-{\int \int }_{\mathsf{\Omega }}{\rho }_{i|j}\left(x_i\right|x_j){\displaystyle \frac{\partial \left(F_j{\rho }_j\right)}{\partial x_j}}d{x}_{i}d{x}_j,$$ (4)

where ${\rho }_{i|j}\left(x_i\right|x_j)=\frac{\rho (x_i,x_j,t)}{\rho (x_j,t)}$ and $i,j=1,2$ with different $i,j$ at the same time.

2.2. Discrete Mappings

Similarly, Liang and Kleeman [6] also gave the formalism about a system in the discrete mapping form. Considering a 2D transformation

$$\mathsf{\Phi }:\mathsf{\Omega }\to \mathsf{\Omega },(x_1,x_2)\to \left({\mathsf{\Phi }}_1\left(\mathbf{x}\right),{\mathsf{\Phi }}_2\left(\mathbf{x}\right)\right),$$

where $\mathbf{x}=(x_1,x_2)\in \mathsf{\Omega }$ and $\mathsf{\Omega }:={\mathsf{\Omega }}_1\times {\mathsf{\Omega }}_2.$ The evolution of the density of $\mathsf{\Phi }$ is driven by the Frobenius–Perron operator ($F-P$ operator) $P:L^1\left(\mathsf{\Omega }\right)\to L^1\left(\mathsf{\Omega }\right)$ [18]. The entropy increases as

$$\begin{array}{cc}\hfill \Delta H& =-\int \int P\rho logP\rho d{x}_{1}d{x}_2+\int \int \rho log\rho d{x}_{1}d{x}_2\hfill \\ & =-\int \int \rho (x_1,x_2)log\left|J^{-1}\right|d{x}_{1}d{x}_2,\hfill \end{array}$$

where $J^{-1}$ is the Jacobian matrix of $\mathsf{\Phi }$. When ${\mathsf{\Phi }}_j$ is invertible in 2D transformations,

$$\Delta H_j^{\ast }=Elog\left|J_j\right|.$$ (5)

The entropy of $X_j$ increases as

$$\Delta H_j=-{\int }_{{\mathsf{\Omega }}_j}\left({\int }_{{\mathsf{\Omega }}_i}P\rho d{x}_i\right)log\left({\int }_{{\mathsf{\Omega }}_i}P\rho d{x}_i\right)d{x}_j+{\int }_{{\mathsf{\Omega }}_j}{\rho }_{j}log{\rho }_{j}d{x}_j,$$

where ${\rho }_j$ is the marginal density of $X_j.$ When ${\mathsf{\Phi }}_j$ is noninvertible in 2D transformations,

$$\begin{array}{cc}\hfill \Delta H_j^{\ast }=& \int {\rho }_j\left(x_j\right)log{\rho }_j\left(x_j\right)d{x}_j\hfill \\ & -\int \int P_j{\rho }_j\left({\mathsf{\Phi }}_j(x_i,x_j)\right)log{P}_j{\rho }_j\left({\mathsf{\Phi }}_j(x_i,x_j)\right)\rho \left(x_i\right|x_j)\left|J_j\right|d{x}_{i}d{x}_j,\hfill \end{array}$$ (6)

where $P_j$ is the $F-P$ operator when $x_i$ is frozen as a parameter in $P_j$. The entropy transferring from $X_i$ to $X_j$ is

$$\begin{array}{cc}\hfill T_{i\to j}=& -{\int }_{{\mathsf{\Omega }}_j}\left({\int }_{{\mathsf{\Omega }}_i}P\rho d{x}_i\right)log\left({\int }_{{\mathsf{\Omega }}_i}P\rho d{x}_i\right)d{x}_j\hfill \\ & +\int \int P_j{\rho }_j\left({\mathsf{\Phi }}_j(x_i,x_j)\right)log{P}_j{\rho }_j\left({\mathsf{\Phi }}_j(x_i,x_j)\right)\rho \left(x_i\right|x_j)\left|J_j\right|d{x}_{i}d{x}_j,\hfill \end{array}$$ (7)

where $i,j=1,2$ with different $i,j$ at the same time.

3. n-Dimensional Formalism of Information Transfer

3.1. Continuous Flows

Firstly, we consider a three-dimensional (3D) continuous autonomous system,

$$\frac{d\mathbf{x}}{dt}=\mathbf{F}\left(\mathbf{x}\right),$$ (8)

where $\mathbf{F}=(F_1,F_2,F_3)$ is a known flow vector. Similarly, the probability density $\rho$ associated with Equation (8) satisfies the Liouville equation [18]:

$${\displaystyle \frac{\partial \rho }{\partial t}}+{\displaystyle \frac{\partial \left(F_1\rho \right)}{\partial x_1}}+{\displaystyle \frac{\partial \left(F_2\rho \right)}{\partial x_2}}+{\displaystyle \frac{\partial \left(F_3\rho \right)}{\partial x_3}}=0.$$ (9)

Analogous to the derivation in [6], firstly, multiplying by $(1+log\rho )$ for Equation (9), after some algebraic manipulations:

$${\displaystyle \frac{\partial (\rho log\rho )}{\partial t}}+\mathbf{F}·\left(\nabla ·(\rho log\rho )\right)+\rho (1+log\rho )\nabla ·\mathbf{F}=0.$$ (10)

Then, integrating for Equation (10),

$$\frac{dH}{dt}-{\int \int \int }_{\mathsf{\Omega }}\nabla ·(\rho log\rho \mathbf{F})d{x}_{1}d{x}_{2}d{x}_3-{\int \int \int }_{\mathsf{\Omega }}\rho \nabla ·\mathbf{F}d{x}_{1}d{x}_{2}d{x}_3=0.$$

Assuming that $\rho$ vanishes at the boundaries (the compact support assumption for $\rho$ and the assumption is reasonable in real-world problems [6]), it is found that the time rate of the joint entropy change of $X_1,X_2$ and $X_3,$

$$H\left(t\right)\stackrel{def}{=}-{\int \int \int }_{\mathsf{\Omega }}\rho log\rho d{x}_{1}d{x}_{2}d{x}_3,$$

satisfies

$$\frac{dH}{dt}-{\int \int \int }_{\mathsf{\Omega }}\rho (x_1,x_2,x_3)\nabla ·\mathbf{F}d{x}_{1}d{x}_{2}d{x}_3=0$$

or

$$\frac{dH}{dt}=E(\nabla ·\mathbf{F}),$$

where $E(\nabla ·\mathbf{F})={\int \int \int }_{\mathsf{\Omega }}\rho (\nabla ·\mathbf{F})d{x}_{1}d{x}_{2}d{x}_3.$

As mentioned above, the time rate of change of H equals to the mathematical expectation of the divergence of the flow vector $\mathbf{F}.$ When we are interested in the entropy evolution of a component, $x_k$ in 3D systems, the marginal density is

$${\rho }_k(x_k,t)={\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}\rho (x_i,x_j,x_k,t)d{x}_{i}d{x}_j.$$

The evolution equation of ${\rho }_k$ is derived by taking the integral of Equation (9) with respect to $x_i$ and $x_j$ over the subspace ${\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j:$

$$\frac{\partial {\rho }_k}{\partial t}+\frac{\partial }{\partial x_k}{\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}\rho F_{k}d{x}_{i}d{x}_j=0.$$

The third and fourth terms in Equation (9) have been integrated out with the compact support assumption for $\rho .$ So the entropy for the component

$$H_k\left(t\right)=-{\int }_{{\mathsf{\Omega }}_k}{\rho }_{k}log{\rho }_{k}d{x}_k$$

evolves as

$$\frac{d{H}_k}{dt}={\int \int \int }_{\mathsf{\Omega }}\left[log{\rho }_k\frac{\partial \left(\rho F_k\right)}{\partial x_k}\right]d{x}_{i}d{x}_{j}d{x}_k,$$

i.e.,

$${\displaystyle \frac{d{H}_k}{dt}}=-{\int \int \int }_{\mathsf{\Omega }}\rho \left[{\displaystyle \frac{F_k}{{\rho }_k}}{\displaystyle \frac{\partial {\rho }_k}{\partial x_k}}\right]d{x}_{i}d{x}_{j}d{x}_k.$$ (11)

The Equation (11) states how $H_k$ evolves with time. The evolutionary mechanism of $H_k$ derives from two parts: One is from the evolution itself, $\frac{d{H}_k^{\ast }}{dt};$ another from the transfers of $X_i$ and $X_j$ according to the coupling in the joint density distribution $\rho .$ From Section 2, we know that when $X_k$ evolves on its own, then

$$E\left(\frac{\partial F_k}{\partial x_k}\right)=\frac{d{H}_k^{\ast }}{dt}={\int \int \int }_{\mathsf{\Omega }}\rho \frac{\partial F_k}{\partial x_k}d{x}_{i}d{x}_{j}d{x}_k.$$

Therefore, the rate of information flow/transfer from $X_i,X_j$ to $X_k$ is

$$\begin{array}{cc}\hfill T_{i,j\to k}& ={\displaystyle \frac{d{H}_k}{dt}}-{\displaystyle \frac{d{H}_k^{\ast }}{dt}}={\int \int \int }_{\mathsf{\Omega }}\rho \left({\displaystyle \frac{F_k}{{\rho }_k}}{\displaystyle \frac{\partial {\rho }_k}{\partial x_k}}+{\displaystyle \frac{\partial F_k}{\partial x_k}}\right)d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int \int \int }_{\mathsf{\Omega }}{\displaystyle \frac{\rho }{{\rho }_k}}{\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int \int \int }_{\mathsf{\Omega }}{\rho }_{i,j|k}(x_i,x_j|x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{i}d{x}_{j}d{x}_k,\hfill \end{array}$$ (12)

where ${\rho }_{i,j|k}(x_i,x_j|x_k)=\frac{\rho (x_i,x_j,x_k,t)}{\rho (x_k,t)}$ and $i,j,k=1,2,3$ with different $i,j,k$ at the same time.

In particular, if $F_1=F_1\left(x_1\right)$ has no dependence on $x_2,$ then $T_{2\to 1}=0$. There is no information transfer from random variable component $X_2$ to $X_1.$ This holds true with the transfers defined in LK2005 formalism. Obviously, in system (8), when $F_1$ has no dependence on $x_2,x_3,$ there should be no information transfer from $X_2,X_3$ to $X_1,$ but there is possibility that the transfers in other directions may be nonzero when $F_2$ depends on $x_1,x_3$ or $F_3$ depends on $x_1,x_2.$ This is consistent with the information transfer defined in Equation (12). As a matter of fact, an important property of the transfer is given below.

Theorem 1.

If $F_k$ is independent of $x_i,x_j$ in system (8) with different $i,j,k$, then $T_{i,j\to k}=0.$

Proof of Theorem 1

According to the formalism of information transfer for system (8), with the notation of $F_k=F_k\left(x_k\right),$

$$\begin{array}{cc}\hfill T_{i,j\to k}& =-{\int \int \int }_{\mathsf{\Omega }}{\rho }_{i,j|k}(x_i,x_j|x_k)\frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}{\rho }_{i,j|k}(x_i,x_j|x_k)d{x}_{i}d{x}_j\right)\frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}d{x}_k\hfill \\ & =-{\int }_{{\mathsf{\Omega }}_k}\frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}d{x}_k=0.\hfill \end{array}$$

 □

It is worth noting that, while $X_k$ gains information from $X_i$ or $X_j$, or $X_i$ and $X_j,$ $X_i$ or $X_j$ might have no dependence on $X_k$ in 3D systems. An important property about information transfer is its asymmetry among the components [1]. In addition, it is interesting to note that the formalism of 3D systems can be reduced to 2D cases under the condition that one variable does not depend on another variable. For example, If the evolution of $X_k$ is independent of $X_i$, then

$$\begin{array}{cc}\hfill T_{i,j\to k}& =-{\int \int \int }_{\mathsf{\Omega }}{\rho }_{i,j|k}(x_i,x_j|x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int \int }_{{\mathsf{\Omega }}_j\times {\mathsf{\Omega }}_k}\left({\int }_{{\mathsf{\Omega }}_i}\rho (x_i,x_j,x_k)d{x}_i\right){\displaystyle \frac{1}{\rho \left(x_k\right)}}{\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{j}d{x}_k\hfill \\ & =-{\int \int }_{{\mathsf{\Omega }}_j\times {\mathsf{\Omega }}_k}\rho \left(x_j\right|x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{j}d{x}_k=T_{j\to k}.\hfill \end{array}$$ (13)

In particular, when $X_k$ is independent of $X_i$ and $X_j$,

$$\begin{array}{c}\hfill T_{i,j\to k}=T_{i\to k}=T_{j\to k}=0.\end{array}$$

According to Theorem 1, the results are apparent. Furthermore, when $X_k$ depends on $X_i$ and $X_j$,

$$\begin{array}{cc}\hfill T_{i,j\to k}& =-{\int \int \int }_{\mathsf{\Omega }}{\rho }_{i,j|k}(x_i,x_j|x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int \int \int }_{\mathsf{\Omega }}\rho \left(x_j\right|x_k)·{\rho }_{i|j,k}\left(x_i\right|x_j,x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{i}d{x}_{j}d{x}_k\hfill \\ & =-{\int }_{{\mathsf{\Omega }}_i}\left({\int \int }_{{\mathsf{\Omega }}_j\times {\mathsf{\Omega }}_k}{\rho }_{j|k}\left(x_j\right|x_k){\displaystyle \frac{\partial \left(F_k{\rho }_k\right)}{\partial x_k}}d{x}_{j}d{x}_k\right){\rho }_{i|j,k}\left(x_i\right|x_j,x_k)d{x}_i\hfill \\ & =-{\int }_{{\mathsf{\Omega }}_i}{T}_{j\to k}·{\rho }_{i|j,k}\left(x_i\right|x_j,x_k)d{x}_i,\hfill \end{array}$$

or

$$\begin{array}{c}\hfill T_{i,j\to k}=-{\int }_{{\mathsf{\Omega }}_j}{T}_{i\to k}·{\rho }_{j|i,k}\left(x_j\right|x_i,x_k)d{x}_j.\end{array}$$ (14)

From the above derivations, we can see that our formalisms are further intensified by emphasizing the inherent relation with the formalisms in 2D systems. The information flows from two variables and the high order interactions between them to another variable are quantified by formula (12). These are generalized forms of the LK2005 formalism. In Section 4, we will validate the conclusions by the applications of all formulas in the Lorenz and Chua’s systems. Moreover, when several variables are involved, the formalisms are capable to tackle information transfers of a multi-dimensional system.

Combining the Liouville equation

$${\displaystyle \frac{\partial \rho }{\partial t}}+{\displaystyle \frac{\partial \left(F_1\rho \right)}{\partial x_1}}+{\displaystyle \frac{\partial \left(F_2\rho \right)}{\partial x_2}}+\cdots +{\displaystyle \frac{\partial \left(F_n\rho \right)}{\partial x_n}}=0,$$ (15)

with Equation (3), $\frac{dH}{dt}=E(\nabla ·\mathbf{F})$ in n-dimensional situations, we can generalize the formalism to n-dimensional continuous and deterministic autonomous systems in the same way. For example, the transfer of information from components $X_2,X_3,\dots ,X_n$ to $X_1$ is

$$T_{2,3,\dots ,n\to 1}=-{\int }_{\mathsf{\Omega }}{\rho }_{2,3,\dots ,n}(x_2,x_3,\dots ,x_n|x_1)\frac{\partial \left(F_1{\rho }_1\right)}{\partial x_1}d{x}_{1}d{x}_2\dots d{x}_n.$$

Hence, Theorem 1 can be generalized to multi-dimensional cases.

3.2. Discrete Mappings

For a 3D transformation $\mathsf{\Phi }:\mathsf{\Omega }\to \mathsf{\Omega },(x_1,x_2,x_3)\to \left({\mathsf{\Phi }}_1\left(\mathbf{x}\right),{\mathsf{\Phi }}_2\left(\mathbf{x}\right),{\mathsf{\Phi }}_3\left(\mathbf{x}\right)\right),$ the evolution of its density is driven by the Frobenius–Perron operator ($F-P$ operator) $P:L^1\left(\mathsf{\Omega }\right)\to L^1\left(\mathsf{\Omega }\right)$ [18]. Similar to the 2D case, after some efficient computations, the entropy transfer from $X_i,X_j$ to $X_k$ in three-dimensional mappings has the following form:

$$\begin{array}{cc}\hfill T_{i,j\to k}=& \Delta H_k-\Delta H_k^{\ast }\hfill \\ \hfill =& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k.\hfill \end{array}$$ (16)

We also give a theorem for the discrete mappings and highlight the relationship between two-dimensional formalisms and generalized formalisms. The formalisms can be extended to high-dimensional situations as well. The detailed processes are demonstrated in Appendix A.

4. The Application of Multi-Dimensional Formalism of Information Transfer

4.1. The Lorenz System

In this section, we propose an application to study the information flows about the Lorenz system [33]:

$$\left\{\begin{array}{c}\begin{array}{c}{\displaystyle \frac{d{x}_1}{dt}}=\sigma (x_2-x_1)\hfill \\ {\displaystyle \frac{d{x}_2}{dt}}=x_1(r-x_3)-x_2\hfill \\ {\displaystyle \frac{d{x}_3}{dt}}=x_1{x}_2-b{x}_3\hfill \end{array}\hfill \end{array}\right.,$$

where $\sigma ,r$ and b are parameters, $x_1,x_2$ and $x_3$ are the system state variables, and t is time. A chaotic attractor of Lorenz system with $\sigma =10,r=28,b=\frac{8}{3}$ is shown in Figure 1.

Figure 1.

The Lorenz attractor with initial value (1,1,1).

Firstly, we need to obtain the joint probability density function $\rho (x_1,x_2,x_3)$ of $\mathbf{X}$ to calculate information flows among the variables. For a deterministic system with known dynamics, the underlying evolution of the joint density $\rho (x_1,x_2,x_3)$ can be obtained by solving the Liouville equation. Taking into account of the computational load, we estimate the joint density $\rho (x_1,x_2,x_3)$ via numerical simulations. The steps are summarized as follows:

• Initialize the joint density $\rho (x_1,x_2,x_3)$ with a preset distribution ${\rho }_0$, then generate an ensemble through drawing samples randomly according to the initial distribution ${\rho }_0$.

• Partition the sample space $\mathsf{\Omega }$ into “bins”.

• Obtain an ensemble prediction for the Lorenz system at every time step.

• Estimate the three-variable joint probability density function $\rho$ via bin counting at every time step.

The Lorenz system is solved by applying a fourth order Runge–Kutta method with a time step $\Delta t=0.01$. According to Figure 1, the computation domain is restricted to $\mathsf{\Omega }\equiv [-30,30]\times [-30,30]\times [0,60],$ which includes the attractor of the Lorenz system. We discretize the sample space into $60\times 60\times 60=21,600$ bins to ensure covering the whole attractor and one draw per bin on average via making 21,600 random draws. Initially, we assume $\mathbf{X}$ is distributed as a Gaussian process $N\left(u\right(t),\Sigma (t\left)\right)$, with a mean u and a covariance matrix $\Sigma$:

$$u\left(0\right)=\left[\begin{array}{c}{u}_1\\ u_2\\ u_3\end{array}\right],\Sigma \left(0\right)=\left[\begin{array}{ccc}{\sigma }_1^2& 0& 0\\ 0& {\sigma }_2^2& 0\\ 0& 0& {\sigma }_3^2\end{array}\right].$$

Although we have used different parameters u and ${\sigma }_d^2$ $(d=1,2,3)$ to compute information flows for the Lorenz system, the final results are the same and the trends stay invariant. The parameters u and ${\sigma }_d^2$ can be adjusted for different experiments. Here we only show the results of one experiment with $u_d=4$ and ${\sigma }_d^2=4$. The ensemble is developed by drawing sample randomly in the light of a pre-established distribution ${\rho }_0\left(\underset{\_}{x}\right)$. We obtain an ensemble of $\mathbf{X}$ and estimate the three-variable joint probability density function $\rho (x_1,x_2,x_3,t)$ by the way of counting the bins, at every time step. As the equations are integrated forward in the Lorenz system, $\rho$ can be estimated as a function of time and describe the statistics of the system. A detailed discussion on probability estimation through bin counting are referred to [20,25]. The sample data with initial value (1,1,1) and an estimated marginal density of $x_1,x_2$ and $x_3$ are displayed in Figure 2.

Figure 2.

Left panel: a sample data ($X_1,X_2$ and $X_3$) of the Lorenz system generated by a fourth order Runge–Kutta method with $\Delta t=0.01$. Right panel: an estimated marginal density of $x_1,x_2$ and $x_3$ via counting the bins and initializing with a Gaussian distribution, respectively.

Through formula (12), the information transfer within three variables can be computed. There are nine transfer series in the Lorenz system, but here we mainly focus on the couple effect from two components to another component, that is, $T_{i,j\to k},i,j,k=1,2,3$ with different $i,j,k$ at the same time. A nonzero $T_{i,j\to k}$ means that $X_i$ and $X_j$ are causal to $X_k$, and the value means how much uncertainty that $X_i$ and $X_j$ bring to $X_k$. Among all the transfers, it is clearly shown that any two variables drive the other variable in the dynamics except the evolution of $X_1$ which only depends on $X_2.$ For the sake of revealing some underlying information in the chaotic dynamical system better, we also give information transfer among the components over space with a Gaussian distribution initialization and the averaged density over time via using the following formula: $S_{i,j\to k}=-\int \int {\overline{\rho }}_{i,j|k}(x_i,x_j|x_k,t)\frac{\partial \left(F_k{\overline{\rho }}_k\left(t\right)\right)}{\partial x_k}d{x}_{i}d{x}_j$, which characterizes the strength of information transfer at different planes of $x=x_i$. That is to say, it demonstrates the information transfer of $x_j$ and $x_k$ to $x_i$ plane, whose relative values represent the magnitudes of information transfer. The calculation results are plotted in the left panel and the right one of Figure 3, respectively. According to the magnitude of parameters in the Lorenz system and the definition of rigorous 3D formalisms, the information transfer from $X_1,X_2$ to $X_3$ is the smallest. The results are just as we expected, $\left|T_{1,2\to 3}\right|<\left|T_{1,3\to 2}\right|<\left|T_{2,3\to 1}\right|$, as shown in the left panel of Figure 3. Meanwhile, we can get much information through numerical simulations. For example, the information transfer from $X_2$ and $X_3$ on $X_1$ is larger than that of $X_1$ and $X_3$ on $X_2$ in the Lorenz system, which is helpful for us to better analyze the system and the fields of interest. Only the absolute value of T measures the information transfer among the variables [23]. As the ensemble evolution is carried forth, any two variables aim to reduce the uncertainty of the other variable [24]; in other words, any two variables tend to stabilize the other variable. All information flows go to constants, which means that the system tends to be stable simultaneously. Comparing the left panel with the right one in Figure 3, we can find that not only the information flow from $X_2$ and $X_3$ to $X_1$ is the largest at different times, but also the total information transfer is the largest at $x_1$ plane, and the strength of information transfer obeys a distribution in each direction of x. Repeated experiments are found to be in line with the results no matter whatever the initialization is given.

Figure 3.

Left panel: the multivariate information flow of the Lorenz system: blue dot-dash line: $T_{2,3\to 1}$; green star line: $T_{1,3\to 2}$; red solid line: $T_{1,2\to 3}$ (in nats per unit time); Right panel: the information strength of transfer in the Lorenz system: blue dot-dash line: $S_{2,3\to 1}$; green star line: $S_{1,3\to 2}$; red solid line: $S_{1,2\to 3}$ (arbitrary unit).

In particular, we compute the transfer, $T_{2\to 1}$, then compare $T_{2\to 1}$ with the transfer, $T_{2,3\to 1}$ in Figure 4, as well as plot the transfers $T_{1\to 2}$, $T_{3\to 2}$ and $T_{1,3\to 2}$ in Figure 5.

Figure 4.

$T_{2\to 1}$ and $T_{2,3\to 1}$ in the Lorenz system (in nats per unit time).

Figure 5.

$T_{1\to 2}$, $T_{3\to 2}$ and $T_{1,3\to 2}$ in the Lorenz system (in nats per unit time).

Since the evolution of $X_1$ is independent of $X_3$ and the evolution of $X_2$ depends on $X_1$ and $X_3$ in the Lorenz system, the transfer $T_{2\to 1}$ should be equal to $T_{2,3\to 1}$ and neither the transfer $T_{1\to 2}$ nor $T_{3\to 2}$ should not be equal to $T_{1,3\to 2}$ according to the derivations in Section 3.1. As expected, there is almost no difference between the two flows in Figure 4. The interpretation of the results is that $X_3$ is not causal to $X_1$ in the Lorenz system. The result agrees well with theoretical analysis, which also validates our formalisms. But the graphs $T_{1\to 2}$ and $T_{3\to 2}$ are quite different from the graph $T_{1,3\to 2}$ in Figure 5, as that both $X_1$ and $X_3$ are causal to $X_2$ in the Lorenz system. From Figure 4 and Figure 5, we can find that the information flow $T_{2\to 1}$ is different from $T_{1\to 2}$, as a property of asymmetry of the information transfer. There exists hidden sensitivity information in information transfer processes of high-dimensional dynamical systems: whether or not one variable brings more uncertainty to another variable. Comparing the magnitudes of three flows in Figure 5, we can say that $X_3$ is more sensitive to $X_2$ than $X_1$ to $X_2$ from the sensitivity analysis point of view. All the above differences are exactly the embodiment of the differences between the information flows in multi-dimensional dynamical systems and the LK2005 formalism. The proposed formalisms can be used to measure information transfers among the variables in dynamical systems and the numerical results can show how the measurement behavior with time, compared with the qualification of information transfer between two variables [4] and the transfer mutual information method [32]. For example, it can be quantified the influence that $x_3$ on the relationship between $x_1$ and $x_2$ using the transfer mutual information method in the Lorenz system. With our generalized formalisms, we can give quantitatively the influence from $x_3$ to the relationship between $x_1$ and $x_2$ as a dynamical process and other relationships (such as the asymmetrical influence between two variables) among the variables for analyzing the system better. To test the influence of error propagation on the measurement of information transfers, we use a different natural interval extension to compute information transfers according to the striking method [34]. In other words, we compute information transfers using formula (12) in the Lorenz system with the rewritten second equation, that is, $r{x}_1-x_1{x}_3$ is used to replace $x_1(r-x_3)$. For the Lorenz system, the results show that, the algorithm performs well (the relative error < 2%). All simulations are performed on a 64-bit Matlab R2016a environment. The physical consistency of the proposed approach in this paper can be explained as that a direction of the phase space is frozen in order to extract information transfers from the other two directions [3]. In addition, nonlinearity may lead a deterministic system to chaos, which causes the “spikes” in the right panel of Figure 3 and corresponds to intermittent switching in the chaotic dynamics. As the remarkable theory stated in [35], it indicates when the dynamics are about to switch lobes of the attractor in the Lorenz system.

Since Liouville equations and Frobenius–Perron analysis describe an ensemble of trajectories, we can use the generated formalisms of information flow as a sensitivity analysis index to perform dynamic sensitivity information analysis instead of the preceding widely used methods such as repeated calculation of principal component coefficients [36,37], construction of functional metamodels [31,38], calculation of moving average of the sensitivity index [39] and direct perturbation analysis of a dynamical system [40]. Using information flow to identify sensitive variables is directly based on the statistical perspective, which can improve numerical accuracy and efficiency while reduce the calculation load, compared with conventional dynamic sensitivity analysis methods. We cannot only quantify how much the uncertainty among variables of a system, but also understand how they influence system behavior, so it may be measured and used for prediction and control in realistic applications.

Furthermore, we use Equation (15) to compute information transfers, $T_{yzw\to x}$, $T_{xzw\to y}$, $T_{xyw\to z}$, and $T_{xyz\to w}$ with the same strategy in the four-dimensional (4D) dynamical system:

$$\left\{\begin{array}{c}\begin{array}{c}{\displaystyle \frac{dx}{dt}}=12(y-x)\hfill \\ {\displaystyle \frac{dy}{dt}}=23x-xz-y+w\hfill \\ {\displaystyle \frac{dz}{dt}}=xy-2.1z\hfill \\ {\displaystyle \frac{dw}{dt}}=-6y-0.2w\hfill \end{array}\hfill \end{array}\right.,$$

whose results are shown in Figure 6.

Figure 6.

Left panel: an estimated marginal density of $x,y,z$ and w via counting the bins and initializing with a Gaussian distribution, respectively; Right panel: the multivariate information flow over time of a 4D dynamical system.

The generalized formalisms are useful to deal with universal problems, which is not difficult to be applied to higher-dimensional cases.

4.2. The Chua’s System

Since it is the first analog circuit to realize chaos in experiments, the initial Chua’s system is a well-known dynamical model [41]. The Chua’s system is described in reference [42] and there are many researches on its dynamical behavior [43,44]. Here we present an investigation of the information flows within the smooth Chua’s system [45]:

$$\left\{\begin{array}{c}\begin{array}{c}{\displaystyle \frac{dx}{dt}}=p(x+y-xln\sqrt{1+x^2})\hfill \\ {\displaystyle \frac{dy}{dt}}=x-y+z\hfill \\ {\displaystyle \frac{dz}{dt}}=-qy\hfill \end{array}\hfill \end{array}\right.,$$

where $p,q$ are parameters, $x,y$ and z are state variables in $\mathbf{R}$ and $t\in {\mathbf{R}}^{+}.$ When $p=11$ and $q=14.87$, a chaotic attractor of the Chua’s system is shown in Figure 7.

Figure 7.

The attractor of Chua’s system with $x\left(0\right)=-3,y\left(0\right)=2,z\left(0\right)=1.$ The former three trajectories are $x,z$-plane,$x,y$-plane and $y,z$-plane, respectively. The last trajectory is a 3D plot of $x,y$ and $z.$

As mentioned before, using the same estimation procedures, we can obtain the density $\rho (x,y,z)$ of $\mathbf{R}$ by counting the bins at each step. From Figure 7, the appropriate computation domain $\mathsf{\Omega }\equiv [-10,10]\times [-10,10]\times [-10,10]$ which includes an attractor of the Chua’s system can be selected to estimate the three-variable joint probability density function. The following computation is demonstrated by applying a fourth order Runge–Kutta method. Similarly, we only show the results of one experiment after computing information flows multiple times by using different parameters. Suppose that $\mathbf{R}$ is distributed as a Gaussian process $N\left(u\right(t),\Sigma (t\left)\right),$ with a mean u and a covariance matrix $\Sigma$ in the initial state:

$$u\left(0\right)=\left[\begin{array}{c}9\\ 9\\ 9\end{array}\right],\Sigma \left(0\right)=\left[\begin{array}{ccc}9& 0& 0\\ 0& 9& 0\\ 0& 0& 9\end{array}\right].$$

Due to the additional fact that the smooth Chua’s circuit has a highly non-coherent dynamics [46], we discretize the sample space into $200\times 200\times 200=8,000,000$ bins to adequately understand the information transfer and the behavior of the system over time. A sample data and an estimation result of three marginal densities are shown in Figure 8, and we can find that the dynamical behaviors of the system are consistent with the results, such as symmetry. Using formula (12) to compute the information transfers within three variables of Chua’s system. Firstly, we discuss the coupling effect from two components to the other component, the calculation results are demonstrated in Figure 9.

Figure 8.

Left panel: a sample data ($X,Y$ and Z) of the Chua’s system generated by a fourth order Runge–Kutta method with $\Delta t=0.01$; Right panel: the purple line, black line, and blue line represent an estimated marginal density of $x,y,z$ by counting bins, respectively.

Figure 9.

Left panel: the multivariate information flow of the Chua’s system: green dot-dash line: $T_{y,z\to x}$; red dot-dash line: $T_{x,z\to y}$; blue dot-dash line: $T_{x,y\to z}$ (in nats per unit time); Right panel: the information strength of transfer in the Chua’s system: green dot-dash line: $S_{y,z\to x}$; red dot-dash line: $S_{x,z\to y}$; blue dot-dash line: $S_{x,y\to z}$ (arbitrary unit).

Secondly, we compute the transfers, $T_{y\to z}$ and $T_{z\to y}$, then compare $T_{y\to z}$ with the transfer, $T_{x,y\to z}$ and $T_{z\to y}$ with $T_{x,z\to y}$ in Figure 10 and Figure 11, respectively. We also show the corresponding results of the strength of information transfer among the components with a Gauss distribution initialization and the averaged density over time in Figure 9.

Figure 10.

$T_{y\to z}$ and $T_{x,y\to z}$ in the Chua’s system (in nats per unit time).

Figure 11.

$T_{z\to y}$ and $T_{x,z\to y}$ in the Chua’s system (in nats per unit time).

Since X causes Y but does not cause Z in the Chua’s system, the numerical results of Figure 10 and Figure 11 conform with the derivations of Equations (13) and (14) in Section 3.1. More specifically, there is almost no difference between the two flows in Figure 10, however, there exists large disparity between the two flows in Figure 11. The results also verify our formalisms. In addition, as shown in Figure 10 and Figure 11, we can see that the information flow $T_{y\to z}$ is different from $T_{z\to y}$ due to the asymmetry of information transfer. All simulations are performed on a 64-bit Matlab R2016a environment. We are able to estimate that one variable makes another variable more uncertain or more predictable via the generalized formalisms. Besides, we can identify sensitive variables by computing information transfers among the variables in dynamical systems.

Compared with the Lorenz system, the Chua’s system embodies in engineering systems besides that their discoveries were extraordinary and changed scientific thinking [46]. It can be used as another means to research, experiment and think about humanity, identity and art, etc. [47,48]. In studying visualization of the dynamics of Chua’s circuit through computational models, the quantitative transformations of behavior are being taken into account [46]. The multi-dimensional formalisms of information flow enable us to improve our ability to estimate, predict, and control complex systems in many diverse fields. Furthermore, most existing approaches in control and synchronization of chaotic systems require adjusting the parameters of the model and estimating system parameters, which become an active area of research [49], and an additional benefit provided by the multi-dimensional formalisms of information flow is parameter estimation. We can compute information flows of the simulation model with different sets of parameters and do the same procedure for obtaining a group of feedback, then determine the optimal parameters that cater for the actual needs in order to put insight into complex behavior of models by comparing the change rates.

5. Conclusions

Based on the LK2005 formalism, we propose a rigorous and general formalism of the information transfer among multi-dimensional complex dynamical system components, for continuous flows and discrete mappings, respectively. Information transfers are quantified through entropy transfers from some components to another component, enabling us to better understand the physical mechanism underlying the superficial behavior and explore deeply hidden information in the evolution of multi-dimensional dynamical systems. When the generalized formalisms are reduced to 2D cases, the results are consistent with the LK2005 formalism. We mainly focus on the study of 3D systems and apply the formalisms to investigate information transfers for the Lorenz system and the Chua’s system. In the above-mentioned two cases, we show that information flows of the whole evolution and the strength of information transfer at different planes, which implies that how uncertainty propagates and how dynamic essential information in the system transports. The results of experiments on the generalized formalisms conform with observations and empirical analysis in the literature, whose application may benefit many diverse fields. Compared with the qualification of information transfer between two variables [4] and the transfer mutual information method [32], the generalized formalisms are helpful for analyzing the relationships among the variables in dynamical systems and the research of complex systems. Moreover, since the formalism is built on the statistical nature of information, it has the potential to perform sensitivity analysis in multi-dimensional complex dynamical systems and advance our ability to estimate, predict and control these systems. In practice, for complex high-dimensional dynamical systems, it is not easy to give the dynamics analytically. Considering many critical data-driven problems are primed to take advantage of progress in the data-driven discovery of dynamics [35], we are developing a dynamic-free formulation to analyze information flows of multi-dimensional dynamical systems.

In the future, the formalism will be further generated to high-dimensional stochastic dynamical systems and time-delay systems. Meanwhile, future research should investigate how the information flow as a new indicator can be deployed in the frame of dynamic sensitivity analysis.

Appendix A. Discrete Mappings

Now consider a 3D transformation

$$\mathsf{\Phi }:\mathsf{\Omega }\to \mathsf{\Omega },(x_1,x_2,x_3)\to \left({\mathsf{\Phi }}_1\left(\mathbf{x}\right),{\mathsf{\Phi }}_2\left(\mathbf{x}\right),{\mathsf{\Phi }}_3\left(\mathbf{x}\right)\right)$$ (A1)

and the Frobenius–Perron operator ($F-P$ operator) $P:L^1\left(\mathsf{\Omega }\right)\to L^1\left(\mathsf{\Omega }\right)$ [18] which steers the evolution of its density. Loosely, given a density $\rho =(x_1,x_2,x_3),$P is defined as that

$${\int \int \int }_{w}P\rho (x_1,x_2,x_3)d{x}_{1}d{x}_{2}d{x}_3={\int \int \int }_{{\mathsf{\Phi }}^{-1}\left(w\right)}\rho (x_1,x_2,x_3)d{x}_{1}d{x}_{2}d{x}_3,$$

where w represents any subset of $\mathsf{\Omega }.$ When $\mathsf{\Phi }$ is invertible, P can be expressed clearly as $P\rho \left(\underset{\_}{x}\right)=\rho \left[{\mathsf{\Phi }}^{-1}\left(\underset{\_}{x}\right)\right]\left|J^{-1}\right|$, where $J^{-1}=J^{-1}(x_1,x_2,x_3)=det\left[\frac{\partial \left({\mathsf{\Phi }}^{-1}(x_1,x_2,x_3)\right)}{\partial (x_1,x_2,x_3)}\right]$ is the determinant of the Jacobian matrix of $\mathsf{\Phi }.$ Similar to the two-dimensional case, the entropy increases

$$\begin{array}{cc}\hfill \Delta H=& -\int \int \int P\rho logP\rho d{x}_{1}d{x}_{2}d{x}_3+\int \int \int \rho log\rho d{x}_{1}d{x}_{2}d{x}_3\hfill \\ \hfill =& -\int \int \int \rho \left({\mathsf{\Phi }}^{-1}(x_1,x_2,x_3)\right)\left|J^{-1}\right|log\left[\rho \left({\mathsf{\Phi }}^{-1}(x_1,x_2,x_3)\right)\left|J^{-1}\right|\right]d{x}_{1}d{x}_{2}d{x}_3\hfill \\ & +\int \int \int \rho log\rho d{x}_{1}d{x}_{2}d{x}_3\hfill \\ \hfill =& -\int \int \int \rho (v_1,v_2,v_3)\left|J^{-1}\right|\left[log\rho (v_1,v_2,v_3)+log\left|J^{-1}\right|\right]\left|J\right|d{v}_{1}d{v}_{2}d{v}_3\hfill \\ & +\int \int \int \rho log\rho d{x}_{1}d{x}_{2}d{x}_3\hfill \\ \hfill =& -\int \int \int \rho (x_1,x_2,x_3)log\left|J^{-1}\right|d{x}_{1}d{x}_{2}d{x}_3,\hfill \end{array}$$

concisely rewritten as

$$\Delta H=Elog\left|J\right|.$$ (A2)

Meantime, in the case when ${\mathsf{\Phi }}_k$ is invertible of 3D transformations,

$$\Delta H_k^{\ast }=Elog\left|J_k\right|.$$ (A3)

The entropy of $X_k$ increases as

$$\begin{array}{cc}\hfill \Delta H_k=& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +{\int }_{{\mathsf{\Omega }}_k}{\rho }_{k}log{\rho }_{k}d{x}_k,\hfill \end{array}$$ (A4)

where ${\rho }_k$ is the marginal density of $X_k.$

When ${\mathsf{\Phi }}_k$ is noninvertible,

$$\begin{array}{cc}\hfill \Delta H_k^{\ast }& =\int {\rho }_k\left(x_k\right)log{\rho }_k\left(x_k\right)d{x}_k\hfill \\ & -\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k,\hfill \end{array}$$ (A5)

where $P_k$ is the $F-P$ operator when $x_i,x_j$ is frozen as parameters in $P_k$. It is easy to find that Equation (A5) reduces to Equation (A3) when ${\mathsf{\Phi }}_k$ is invertible. Therefore, the entropy transfers from $X_i,X_j$ to $X_k$ can be unified into a form

$$\begin{array}{cc}\hfill T_{i,j\to k}=& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_i}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k,\hfill \end{array}$$ (A6)

where $i,j,k=1,2,3$ with different $i,j,k$ at the same time.

Just as the former case with continuous variables, the information flow obtained by Equation (16) has the following property:

Theorem A1.

If ${\mathsf{\Phi }}_k$ is independent of $x_i,x_j$ in system (A1) with different $i,j,k$, then $T_{i,j\to k}=0.$

The detailed proof of Theorem A1 is presented in Appendix A.1. Moreover, the formalism of 3D system can be reduced to the formalism in 2D cases with the previously mentioned conditions being satisfied. For example, when ${\mathsf{\Phi }}_k$ has no dependence on $x_i$,

$$\begin{array}{cc}\hfill T_{i,j\to k}=& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ \hfill =& -\int P_k{\rho }_{k}log{P}_k{\rho }_{k}d{x}_k\hfill \\ & +\int \int P_k{\rho }_k({\mathsf{\Phi }}_k(x_i,x_j,x_k)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_i(x_i,x_j,x_k)\right)\left|J_k\right|\left({\int }_{{\mathsf{\Omega }}_i}\rho (x_i,x_j|x_k)d{x}_i\right)d{x}_{i}d{x}_j\hfill \\ \hfill =& -\int P_k{\rho }_{k}log{P}_k{\rho }_{k}d{x}_k\hfill \\ & +\int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_j,x_k)\right)\rho \left(x_j\right|x_k)\left|J_k\right|d{x}_{j}d{x}_k=T_{j\to k}.\hfill \end{array}$$

In particular, when ${\mathsf{\Phi }}_k$ has no dependence on $x_i$ and $x_j$,

$$\begin{array}{c}\hfill T_{i,j\to k}=T_{i\to k}=T_{j\to k}=0.\end{array}$$

Furthermore, when ${\mathsf{\Phi }}_k$ has dependence on $x_i$ and $x_j$,

$$\begin{array}{cc}\hfill T_{i,j\to k}=& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ \hfill =& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho \left(x_j\right|x_k)\rho \left(x_i\right|x_j,x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ \hfill =& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +{\int }_{{\mathsf{\Omega }}_i}\left({\int \int }_{{\mathsf{\Omega }}_j\times {\mathsf{\Omega }}_k}{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_i,x_j,x_k)\right)\rho \left(x_j\right|x_k)\left|J_k\right|d{x}_{j}d{x}_k\right)\hfill \\ & ·\rho \left(x_i\right|x_j,x_k)d{x}_i\hfill \\ \hfill =& -{\int }_{{\mathsf{\Omega }}_i}{T}_{j\to k}·\rho \left(x_i\right|x_j,x_k)d{x}_i\hfill \\ & +{\int }_{{\mathsf{\Omega }}_i}\left({\int }_{{\mathsf{\Omega }}_k}{P}_k{\rho }_{k}log{P}_k{\rho }_{k}d{x}_k\right)·\rho \left(x_i\right|x_j,x_k)d{x}_i-{\int }_{{\mathsf{\Omega }}_k}{P}_k{\rho }_{k}log{P}_k{\rho }_{k}d{x}_k\hfill \end{array}$$

The above formalisms can also be generalized to n-dimensional systems by efficient processing of the relationship between the $F-P$ operator

$${\int }_{w}P\rho (x_1,x_2,\dots x_n)d{x}_{1}d{x}_2\dots d{x}_n={\int }_{{\mathsf{\Phi }}^{-1}\left(w\right)}\rho (x_1,x_2,\dots x_n)d{x}_{1}d{x}_2\dots d{x}_n$$

and the entropy evolution at different time steps. For example, the transfer of entropy from $X_2,X_3,\dots ,X_n$ to $X_1$ is

$$\begin{array}{cc}\hfill T_{2,3,\dots ,n\to 1}=& -{\int }_{{\mathsf{\Omega }}_1}\left({\int }_{{\mathsf{\Omega }}_{2\dots n}}P\rho d{x}_{2}d{x}_3\dots d{x}_n\right)log\left({\int }_{{\mathsf{\Omega }}_{2\dots n}}P\rho d{x}_{2}d{x}_3\dots d{x}_n\right)d{x}_1\hfill \\ & +{\int }_{\mathsf{\Omega }}{P}_1{\rho }_1\left({\mathsf{\Phi }}_1(x_1,x_2,\dots ,x_n)\right)log{P}_1{\rho }_1\left({\mathsf{\Phi }}_1(x_1,x_2,\dots ,x_n)\right)\hfill \\ & ·\rho (x_2,x_3,\dots ,x_n|x_1)\left|J_1\right|d{x}_{1}d{x}_2\dots d{x}_n.\hfill \end{array}$$

Here ${\mathsf{\Omega }}_{2\dots n}$ is the simplified script of ${\mathsf{\Omega }}_2\times {\mathsf{\Omega }}_3\times \dots \times {\mathsf{\Omega }}_n.$ Similar to the continuous cases, the generalized version of the property of Theorem A1 is also suitable for multi-dimensional discrete mappings.

Appendix A.1.

Proof of Theorem A1

We only need to show that when ${\mathsf{\Phi }}_k$ is independent of $x_i,x_j$ in 3D system,

$$\Delta H_k^{\ast }=\Delta H_k.$$

According to Equation (A4) and Equation (A5), we only need to prove

$$\begin{array}{c}-{\int \int \int }_{\mathsf{\Omega }}{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ =-{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k.\hfill \end{array}$$

According to the definition of the $F-P$ operator and the condition that ${\mathsf{\Phi }}_k$ is independent of $x_i,x_j$ at the same time,

$$\begin{array}{c}-{\int \int \int }_{\mathsf{\Omega }}{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ =-{\int }_{{\mathsf{\Omega }}_k}{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)\left|J_k\right|{\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}\rho (x_i,x_j|x_k)d{x}_{i}d{x}_{j}d{x}_k\hfill \end{array}$$

because ${\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}\rho (x_i,x_j|x_k)d{x}_{i}d{x}_{j}d{x}_k=1$

$$\begin{array}{c}-{\int }_{{\mathsf{\Omega }}_k}{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)\left|J_k\right|{\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}\rho (x_i,x_j|x_k)d{x}_{i}d{x}_{j}d{x}_k\hfill \\ =-{\int }_{{\mathsf{\Omega }}_k}{P}_k{\rho }_k\left(y_k\right)log{P}_k{\rho }_k\left(y_k\right)d{y}_k\hfill \\ =-{\int }_{{\mathsf{\Omega }}_k}{p}_k{\rho }_k\left(x_k\right)log{P}_k{\rho }_k\left(x_k\right)d{x}_k\hfill \\ =-{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \end{array}$$

where $y_1={\mathsf{\Phi }}_k(x_k,x_i,x_j).$ So

$$\begin{array}{cc}\hfill \Delta H_k^{\ast }=& \int {\rho }_k\left(x_k\right)log{\rho }_k\left(x_k\right)d{x}_k\hfill \\ & -\int \int \int P_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)log{P}_k{\rho }_k\left({\mathsf{\Phi }}_k(x_k,x_i,x_j)\right)\rho (x_i,x_j|x_k)\left|J_k\right|d{x}_{i}d{x}_{j}d{x}_k\hfill \\ \hfill =& -{\int }_{{\mathsf{\Omega }}_k}\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)log\left({\int \int }_{{\mathsf{\Omega }}_i\times {\mathsf{\Omega }}_j}P\rho d{x}_{i}d{x}_j\right)d{x}_k\hfill \\ & +{\int }_{{\mathsf{\Omega }}_k}{\rho }_{k}log{\rho }_{k}d{x}_k=\Delta H_k.\hfill \end{array}$$

 □

Author Contributions

Y.Y. proposed the original idea, implemented the experiments, analyzed the data and wrote the paper. X.D. contributed to the theoretical analysis and simulation designs and revised the manuscript. All authors read and approved the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.