Symplectic Bregman Divergences

Abstract

We present a generalization of Bregman divergences in finite-dimensional symplectic vector spaces that we term symplectic Bregman divergences. Symplectic Bregman divergences are derived from a symplectic generalization of the Fenchel–Young inequality which relies on the notion of symplectic subdifferentials. The symplectic Fenchel–Young inequality is obtained using the symplectic Fenchel transform which is defined with respect to the symplectic form. Since symplectic forms can be built generically from pairings of dual systems, we obtain a generalization of Bregman divergences in dual systems obtained by equivalent symplectic Bregman divergences. In particular, when the symplectic form is derived from an inner product, we show that the corresponding symplectic Bregman divergences amount to ordinary Bregman divergences with respect to composite inner products. Some potential applications of symplectic divergences in geometric mechanics, information geometry, and learning dynamics in machine learning are touched upon.

1. Introduction

Symplectic geometry [1,2,3] was historically pioneered by Lagrange around 1808–1810 [4,5,6] where the motions and dynamics (evolution curves) of a finite set of m point mass particles in a time interval T are analyzed in the phase space by a 1D curve $\mathcal{C}=\{c\left(t\right)=(q_1,p_1,\dots ,q_m,p_m)\left(t\right):t\in T\subset \mathbb{R}\}\subset {\mathbb{R}}^{2n}$, where $q_i\left(t\right)\in {\mathbb{R}}^n$’s denote the point locations at time t and $p_i\left(t\right)\in {\mathbb{R}}^n$’s encode the momentum, i.e., $p_i\left(t\right)=m_i{\dot{q}}_i$ with ${\dot{q}}_i={\displaystyle \frac{d}{\mathrm{d}t}}{q}_i\left(t\right)$. See Figure 1. (Notice that Joseph-Louis Lagrange (1736–1813) was 72 years old in 1808, and is famous for his treatise on analytic mechanics [7,8] published first in french in 1788 when he was 52 years old).

Figure 1.

The motion of a single point particle $q\left(t\right)$ with mass m and momentum $p\left(t\right)=m\dot{q}\left(t\right)$ on a 1D line can be modeled as a curve $\mathcal{C}=\left\{c\right(t)=(q\left(t\right),p\left(t\right)):t\in T\subset \mathbb{R}\}$ in the phase space ${\mathbb{R}}^2$.

The Hamiltonian coupled equations [9] governing the system motion are written in the phase space as follows:

$$\begin{array}{c}\hfill {\displaystyle \frac{{\mathrm{d}q}^i}{\mathrm{d}t}}={\displaystyle \frac{\partial H}{\partial p_i}},{\displaystyle \frac{{\mathrm{d}p}_i}{\mathrm{d}t}}=-{\displaystyle \frac{\partial H}{\partial q^i}},\end{array}$$ (1)

where $H(q,p,t)$ is the Hamiltonian describing the system. Lagrange originally started a new kind of calculus, “symplectic calculus”. Symplectic geometry can be thought as the first discovered non-Euclidean geometric structure since hyperbolic geometry is usually considered to be first studied by Lobachevsky and Bolyai around 1820–1930. We refer to the paper entitled “The symplectization of science” [10] for an outreach article on symplectic geometry.

The adverb “symplectic” stems from Greek: It means “braided together” to convey the interactions of point mass particle positions with their momenta. Its use in mathematics originated in the work of Hermann Weyl (see §6 on symplectic groups in [11]). Another synonym adverb of symplectic is “complex” which has been used to describe braided numbers z of $\mathbb{C}=\{z=a+ib:(a,b)\in {\mathbb{R}}^2\}$. Complex has its etymological root in Latin. In differential geometry, symplectic structures are closely related to (almost) complex structures on vector spaces and smooth manifolds [2].

In physics, symplectic geometry is not only at the core of classical mechanics (i.e., conservative reversible mechanics) and quantum mechanics [12], but has also recently been used to model and study dynamics of systems exhibiting dissipative terms [13,14] which are irreversible. As a pure geometry, symplectic geometry can be studied on its own by mathematicians, and gave birth to the field of symplectic topology [15]. Thus, symplectic geometry can be fruitfully applied to various areas beyond its original domain of geometric mechanics. For example, symplectic geometry has been considered in machine learning for accelerating numerical optimization methods based on symplectic integrators [16] and in physics-informed neural networks [17,18] (PINNs).

In this paper, we define symplectic Bregman divergences (Definition 5) which recover as special cases Bregman divergences [19] defined with respect to composite inner products. A Bregman divergence induced by a strictly convex and differentiable (potential) function F (called the Bregman generator) between $x_1$ and $x_2$ of X is defined in [19] (1967) by

$$B_F(x_1:x_2)=F\left(x_1\right)-F\left(x_2\right)-\langle x_1-x_2,\nabla F\left(x_2\right)\rangle ,$$ (2)

where $\langle ·,·\rangle$ is an inner product on X. Let ${\mathrm{\Gamma }}_0\left(X\right)$ denote the set of functions which are lower semi-continuous convex with non-empty effective domains. The convex conjugate $F^{*}\left(x^{*}\right)$ obtained by the Legendre–Fenchel transform $F^{*}\left(x^{*}\right)={sup}_{x\in X}\langle x^{*},x\rangle -F\left(x\right)$ yields a dual Bregman divergence $B_{F^{*}}$ when the function $F\in {\mathrm{\Gamma }}_0\left(X\right)$ is of Legendre type [20,21]:

$$B_{F*}(x_1*:x_2^{*})=F^{*}\left(x_1^{*}\right)-F^{*}\left(x_2^{*}\right)-\langle x_1^{*}-x_2^{*},\nabla F^{*}\left(x_2^{*}\right)\rangle ,$$

such that $B_F(x_1:x_2)=B_{F^{*}}(x_2^{*}:x_1^{*})$ with

$$F^{*}\left(x^{*}\right)=\langle x^{*},{(\nabla F)}^{-1}\left(x^{*}\right)\rangle -F\left({(\nabla F)}^{-1}\left(x^{*}\right)\right).$$

This paper introduces and extends the work of Buliga and Saxcé [13,14] which is motivated by geometric irreversible mechanics. To contrast with [13,14], this expository paper is targeted to an audience familiar with Bregman divergences [19] in machine learning and information geometry [22] but does not assume any prior knowledge in geometric mechanics. Furthermore, we consider only finite-dimensional spaces in this study.

The paper is organized as follows: In Section 2, we define symplectic vector spaces and explain the representation of symplectic forms using dual pairings. We then define the symplectic Fenchel transform and the symplectic Fenchel–Young inequality in Section 3. The definitions of symplectic Fenchel–Young divergences (Definition 4) and symplectic Bregman divergences (Definition 5) are reported in Section 4. In particular, we show how to recover Bregman divergences with respect to composite inner products as special cases in Section 5 (Property 1). In general, symplectic Bregman divergences allow one to define Bregman divergences in dual systems equipped with pairing products. Finally, we recall the role of Bregman divergences in dually flat manifolds of information geometry in Section 6, and motivate the introduction of symplectic Bregman divergences in geometric mechanics (e.g., symplectic BEN principle of [13,14]) and learning dynamics in machine learning.

2. Dual Systems, Linear Symplectic Forms, and Symplectomorphisms

2.1. Symplectic Forms Derived from Dual Systems

We begin with two definitions:

Definition 1 

(Dual system). Let X and Y be finite m-dimensional vector spaces [23] equipped with a pairing product $b(·,·)$, i.e., a bilinear map:

$$b(·,·):X\times Y\to \mathbb{R},$$

such that all continuous linear functionals on X and Y are expressed as $x^{\#}(·)=b(x,·)$ and $y^{\#}(·)=b(·,y)$, respectively. The triplet $(X,Y,b(·,·))$ forms a dual system.

(Notice that when the type of X is different from the type of Y then the bilinear map cannot be symmetric).

Definition 2 

(Symplectic vector space). A symplectic vector space $(V,\omega )$ is a vector space equipped with a map [24] $\omega :Z=V\times V\to \mathbb{R}$ which is

• 1.
  bilinear: $\forall \alpha ,\beta ,{\alpha }^{\prime },{\beta }^{\prime }\in \mathbb{R},\forall z_1,z_2\in Z$, we have

  $$\omega (\alpha z_1+{\alpha }^{\prime }{z}_1^{\prime },z_2,\beta z_2+{\beta }^{\prime }{z}_2^{\prime })=\alpha \beta \omega (z_1,z_2)+\alpha {\beta }^{\prime }\omega (z_1,z_2^{\prime })+{\alpha }^{\prime }\beta \omega (z_1^{\prime },z_2)+{\alpha }^{\prime }{\beta }^{\prime }\omega (z_1,z_2^{\prime }),$$
• 2.skew-symmetric (or alternating): $\omega (z_2,z_1)=-\omega (z_1,z_2)$, and
• 3.non-degenerate: if for a $z_0$, we have $\omega (z,z_0)=0$ for all $z\in Z$ then we have $z_0=0$.

Notice that skew-symmetry implies that $\omega (z,z)=0$ for all $z\in Z$ since $\omega (z,z)=-\omega (z,z)$ and hence $2\omega (z,z)=0$. The map $\omega$ is called a linear symplectic form [24,25].

We define the symplectic form $\omega$ induced by the pairing product of a dual system as follows:

$$\omega (z_1,z_2)=b(x_1,y_2)-b(x_2,y_1),$$ (3)

where $z_1=(x_1,l_1)$ and $z_2=(x_2,l_2)$ belong to $Z=X\oplus Y$.

Let us report several examples of linear symplectic forms:

• Let $X=V$ be a finite n-dimensional vector space with the dual space of linear functionals $Y=V^{*}$ (space of covectors l). The natural pairing $\left(\right(x,l\left)\right)=l\left(x\right)={\sum }_i{x}^i{l}_i$ of a vector $x\in V$ with a covector $l\in V^{*}$ is an example of dual product. (We use the superscript index for indicating components of contravariant vectors and subscript index for specifying components of covariant vectors [9]). We define the symplectic form $\omega$ induced by the natural pairing of vectors with covectors as follows:

  $$\omega (z_1,z_2)=\left((x_1,l_2)\right)-\left((x_2,l_1)\right)=l_2\left(x_1\right)-l_1\left(x_2\right),$$ (4)

  where $z_1=(x_1,l_1)$ and $z_2=(x_2,l_2)$ belong to $Z=V\oplus V^{*}$.
• Consider $(X,\langle ·,·\rangle )$ an inner product space of dimension n. The product space $Z=X\oplus X$ of even dimension n can be equipped with the following map $\omega :Z\times Z\to \mathbb{R}$ induced by the inner product:

  $$\omega (z_1,z_2)=\langle x_1,y_2\rangle -\langle x_2,y_1\rangle ,$$ (5)

  where $z_1=(x_1,y_1)\in Z$ and $z_2=(x_2,y_2)\in Z$.

For example, let $X=\mathbb{R}$ and $\langle x_1,x_2\rangle =x_1{x}_2$. Then $\omega (z_1,z_2)=x_1{y}_2-x_2{y}_1$. This symplectic form can be interpreted as the determinant of the matrix $M=\left[\begin{array}{cc}{x}_1\hfill & x_2\hfill \\ y_1\hfill & y_2\hfill \end{array}\right]$ which corresponds geometrically to the signed orientation of the parallelogram defined by the vectors $z_1=(x_1,y_1)$ and $z_2=(x_2,y_2)$. See Figure 2. (This example indicates the link with integration of 2D manifolds equipped with fields of symplectic forms smoothly varying called differential 2-forms [9]).

Figure 2.

Interpreting a 2D symplectic form $\omega (z_1,z_2)$ as the signed area of a parallelogram with first oriented edge $z_1$ (grey). A pair of vectors defines two possible orientations of the parallelogram: The orientation compatible with $z_1$ and the reverse orientation compatible with $z_2$. $\omega$ is called the standard area form.

In a finite-dimensional vector space, we can express the inner product as $\langle x,y\rangle =x^{\top }Qy$ for a symmetric positive-definite matrix $Q\in {\mathbb{R}}^{n\times n}$. Let $Q=L^{\top }L$ be the Cholesky decomposition of Q. Then we have

$$\langle x,y\rangle ={\left(L^{\top }x\right)}^{\top }I\left(L^{\top }y\right)={\langle L^{\top }x,L^{\top }y\rangle }_0,$$

where I is the $n\times n$ identity matrix and ${\langle x,y\rangle }_0=x^{\top }y$ is the Euclidean inner product. Thus the form ${\omega }_0$ induced by ${\langle ·,·\rangle }_0$ can be expressed using linear algebra as

$${\omega }_0(z_1,z_2)=z_1^{\top }\left[\begin{array}{cc}0\hfill & I\hfill \\ -I\hfill & 0\hfill \end{array}\right]z_2=z_1^{\top }{\mathrm{\Omega }}_0{z}_2,$$

where ${\mathrm{\Omega }}_0\in {\mathbb{R}}^{2n\times 2n}$ is a skew-symmetric matrix: ${\mathrm{\Omega }}_0^{\top }=-{\mathrm{\Omega }}_0$. More generally, we may consider skew-symmetric matrices of the form $\mathrm{\Omega }=\left[\begin{array}{cc}0\hfill & L^{\top }\hfill \\ -L^{\top }\hfill & 0\hfill \end{array}\right]$ to define the symplectic form ${\omega }_Q$ induced by the inner product ${\langle x,y\rangle }_Q=x^{\top }Qy$.

2.2. Linear Symplectomorphisms and the Groups of Symplectic Matrices

A symplectic form $\omega$ can be expressed as a $2n\times 2n$ matrix $\mathrm{\Omega }=\left[{\omega }_{ij}\right]$ such that ${\omega }_{ij}=\omega (b_i,b_j)$ where $b_1=e_1,\dots ,b_n=e_n,b_{n+1}=f_1,\dots ,b_{2n}=f_n$ are the basis vectors, and $\omega (z_1,z_2)=z_1^{\top }\mathrm{\Omega }{z}_2$.

The Darboux basis [2] of the canonical form ${\omega }_0$ of ${\mathbb{R}}^{2n}$ is such that $\omega (e_i,f_j)={\delta }_{ij}$ and $\omega (e_i,e_j)=\omega (l_i,l_j)=0$ where $\delta$ denotes the Kronecker delta function. ${\mathrm{\Omega }}_0\in \mathrm{Sp}\left(2n\right)$ is the symplectic matrix $\left[\begin{array}{cc}0\hfill & I\hfill \\ -I\hfill & 0\hfill \end{array}\right]$ corresponding to the canonical form ${\omega }_0$ of ${\mathbb{R}}^{2n}$.

A transformation $t:V\to V$ is called a linear symplectomorphism when $\omega (t\left(z_1\right),t\left(z_2\right)=\omega (z_1,z_2)$ (i.e., $t^{*}\omega =\omega$), i.e., when $T^{\top }\mathrm{\Omega }T=\mathrm{\Omega }$ where T be the matrix representation of t. In particular t is a linear symplectomorphism with respect to ${\omega }_0$ when $T^{\top }{\mathrm{\Omega }}_{0}T={\mathrm{\Omega }}_0$. Any symplectic vector space of $(V,\omega )$ dimension $2n$ is symplectomorphic to the canonical symplectic space $({\mathbb{R}}^{2n},{\omega }_0)$.

Linear symplectomorphisms can be represented by symplectic matrices of the symplectic group [11,26] $\mathrm{Sp}\left(2n\right)$:

$$\begin{array}{ccc}\hfill \mathrm{Sp}\left(2n\right)& =& \left\{T:T^{\top }{\mathrm{\Omega }}_{0}T={\mathrm{\Omega }}_0\right\}\subset \mathrm{GL}\left(2n\right),\hfill \\ & =& \left\{T=\left[\begin{array}{cc}A\hfill & B\hfill \\ C\hfill & D\hfill \end{array}\right]:-C^{\top }A+A^{\top }C=0,-C^{\top }B+A^{\top }D=I,-D^{\top }B+B^{\top }D=0\right\},\hfill \end{array}$$

Transpose and inverse of symplectic matrices are symplectic matrices. The inverse of a symplectic matrix T is given by

$$\begin{array}{ccc}\hfill T^{-1}& =& -{\mathrm{\Omega }}_0{T}^{\top }{\mathrm{\Omega }}_0,\hfill \\ & =& \left[\begin{array}{cc}{D}^{\top }\hfill & -B^{\top }\hfill \\ -C^{\top }\hfill & A^{\top }\hfill \end{array}\right]\hfill \end{array}$$

Symplectic matrices of $\mathrm{Sp}\left(2n\right)$ have unit determinant ($\mathrm{Sp}\left(2n\right)\subset \mathrm{SL}\left(2n\right)\subset \mathrm{GL}\left(2n\right)$), and in the particular case of $n=1$, $\mathrm{Sp}\left(2\right)$ corresponds precisely to the set of matrices with unit determinant. Thus rotation matrices of $\mathrm{SO}\left(2\right)$ which have unit determinant for a subgroup of $\mathrm{Sp}\left(2\right)$.

Sesquilinear symplectic forms can also be defined on complex linear spaces [27].

3. Symplectic Fenchel Transform, Symplectic Subdifferentials, and Symplectic Fenchel–Young (in)Equality

Let $F:Z=X\times Y\to \mathbb{R}\cup \{+\infty \}$ be a convex lower semi-continuous (lsc) function called a potential function.

Definition 3 

(Symplectic Fenchel conjugate). The symplectic Fenchel conjugate $F^{*\omega }\left(z^{\prime }\right)$ is defined by

$$F^{*\omega }\left(z^{\prime }\right)=\underset{z\in Z}{sup}\left\{\omega (z^{\prime },z)-F\left(z\right)\right\}.$$

Notice that since $\omega$ is skew-symmetric, the order of the arguments in $\omega$ is important: The symplectic Fenchel transform optimizes with respect to the second argument of $\omega (·,·)$.

The symplectic subdifferential of F at z is defined by

$${\partial }^{\omega }F\left(z\right)=\left\{z_1\in Z:\forall z_2\in Z,F(z+z_2)\ge F\left(z\right)+\omega (z_1,z_2)\right\}.$$

The differential operator ${\partial }^{\omega }$ is a set-valued operator: ${\partial }^{\omega }:\mathcal{F}\rightrightarrows Z$, where $\mathcal{F}$ is the set of potential functions. An element of the symplectic subdifferential of F at z is called a symplectic subgradient.

Remark 1. 

Moreau generalized the Fenchel conjugate using a cost function [28]. In particular, the duality induced by logarithmic cost function was studied in [29], and lead to a generalization of Bregman divergences called the logarithmic divergences which are canonical divergences of constant section curvature manifolds in information geometry.

Remark 2. 

In geometric mechanics [2], the symplectic gradient on a symplectic manifold $(M,\omega )$ is the Hamiltonian vector field, i.e., the vector field $X_H$ such that the Halmitonian mechanics equation writes concisely as $\omega (X_H,·)=\mathrm{d}H$.

Theorem 1 

(Symplectic Fenchel–Young inequality, Theorem 2.3 of [13,14]). Let $F\left(z\right)$ be a convex (i.e., $F\left(z\right)=F(x,y)$ is joint convex, i.e., convex with respect to $z=(x,y)$) and lower semi-continuous function. Then the following inequality holds:

$$\forall z,z^{\prime }\in Z,F\left(z\right)+F^{*\omega }\left(z^{\prime }\right)\ge \omega (z^{\prime },z),$$

with equality if and only if $z^{\prime }\in {\partial }^{\omega }\left(z\right)$.

Let us again notice that the argument order in $\omega (·,·)$ is important.

Assume that the potential functions are smooth and that symplectic subdifferentials consist only of single-element sets (singletons). By abuse of language, we shall call in this paper the symplectic gradient of F the single element of the symplectic subdifferential ${\partial }^{\omega }$, and denote it by ${\nabla }^{\omega }F$: ${\partial }^{\omega }F\left(z\right)=\left\{{\nabla }^{\omega }F\left(z\right)\right\}$. (Our terminology and notation is thus not to be confused with the Hamiltonian vector field $X_H$ of geometric mechanics).

4. Symplectic Fenchel–Young Divergences and Symplectic Bregman Divergences

Divergences are smooth dissimilarity functions (see Section 4.2 of [30]). From the symplectic Fenchel–Young inequality of Theorem 1, we can define the symplectic Fenchel–Young divergence as follows:

Definition 4 

(Symplectic Fenchel–Young divergence). Let $F:Z=X\times Y\to \mathbb{R}$ be a smooth convex function. Then the symplectic Fenchel–Young divergence is the following non-negative measure of dissimilarity between z and $z^{\prime }$:

$$Y_F(z,z^{\prime })=F\left(z\right)+F^{*\omega }\left(z^{\prime }\right)-\omega (z^{\prime },z)\ge 0.$$ (6)

We have $Y_F(z,z^{\prime })=0$ if and only if $z^{\prime }\in {\partial }^{\omega }F\left(z\right)$, i.e., $z^{\prime }={\nabla }^{\omega }F\left(z\right)$ when F is smooth.

Let us now define the symplectic Bregman divergence $B_F^{\omega }(z_1:z_2)$ as $Y_F(z_1,z_2^{\prime })$ where $z_2^{\prime }={\nabla }^{\omega }F\left(z_2\right)$. Using the following identity derived from the symplectic Fenchel–Young equality:

$$F^{*\omega }\left({\nabla }^{\omega }F\left(z\right)\right)=\omega ({\nabla }^{\omega }F\left(z\right),z)-F\left(z\right),$$

and the bilinearity of the symplectic form, we obtain:

$$\begin{array}{ccc}\hfill B_F^{\omega }(z_1:z_2)& =& Y_F(z_1,z_2^{\prime }),\hfill \\ & =& F\left(z_1\right)+F^{*\omega }\left(z_2^{\prime }\right)-\omega (z_2^{\prime },z_1),\hfill \\ & =& F\left(z_1\right)+\omega ({\nabla }^{\omega }F\left(z_2\right),z_2)-F\left(z_2\right)-\omega ({\nabla }^{\omega }F\left(z_2\right),z_1),\hfill \\ & =& F\left(z_1\right)-F\left(z_2\right)-\omega ({\nabla }^{\omega }F\left(z_2\right),z_1-z_2).\hfill \end{array}$$ (7)

Since $\omega$ is skew-symmetric, we can also rewrite Equation (7) equivalently as

$$B_F^{\omega }(z_1:z_2)=F\left(z_1\right)-F\left(z_2\right)+\omega (z_1-z_2,{\nabla }^{\omega }F\left(z_2\right)).$$ (8)

Definition 5 

(Symplectic Bregman divergence). Let $(Z=X\times Y,\omega )$ be a symplectic vector space. Then the symplectic Bregman divergence between $z_1$ and $z_2$ of Z induced by a smooth convex potential $F\left(z\right)$ is

$$B_F^{\omega }(z_1:z_2)=F\left(z_1\right)-F\left(z_2\right)-\omega ({\nabla }^{\omega }F\left(z_2\right),z_1-z_2),$$

where the symplectic subdifferential gradient is the singleton ${\partial }^{\omega }F\left(z\right)=\left\{{\nabla }^{\omega }F\left(z\right)\right\}$.

Remark 3. 

The ordinary Bregman divergences (BDs) have been generalized to non-smooth strictly convex potential functions using a subdifferential map in [31,32,33] to choose among several potential subgradients at a given location. Similarly, we can extend symplectic Bregman divergences to non-smooth strictly convex potential functions using a symplectic subdifferential map.

5. Particular Cases Recover Composite Bregman Divergences

When $Y=X$ and $(X,\langle ·,·\rangle )$ is an inner-product space, we may consider the composite inner-product on $Z=X\times X$:

$$\langle \langle z_1,z_2\rangle \rangle =\langle x_1,y_1\rangle +\langle x_2,y_2\rangle ,$$

with $z_1=(x_1,y_1)$ and $z_1=(x_2,y_2)$.

Let $I:Z\to Z$ be the linear function $I\left(z\right)=z$ and denote by $J:Z\to Z$ the linear function defined by

$$J\left(z\right)=J(x,y)=(-y,x).$$

Notice that this definition of J makes sense because $X=Y$ and thus $(-y,x)\in Z$. We check that we have $J^2(x,y)=J(-y,x)=(-x,-y)=-(x,y)$, i.e., $J^2=-I$. Furthermore, we have $g(z_1,z_2)=\omega (z_1,J{z}_2)=\langle x_1,x_2\rangle +\langle y_1,y_2\rangle$ that is a positive definite inner product. That is, the automorphism J is a complex structure $\omega$-compatible (J is a symplectomorphism).

We can express the symplectic form $\omega (z_1,z_2)=\langle x_1,y_2\rangle -\langle x_2,y_1\rangle$ induced by the inner product using the composite inner product as follows:

$$\begin{array}{ccc}\hfill \omega (z_1,z_2)& =& \langle x_1,y_2\rangle -\langle x_2,y_1\rangle =\langle \langle J\left(z_1\right),z_2\rangle \rangle ,\hfill \\ \hfill \omega (-J{z}_1,z_2)& =& \langle \langle z_1,z_2\rangle \rangle .\hfill \end{array}$$

Similarly, the symplectic subdifferential of F can be expressed using the ordinary subdifferential (and vice versa) as follows:

$$\begin{array}{ccc}\hfill z^{\prime }\in {\partial }^{\omega }F\left(z\right)& \iff & J\left(z^{\prime }\right)\in \partial F\left(z\right),\hfill \\ \hfill z^{\prime }\in \partial F\left(x\right)& \iff & -J\left(z^{\prime }\right)\in {\partial }^{\omega }F\left(z\right).\hfill \end{array}$$

When subdifferentials are singletons, we thus have

$$\begin{array}{ccc}\hfill J\left({\nabla }^{\omega }F\left(z\right)\right)& =& \nabla F\left(z\right),\hfill \\ \hfill {\nabla }^{\omega }F\left(z\right)& =& -J(\nabla F(z\left)\right).\hfill \end{array}$$

Last, the symplectic Fenchel conjugate of F is related by the ordinary Fenchel conjugate $F^{*}$ of F as follows:

$$F^{*\omega }\left(z\right)=F^{*}\left(J\left(z\right)\right).$$

Thus in that case the symplectic Bregman divergence amounts to an ordinary Bregman divergence:

$$\begin{array}{ccc}\hfill B_F^{\omega }(z_1:z_2)& =& F\left(z_1\right)-F\left(z_2\right)-\omega ({\nabla }^{\omega }F\left(z_2\right),z_1-z_2),\hfill \\ & =& F\left(z_1\right)-F\left(z_2\right)+\omega (-{\nabla }^{\omega }F\left(z_2\right),z_1-z_2),\hfill \\ & =& F\left(z_1\right)-F\left(z_2\right)+\omega (J(\nabla F\left(z_2\right)),z_1-z_2),\hfill \\ & =& F\left(z_1\right)-F\left(z_2\right)-\langle \langle z_1-z_2,\nabla F\left(z_2\right)\rangle \rangle ,\hfill \\ & =& B_F(z_1:z_2).\hfill \end{array}$$

Property 1. 

When the symplectic form ω is induced by an inner product $\langle ·,·\rangle$ of X, the symplectic Bregman divergence $B_F^{\omega }(z_1:z_2)$ between $z_1=(x_1,y_1)$ and $z_2=(x_2,y_2)$ of $Z=X\times X$ amounts to an ordinary Bregman divergence with respect to the composite inner-product $\langle \langle z_1,z_2\rangle \rangle =\langle x_1,y_1\rangle +\langle x_2,y_2\rangle$:

$$B_F^{\omega }(z_1:z_2)=B_F(z_1:z_2)=F\left(z_1\right)-F\left(z_2\right)-\langle \langle z_1-z_2,\nabla F\left(z_2\right)\rangle \rangle .$$

Furthermore, if the potential function $F\left(z\right)$ is separable, i.e., $F\left(z\right)=F_1\left(x\right)+F_2\left(y\right)$ for Bregman generators $F_1$ and $F_2$, then we have $B_F^{\omega }(z_1:z_2)=B_{F_1}(x_1:x_2)+B_{F_2}(y_1,y_2)$ where the Bregman divergences $B_{F_1}$ and $B_{F_2}$ are defined with respect to the inner product of X.

Notice that the symplectic Fenchel–Young inequality can be rewritten using the ordinary Fenchel–Young inequality and the linear function J as:

$$\begin{array}{ccc}\hfill F\left(z\right)+F^{*\omega }\left(z^{\prime }\right)& \ge & \omega (z^{\prime },z),\hfill \\ \hfill F\left(z\right)+F^{*}\left(J\left(z^{\prime }\right)\right)\ge \langle \langle J\left(z^{\prime }\right),z\rangle \rangle .\end{array}$$

6. Summary, Discussion, and Perspectives

Since its inception in operations research, Bregman divergences [19] have proven instrumental in many scientific fields including information theory, statistics, and machine learning, just to cite a few. Let $(X,\langle ·,·\rangle )$ be a Hilbert space, and $F:X\to \mathbb{R}$ a strictly convex and smooth real-valued function. Then the Bregman divergence induced by F is defined in [19] (1967) by

$$B_F(x_1:x_2)=F\left(x_1\right)-F\left(x_2\right)-\langle x_1-x_2,\nabla F\left(x_2\right)\rangle .$$

In this work, we consider finite-dimensional vector spaces equipped with an inner product.

In information geometry [22,34,35], a smooth dissimilarity $\mathcal{D}(p,q)$ between two points p and q on an n-dimensional smooth manifold M induces a statistical structure on the manifold [36], i.e., a triplet $(g,\nabla ,{\nabla }^{*})$ where the Riemannian metric tensor g and the torsion-free affine connections ∇ and ${\nabla }^{*}$ are induced by the divergence $\mathcal{D}$. The duality in information geometry is expressed by the fact that the mid-connection ${\displaystyle \frac{\nabla +{\nabla }^{*}}{2}}$ corresponds to the Levi-Civita connection induced by g. To build the divergence-based information geometry [37], the divergence $\mathcal{D}(p,q)$ is interpreted as a scalar function on the product manifold $M\times M$ of dimension $2n$. Thus, the divergence $\mathcal{D}$ is called a contrast function [36] or yoke [38]. Conversely, a statistical structure $(g,\nabla ,{\nabla }^{*})$ on an n-dimensional manifold M induces a contrast function [39]. When the statistical manifold $(M,g,\nabla ,{\nabla }^{*})$ is dually flat with $\theta (·)$ the global ∇-affine coordinate system and $\eta (·)$ the global ${\nabla }^{*}$-affine coordinate system [40], there exists two dual global potential functions $\varphi$ and ${\varphi }^{*}$ on the manifold M such that $\varphi \left(p\right)=F\left(\theta \right(p\left)\right)$ and ${\varphi }^{*}\left(\eta \left(p\right)\right)=F^{*}\left(\eta \left(p\right)\right)$ where $F^{*}\left(\eta \right)$ is the Legendre–Fenchel convex conjugate of $F\left(\theta \right)$. The canonical dually flat divergence on M is then defined by

$$\mathcal{D}(p,q)=F\left(\theta \left(p\right)\right)+F^{*}\left(\eta \left(p\right)\right)-\sum _{i=1}^n{\theta }_i\left(p\right){\eta }_i\left(q\right),$$

and amounts to a Fenchel–Young divergence or equivalently a Bregman divergence:

$$\mathcal{D}(p,q)=Y_F(\theta \left(p\right):\theta \left(q\right))=B_F(\theta \left(p\right):\theta \left(q\right)),$$

where the Fenchel–Young divergence is defined by

$$Y_F(\theta :{\eta }^{\prime })=F\left(\theta \right)+F^{*}\left({\eta }^{\prime }\right)-\sum _{i=1}^n{\theta }^i{\eta }_i^{\prime }.$$

The Riemannian metric g of a dually flat space can be expressed as $g=\nabla \mathrm{d}\varphi ={\nabla }^{*}\mathrm{d}{\varphi }^{*}$ or in the $\theta$-coordinates by $g_{ij}\left(\theta \right)={\displaystyle \frac{{\partial }^2}{\partial {\theta }_i\partial {\theta }_j}}F\left(\theta \right)$ and in the $\eta$-coordinates by $g_{ij}\left(\eta \right)={\displaystyle \frac{{\partial }^2}{\partial {\eta }_i\partial {\eta }_j}}{F}^{*}\left(\eta \right)$. That is, g is a Hessian metric [40], $(g,\nabla )$ a Hessian structure and $(g,{\nabla }^{*})$ a dual Hessian structure. In differential geometry, $(M,g,\nabla )$ is called a Hessian manifold which admits a dual Hessian structure $(g,{\nabla }^{*})$. In particular, a Hessian manifold is of Koszul type [40] when there exists a closed 1-form $\alpha$ such that $g=\nabla \alpha$.

Remark 4. 

Notice that the potential functions F and $F^{*}$ are not defined uniquely although the potential functions ϕ and ${\varphi }^{*}$ on the manifold are. Indeed, consider the generator $\overline{F}\left(\theta \right)=F(A\theta +b)+\langle c,\theta \rangle +d$ for invertible matrix $A\in \mathrm{GL}(d,\mathbb{R})$, vectors $b,c\in {\mathbb{R}}^d$ and scalars $d\in \mathbb{R}$. The gradient of the generator $\overline{F}$ is $\eta =\nabla \overline{F}\left(\theta \right)=A^{\top }\nabla F(A\theta +b)+c$. Solving the equation $\nabla \overline{F}\left(\theta \right)=\eta$ yields the reciprocal gradient $\theta \left(\eta \right)=\nabla \overline{G}\left(\eta \right)=A^{-1}\nabla G\left(A^{-\top }(\eta -c)\right)-b$ from which the Legendre convex conjugate is obtained as $\overline{G}\left(\eta \right)=\langle \eta ,\nabla \overline{G}\left(\eta \right)\rangle -F(\nabla \overline{G}\left(\eta \right))$. We have $B_F({\theta }_1:{\theta }_1)=B_{\overline{F}}({\overline{\theta }}_1:{\overline{\theta }}_2)$ where $\overline{\theta }=A^{-1}(\theta -b)$.

It has been shown that a divergence $\mathcal{D}$ also allows one to define a symplectic structure $\omega$ on a statistical manifold [38,41]. The symplectic vector space $({\mathbb{R}}^{2n},{\omega }_0)$ viewed as a symplectic manifold has symplectic form ${\omega }_0={\sum }_{i=1}^n\mathrm{d}{x}^i\wedge \mathrm{d}{y}_i=-d\left({\sum }_{i=1}^n{y}_i\mathrm{d}{x}^i\right)$. There are no local invariants but only global invariants on symplectic manifolds (symplectic topology). That is, a symplectic structure is flat.

In this expository paper, we have defined symplectic Fenchel–Young divergences and equivalent symplectic Bregman divergences by following the study of geometric mechanics reported in [13,14]. The symplectic Bregman divergence between two points $z_1$ and $z_2$ on a symplectic vector space $(Z,\omega )$ induced by a convex potential function $F:Z\to \mathbb{R}$ is defined by

$$B_F^{\omega }(z_1:z_2)=F\left(z_1\right)-F\left(z_2\right)+\omega (z_1-z_2,{\nabla }^{\omega }F\left(z_2\right)),$$

where ${\nabla }^{\omega }F$ has been called the symplectic gradient in this paper, and assumed to be the unique symplectic subdifferential at any $z\in Z$, i.e., ${\partial }^{w}F\left(z\right)=\left\{{\nabla }^{\omega }F\left(z\right)\right\}$. Symplectic Bregman divergences are used to define Bregman divergences on dual systems (Figure 3). In the particular case of dual system $(X,X,\langle ·,·\rangle )$, we recover ordinary Bregman divergences with composite inner products.

Figure 3.

Bregman divergences generalized to dual systems $(X,Y,b(·,·))$: A symplectic form $\omega$ on the space $Z=X\oplus Z$ is induced by the pairing product. The Bregman divergence on the dual system is then defined as the symplectic Bregman divergence on the symplectic vector space $(Z,\omega )$.

In finite $2n$-dimensional symplectic vector spaces, linear symplectic forms $\omega$ can be represented by symplectic matrices of the matrix group $\mathrm{Sp}\left(2n\right)$. Buliga and de Saxcé [13,14] considered geometric mechanics with dissipative terms, and stated the following “symplectic Brezis–Ekeland–Nayroles principle” (SBEN principle for short):

Definition 6 

(SBEN principle [13,14]). The natural evolution path $z\left(t\right)=z_{\mathrm{rev}}\left(t\right)+z_{\mathrm{irr}}\left(t\right)\in Z$ for $t\in [0,T]$ in a geometric mechanic system with convex dissipation potential $\varphi \left(z\right)$ minimizes among all admissible paths ${\int }_0^T{Y}_F^{\omega }(\dot{z}\left(t\right),{\dot{z}}_{\mathrm{irr}}\left(t\right))\mathrm{d}t$ and satisfies $Y_F^{\omega }(\dot{z}\left(t\right),{\dot{z}}_{\mathrm{irr}}\left(t\right))=0$ for all $t\in [0,T]$, where $Y_F^{\omega }$ denotes the symplectic Fenchel–Young divergence induced by ϕ, and $z_{\mathrm{rev}}\left(t\right)$ and $z_{\mathrm{irr}}\left(t\right)$ are the reversible and irreversible parts of the particle $z\left(t\right)$, respectively.

The decomposition of $z=z_{\mathrm{rev}}+z_{\mathrm{irr}}$ into two parts can be interpreted as Moreau’s proximation [42,43] associated to the potential function $\varphi$: Indeed, let $F\left(z\right)$ be a convex function of $Z={\mathbb{R}}^d$. Then for all $z\in {\mathbb{R}}^d$, we can uniquely decompose z as $z=z+z^{*}$ such that $F\left(z\right)+F^{*}\left(z^{*}\right)=\langle z,z*\rangle$ (Fenchel–Young equality) where $z^{*}=\nabla F\left(z\right)$ (see Proposition in Section 4 of [42]). The part z is called the proximation with respect to F, and the part $z^{*}$ is the proximation with respect to the convex conjugate $F^{*}$.

We may consider the non-separable potential functions $F\left(z\right)=F(x,y)=xf(y/x)$ which are obtained from the perspective transform [44,45] of arbitrary convex functions $f\left(u\right)$ to define symplectic Bregman divergences. The perspective functions $F(x,y)$ are jointly convex if and only if their corresponding generators f are convex. Such perspective transforms play a fundamental role in information theory [46] and information geometry [22].

In machine learning, symplectic geometry has been used for designing accelerated optimization methods [16,47] (Bregman–Lagrangian framework) and physics-informed neural networks [17,18] (PINNs).

This paper aims to spur interest in either designing or defining symplectic divergences from first principles, and to demonstrate their roles when studying thermodynamics [48] or the learning dynamics of ML and AI systems.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

Author Frank Nielsen is employed by the company Sony Computer Science Laboratories Inc. The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The author declares no conflicts of interest.

Funding Statement

This research received no external funding.