A Transportation L^p Distance for Signal Analysis

Abstract

Transport based distances, such as the Wasserstein distance and earth mover'sdistance, have been shown to be an effective tool in signal and image analysis. The success of transport based distances is in part due to their Lagrangian nature which allows it to capture the important variations in many signal classes. However these distances require the signal to be nonnegative and normalized. Furthermore, the signals are considered as measures and compared by redistributing (transporting) them, which does not directly take into account the signal intensity. Here we study a transport-based distance, called the TL^p distance, that combines Lagrangian and intensity modelling and is directly applicable to general, non-positive and multi-channelled signals. The distance can be computed by existing numerical methods. We give an overview of the basic properties of this distance and applications to classification, with multi-channelled non-positive one-dimensional signals and two-dimensional images, and color transfer.

1 Introduction

Enabled by advances in numerical implementation [4, 5, 16, 53, 71, 74], and their Lagrangian nature, transportation based distances for signal analysis are becoming increasingly popular in a large range of applications. Recent applications include astronomy [9, 18, 19], biomedical sciences [3, 25–27, 77, 81, 82, 88, 89], colour transfer [14, 17, 49, 62, 63], computer vision and graphics [7, 44, 60, 65, 68, 74, 75], imaging [36, 40, 64], information theory [78], machine learning [1, 15, 20, 34, 37, 48, 76], operational research [69] and signal processing [54, 58].

The success of transport based distances is due to the large number of applications that consider signals that are Lagrangian in nature (spatial rearrangements, i.e. transport, are a key factor when considering image differences). Many signals contain similar features for which transport based distances will outperform distances that only consider differences in intensity, such as the L^p distance. Optimal transport (OT) distances, for example the earth mover'sdistance or Wasserstein distance, are examples of transport distances. However these distances do not directly account for signal intensity. The L^p distance is the other extreme, this distance is based on intensity and does not take into account Lagrangian properties.

In this paper we develop the TL^p distance introduced in [21] which combines both Lagrangian and intensity based modeling. Our aim is to show that by including both transport and intensity within the distance we can better represent the similarities between classes of data in many problems. For example, if a distance can naturally differentiate between classes, that is the within class distance is small compared to the between class separation, then the classification problem is made easier. This requires designing distances that can faithfully represent the structure within a given data set.

In the majority of the literature optimal transport distances interpret signals as either probability measures or as densities of probability measures. This places restrictions on the type of signals one can consider. Probability measures must be non-negative, integrate to unity and be real valued (i.e. cannot be applied to multi-channelled signals). In order to apply OT distances to a wider class of signals one has to use ad-hoc methods, which do not necessarily preserve metric properties, to transform the signal into a probability measure. This can often dampen the features, for example renormalization may reduce the intensity range of a signal. We do however note the works of Liero, Mielke and Savaré [42, 43], Chizat, Peyré, Schmitzer and Vialard [13, 14] and Kondratyev, Monsaingeon and Vorotnikov [38] who develop an optimal transport metric that is applicable to un-normalised positive measures. Similarly Pele and Wermen propose a variant of the earth movers distance that is applicable to un-normalised positive measures [59]. Whilst these are also promising avenues research there are still restrictive assumptions, such as signals must be non-negative and real valued, required in order to apply these distances.

Extensions to matrix valued optimal transport have been made in [11, 12, 52]. In [52] the authors propose a method for defining an optimal transport distance between matrix valued densities. As in the scalar valued case for a suitable class of matrix valued densities there exists a non-empty set of couplings. A matrix valued optimal transport distance is defined by minimising over the set of couplings a cost function that penalises both the transport of mass and rotations of the coupling. Similar ideas in [11, 12] use an analogue of the Brenier and Benamou formulation of optimal transport [4] to define a matrix valued optimal transport distance. Whilst these distances are applicable to matrix valued signals they still require the assumption of positivity (in this case positive definite). These distances are also specific to n × n matrices which does not include vectors.

The TL^p distance does not need the signal to be a probability measure and therefore the above restrictions do not apply. Rather, the TL^p distance models the intensity directly. The applicability of the distance is sufficiently general as to include non-positive, multi-channelled and un-normalised signals on discrete or continuous domains.

Another property of OT distances, due to the lack of intensity modeling, is its insensitivity to high frequency perturbations. This is due to transport being on the order of the wavelength of the perturbation. Depending upon the application this can be an advantage or a disadvantage. For example in texture modeling one would want to be able to discriminate between a highly oscillating image and a constant image. On the other hand, the lack of sensitivity to high frequency noise, make the OT distance stable under such perturbations. Since the TL^p distance directly models intensity then it inherits sensitivity to high frequency noise.

The aim of this paper is to develop the TL^p distance and demonstrate its applicability in a range of applications. We consider classification problems on data sets where we show that the TL^p distance better represents the underlying geometry, i.e. achieves a better between class to within class distance, than popular alternative distances.

We also consider the colour transfer problem in a context where spatial information, as well as intensity, is important. To apply standardised tests in applications such as medical imaging it is often necessary to normalise colour variation [33, 45, 73]. One solution is to match the means and variance of each colour channel (in some colour space e.g. RGB or LAB). However, by transferring the colour of one image onto the other it is possible to recolour an image with exactly the same colour profile.

A popular method is to use the OT distance on the histogram of images [14, 17, 49, 62, 63]. This allows one to take into account the intensity of pixels but includes no spatial information. The TL^p distance is able to include both spatial and intensity information.

Our methodology, therefore, has more in common with registration methods that aim to find a transformation that maximizes the similarity between two images where our measure of similarity includes both spatial and intensity information. One should compare our approach to [27] where the authors develop a numerical method for the Monge formulation of OT with the addition of an intensity term for image warping and registration. However, unlike the method presented in [27], the formulation presented here defines a metric.

Paper Overview

The outline for this paper is the following. In the next section we review OT and give a formal definition of the TL^p distance followed by examples to illustrate its features and to compare with the OT and L^p distances. In Section 3 we give a more general definition and explain some of its key properties. In Section 4 we include applications of the TL^p distances. We first consider classification on synthetic and real- world signals and images. The data sets contain non-positive and un-normalised signals in either one or two dimensions. In addition one of the data sets is multi-channelled. A further application to the colour transfer problem is then given. Conclusions are given in Section 5.

2 Formal Definitions and Examples

2.1 Review of Optimal Transport and the TL^p Distance

We begin by reviewing optimal transport in first the Kantorovich formulation and then the Monge formulation.

The Kantorovich Formulation of Optimal Transport

For measures μ and ν on Ω ⊂ ℝ^d with the same mass and a continuous cost function c: Ω × Ω → [0, ∞) the Kantorovich formulation of OT is given by

$$\text{OT}\left(\mu ,\nu \right)=\underset{\pi }{min}{\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}c\left(x,y\right)\mathrm{d}\pi \left(x,y\right)$$ (1)

where the minimum is taken over probability measures π on Ω × Ω such that the first marginal is μ and the second marginal is ν, i.e. π (A × Ω) = μ (A) and π (Ω × B) = ν(B) for all open sets A and B. We denote the set of such π by П(μ, ν). We call measures π ∈ П(μ, ν) transport plans since π (A × B) is the amount of mass in A that is transferred to B. Minimizers π* of OT(μ, ν), which we call optimal plans, exist when c is lower semi-continuous [84, Theorem 4.1]. When c is a metric OT(μ, ν) is also known as the earth mover's distance.

A common choice is $c\left(x,y\right)={\left|x-y\right|}^p={\sum }_{i=1}^d{\left|x_i-y_i\right|}^p$ in which case we define $d_{\text{OT}}\left(\mu ,\nu \right)=\sqrt[p]{\text{OT}\left(\mu ,\nu \right)}$. When p = 2 this is known as the Wasserstein distance. We will call d_OT the OT distance. With an abuse of notation we will sometimes write d_OT(f, g) when μ and ν have densities f and g respectively.

When μ has a continuous density then the support of any optimal plan π* is contained on the graph of a function T*. In particular this implies π*(A, B) = μ ({x : x ∈ A, T*(x) ∈ B}) and furthermore that the optimal plan defines a mapping between μ and ν, see for example Figure 1a. This leads us to the Monge formulation of OT.

Figure 1.

On the left an example optimal transport map for OT, on the right an illustration of the pushforward measure.

The Monge Formulation of Optimal Transport

An appealing property of optimal transport distances are their formulation in a Lagrangian setting. One can rewrite the optimal transport problem in the Monge formulation as

$${\text{OT}}_M\left(\mu ,\nu \right)=\underset{T}{\text{inf}}{\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}c\left(x,T\left(x\right)\right)\mathrm{d}\mu \left(x\right)$$ (2)

where the infimum is taken over transport maps T : Ω→ Ω that rearrange μ into ν, i.e. ν = T_#μ where we define the pushforward of μ onto the range of T by T_#μ(A) = μ(T⁻¹(A)), see Figure 1b. Historically the Monge formulation comes before the Kantorovich formulation; Monge formulated OT for the cost function c(x, y) = |x − y| in 1781 [47] and Kantorovich formulated OT (whilst being unaware of Monge's work) in 1942 [31]. In 1948 Kantorovich made the connection between his work and Monge's [32].

The Monge formulation is a non-convex optimization problem with nonlinear constraints. However when, for example, μ and ν have densities, then optimal transport maps T* exist and give a natural interpolation between two measures. In particular when c(x, y) = |x − y|^p the map T_t(x) = (1 – t)x + tT* (x) describes the path of particle x and furthermore the measure of μ pushed forward by T_t is the geodesic (shortest path) between μ and ν. This property has had many uses in transport based morphometry applications such as biomedical [3, 55, 80, 85], super-resolution [36] and has much in common with large deformation diffeomorphism techniques in shape analysis [23, 29].

Optimal Transport in Signal and Image Processing

To further motivate our development of the TL^p distance we point out some features of optimal transport important to signal and image processing. We refer to [35] and references therein for more details and a review of the subject.

Key to the success of OT is the ability to provide generative models which accurately represent various families of data distributions. The success and appeal of OT owes to (1) ability to capture well the signal variations due to spatial rearrangements (shifts, translations, transport), (2) that OT distances are theoretically well understood and have appealing features (for example the Wasserstein distance has a Riemannian structure and geodesics can be characterized), (3) efficiency and accuracy of numerical methods, (4) simplicity compared to other Lagrangian methods such as large deformation diffeomorphic metric mapping.

The Monge formulation of OT defines a mapping between images which has been used in, for example, image registration [24–27, 50, 82, 89] where one wishes to find a common geometric reference frame between two or more images. In addition to the properties listed above the success of OT is due to the fact that (5) the Monge problem is symmetric (i.e. if T is the optimal map from the first image to the second, then T⁻¹ is the optimal map from the second image to the first) and (6) OT provides a landmark-free and parameter-free registration scheme.

We now introduce the TL^p distance in the simplest setting.

The Transportation L^p Distance

In this paper we use the TL^p distance (given in more generality in the next section), for functions f, g : Ω → ℝ^m defined by

$$d_{\mathit{\text{TL}}}^p\left(f,g\right)=\underset{\pi }{min}{\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}{\left|x-y\right|}^p+{\left|f\left(x\right)-g\left(y\right)\right|}^p\mathrm{d}\pi \left(x,y\right)$$

where the minimum is taken over all probability measures π on Ω × Ω such that both the marginals are the Lebesgue measure ℒ on Ω, i.e. π ∊(ℒ, ℒ). This can be understood in the following two ways.

The first is as an optimal transport distance of the Lebesgue measure with itself and cost c(x, y) = |x − y|^p + |f(x) − g(y)|^p. This observation allows one to apply existing numerical methods for OT where the effective dimension is d (recall that Ω ⊆ ℝ ^d). For example, the Sinkhorn framework can be adapted to compute an entropy regularised approximation of the TL^p distance, see Appendix B. 2 for more details.

The second is as an OT distance between the Lebesgue measure raised onto the graphs of f and g. That is, given f, g : Ω → ℝ^m then we define the measures μ̃, ν̃ on the graphs of f and g by

$$\stackrel{\sim }{\mu }\left(A\times B\right)=\mathcal{L}\left(\left\{x:x\in A,f\left(x\right)\in B\right\}\right)$$

and

$$\stackrel{\sim }{\nu }\left(A\times B\right)=\mathcal{L}\left(\left\{y:y\in A,g\left(y\right)\in B\right\}\right)$$

for any open sets A ⊆ Ω, B ⊆ ℝ^m. For example, given the function f : [0,1] → [0,1] defined by f(x) = x and the Lebesgue measure on [0,1], the pushforward of the measure μ onto the graph of f is the measure $\stackrel{\sim }{\mu }\left(C\right)=\frac{1}{\sqrt{2}}\text{Length}\left(\left\{x:\left(x,x\right)\in C\right\}\right)$ for any (measurable) C ⊂ [0,1]²; it is intuitive that μ̃(C) should be proportional to Length({x : (x, x) ∈ C}), the constant of proportionality comes from $\stackrel{\sim }{\mu }\left({\left[0,1\right]}^2\right)=\mu \left(\left[0,1\right]\right)=1$. See also Figure 1b for an example where the measure μ is Gaussian. The TL^p distance between f and g is then the OT distance between μ̃ and ν̃.

Transport (i.e. matching) with respect to the TL^p distance is of the form (x, f (x)) ↦ (y, g(y)) and therefore has two components. We refer to horizontal transport as the transport x ↦ y in Ω, and vertical transport as the transport f (x) ↦ g(y).

Although the TL^p distance is a special case of OT we will, in order to make a clearer distinction between classical OT distances and the TL^p distance, assume that c(x, y) = |x − y|^p in (1).

In the next section we discuss the behaviour of the TL^p distance through three examples.

2.2 Examples Illustrating the Behaviour of TL^p

No mass renormalization

Unlike for OT, in the TL^p distance there is no need to assume that f and g are non-negative or that they have the same mass. If a signal is negative then a typical (ad-hoc) fix in OT is to add a constant to make the signal non-negative before computing the distance. How to choose this constant is often unclear unless a lower bound is known a-priori. Furthermore this may damage sensitivity to translations as the defining features of the signal become compressed. For example, considering the functions in Figure 2a, let g = f (∙ – ℓ) be the translation of f. OT will lose sensitivity when comparing $\widehat{f}=\frac{f+\alpha }{f\left(f+\alpha \right)}$ and $\widehat{g}=\frac{g+\alpha }{f\left(g+\alpha \right)}$. In particular $d_{OT}\left(\widehat{f},\widehat{g}\right)$ scales with the height of the renormalised function, which is of the order of $\frac{1}{\alpha }$, and the size of the shift: $d_{\text{OT}}\left(\widehat{f},\widehat{g}\right)\alpha \frac{h_o\ell }{\alpha }$ where h₀ is the height of f. To ensure positivity one must choose a large but this also implies a small OT distance. Note also that both L^p and TL^p distances are invariant under adding a constant whereas OT is not.

Figure 2.

A Comparison of TL^p with OT.

Another approach to apply the OT distance to non-positive signals is to decompose each signal into the positive and negative components f = f⁺− f⁻, where f⁺ = max{0, f} and f⁻ = max{0, −f}, and compute the OT distance between each component. Whilst this method may be reasonable depending on the application it is not invariant under addition which could produce some unnatural properties. For example, consider two signals, one slightly negative and one slightly positive. Then applying an (unbalanced) OT distance on the positive and negative components is equivalent to matching both signals to zero. Adding a small constant to the negative signal so that both signals are positive produces a qualitatively different result. Since both the TL^p and L^p distances are invariant under addition neither has this property. To mitigate this issue Bonneel, van de Panne, and Heidrich [8] consider signals decomposed into frequency bands. This also allows them to directly take into account the signal frequency. In some sense our approach is complimentary, as we seek a way to take into account the signal intensity.

Sensitivity to High Frequency Perturbations

The TL^p distance inherits sensitivity to high frequency perturbations from the L^p distance. For example, let g = f + Aξ where ξ is a high frequency perturbation with amplitude A and wavelength ω. Suppose for simplicity that f is constant and ξ is a sinusoid (and that both signals are positive). The function g is the landscape consisting of piles of earth and trenches and f is the flat landscape. The OT distance between f and g measures the cost of moving piles of earth into trenches (in the most efficient manner). The two factors which determine the OT distance are the total amount of earth to be moved (which we assume fixed) and how far we move each piece of soil, which is determined by the wavelength. Hence the OT distance between f and g is on the order of the wavelength ω of ξ, which is small, and independent of the amplitude A. On the other hand both the TL^p distance and the L^p distance are independent of the wavelength but scale linearly with amplitude, see Figure 2b. In particular OT is insensitive to high frequency noise regardless of the size of the amplitude whereas both TL^p and L^p distances scale linearly with the amplitude.

Ability of the TL^p Distance to Track Translations

Another desirable property of both TL^p and OT distances are their ability to keep track of translations for longer than the L^p distance. Let f = Aχ_[0,1] be the indicator function of the set [0,1] on ℝ scaled by A > 1 and g(x) = f (x – ℓ) the translation of f by ℓ. Once ℓ > 1 then the L^p distance can no longer tell how far apart two humps are. On the other hand OT distances can track the hump indefinitely. In this example the TL^p distance couples the graphs of f and g in one of three ways, see Figure 3. The first is when the transport is horizontal only in the graph (Figure 3 top left). In the second (top right) there is a mixture of horizontal and vertical transport. And in the third there is only vertical transport (bottom left), in which case the TL^p distance coincides with the L^p distance. One can calculate the range of the TL^p distance which is on the order of A.

Figure 3.

TL² transport between f (x) = Aχ_[0,1] (black) and g(x) = f(x−ℓ) (red) and the TL² distance (red), L² distance (blue) and OT (black) between f and g (bottom right).

3 Definitions and Basic Properties of the TL^p Distance

In the previous section we defined the TL^p distance for signals defined with respect to the Lebesgue measure. In this section we generalise to signals defined on a general class of measures. We let L^p (μ) be the space of functions f such that ∫_Ω|f (x)|^p dμ(x) < ∞. This is a Banach space with the usual norm.

We treat a signal as a pair (f, μ) where μ ∈ 𝒫_p(Ω) (the set of probability measures with finite p^th moment) and f : Ω →ℝ^m with f ∈ L^p(μ). The generality considered here allows us to treat continuous and discrete signals simultaneously as well as allowing one to design the underlying measure in order to emphasise certain parts of the signal. We are also able to compare signals with different discretisations. However, unless otherwise stated, μ = v is the Lebesgue measure. There is no assumption on the dimension m of the codomain. This allows us to consider multi-channelled signals.

The $T{L}_{\lambda }^p$ distance for pairs (f, μ) ∈ TL^p where

$$T{L}^p:=\left\{\left(f,\mu \right):f\in L^p\left(\mu \right),\mu \in {\mathcal{P}}_p\left(\mathrm{\Omega }\right)\right\}$$

is defined by

$$d_{T{L}_{\lambda }^p}^p\left(\left(f,\mu \right),\left(g,\nu \right)\right)=\underset{\pi \in \Pi \left(\mu ,\nu \right)}{min}{\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}{c}_{\lambda }\left(x,y;f,g\right)\mathrm{d}\pi$$ (3)

$$c_{\lambda }\left(x,y;f,g\right)=\frac{1}{\lambda }{\left|x-y\right|}^p+{\left|f\left(x\right)-g\left(y\right)\right|}^p$$ (4)

and Π(μ, ν) is the space of measures on Ω × Ω such that the first marginal is μ and the second marginal is ν. Note that if f = g is constant then we recover the OT distance between the measures μ and ν. In the special cases, when μ = ν = ℒ are the Lebesgue measure, we write $d_{T{L}_{\lambda }^p}\left(f,g\right):=d_{T{L}_{\lambda }^p}\left(\left(f,\mathcal{L}\right),\left(g,\mathcal{L}\right)\right)$ and, when λ = 1, $\lambda =1,d_{T{L}^p}\left(f,g\right):=d_{T{L}_1^p}\left(f,g\right)$. The result of [21, Proposition 3.3] implies that $d_{T{L}_{\lambda }^p}$ is a metric on TL^p.

Proposition 3.1. [21] For any p ∈ [1, ∞] and λ > 0, $\left(T{L}^p,d_{T{L}_{\lambda }^p}\right)$ is a metric space.

When μ = ν = ℒ is the Lebesgue measure then an admissible plan is the identity plan: π (A × B) = ℒ (A ∩ B). This implies that the $T{L}_{\lambda }^p$ distance is bounded above by the L^p distance (for any λ).

In fact the parameter λ controls how close the distance is to an L^p distance. As λ → 0 then the cost of horizontal transport: $\frac{1}{\lambda }{\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}{\left|x-y\right|}^p\mathrm{d}\pi \left(x,y\right)$, is very expensive which favours transport plans that are approximately the identity mapping. Hence $d_{T{L}_0^p}\left(f,g\right):={lim}_{\lambda \to 0}{d}_{T{L}_{\lambda }^p}\left(f,g\right)={\Vert f-g\Vert }_{L^p}$. The following result, and the remainder of the results in this section, can be found in [79].

Proposition 3.2. [79] Let f, g ∈ L^p (with respect to the Lebesgue measure). The $T{L}_{\lambda }^p$ distance is decreasing as a function of λ and

$$\underset{\lambda \to 0}{lim}{d}_{T{L}_{\lambda }^p}\left(f,g\right)={\Vert f-g\Vert }_{L^p}.$$

Moreover, if either f or g is Lipschitz then

$$d_{T{L}_{\lambda }^p}^p\left(f,g\right)\ge \{\begin{array}{cc}{\varepsilon }^{p-1}\left(\lambda \right){\Vert f-g\Vert }_{L^p}^p& \mathit{\text{if}}p>1\\ {\Vert f-g\Vert }_{L^p}^p& \mathit{\text{if}}p=1\mathit{\text{and}}\lambda <\frac{1}{\kappa }\end{array}$$

where $\epsilon \left(\lambda \right)=\frac{1}{1+{\left(\lambda \kappa \right)}^{\frac{1}{p-1}}}$ and k = (min{Lip(f), Lip(g)})^p.

The above proposition implies that, when p = 1, if $\frac{1}{\lambda }$ is chosen larger than the length scale given by the derivative then the $\mathit{\text{TL}}\frac{1}{\lambda }$ distance is exactly the L¹ distance.

Recall that we can consider the $T{L}_{\lambda }^p$ distance as an OT distance on the graphs of f and g:

$$d_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)=d_{\text{OT}}\left({\left(\text{Id}\times f\right)}_{\#}\mu ,{\left(\text{Id}\times g\right)}_{\#}\nu \right).$$ (5)

When there exists a map T : Ω × ℝ^m → Ω × ℝ^m realising the minimum of the Monge formulation of the RHS then we can understand the transport as a map (x, f (x)) ↦ (y, g(y)). We recall that we refer to the transport x ↦ y in the domain Ω as horizontal transport and transport f(x) ↦ g(y) in the codomain of f and g as vertical transport. We see that horizontal transport is cheap as λ ↦ ∞ and we only pay the cost of vertical transport. For example, if we consider f (x) = χ_[0,1] and g(x) = χ_[1,2] defined on the interval [0, 2] then the mapping T(x, y) = (T₁(x,y), T₂(x,y)) where

$$T_1\left(x,y\right)=\{\begin{array}{cc}x+1& \text{if}y=1\text{and}x\in \left[0,1\right]\\ x-1& \text{if}y=0\text{and}x\in \left[0,1\right]\end{array}$$

and

$$T_2\left(x,y\right)=\{\begin{array}{cc}1& \text{if}y=1\text{and}x\in \left[0,1\right]\\ 0& \text{if}y=0\text{and}x\in \left[0,1\right]\end{array}$$

defines a transport map on the support of (Id × f )_#ℒ. Furthermore, this implies

$$d_{T{L}_{\lambda }^p}^p\left(f,g\right)\le {\int }_0^1\frac{{\left|x-T_1\left(x,1\right)\right|}^p}{\lambda }+{\left|f\left(x\right)-g\left(T_1\left(x,1\right)\right)\right|}^p\mathrm{d}x+{\int }_1^2\frac{{\left|x-T_1\left(x,0\right)\right|}^p}{\lambda }+{\left|f\left(x\right)-g\left(T_1\left(x,0\right)\right)\right|}^p\mathrm{d}x=\frac{2}{\lambda }\to 0\text{as}\lambda \to \infty .$$

In this example $d_{T{L}_{\infty }^p}\left(f,g\right):={lim}_{\lambda \to \infty }{d}_{T{L}_{\lambda }^p}\left(f,g\right)=0$. More generally the $T{L}_{\infty }^p$ distance is an OT distance between the measures f_#μ and g_#ν.

Proposition 3.3. [79] Let Ω ⊆ ℝ^d, f, g : Ω → ℝ^m measurable functions and μ, ν ∈ 𝒫_p(Ω) where p ≥ 1, then

$$\underset{\lambda \to \infty }{lim}{d}_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)=d_{\text{OT}}\left(f_{\#}\mu ,g_{\#}\nu \right)$$

where d_OT is the OT distance (on 𝒫(ℝ^m)) with cost c(x, y) = |x − y|^p.

As the example before the proposition showed, $d_{T{L}_{\infty }^p}\left(f,g\right)$ is not a metric, however is non-negative, symmetric and the triangle inequality holds.

In [4, Section 2] the authors, using the fluid mechanics formulation of optimal transport, interpolate the optimal transport distance with quadratic cost with the L² distance. The resulting interpolated distance can still be written in the fluid mechanics formulation which naturally gives rise to geodesics. By contrast the TL^p distance interpolates between L^p and the optimal transport distance of the push forward measures. This is well defined for any p ≥ 1 (unlike the previous method which requires p = 2) however geodesics do not exist in the TL^p space. One must also treat the signals as probability measures in the approach of [4].

We observe that when μ is a uniform measure (either in the discrete or continuous sense) the measure f_#μ is the histogram of f. The OT distance between histograms is a popular tool in histogram specification. Minimizers to the Monge formulation of d_OT(f_#μ, g_#ν) define a mapping between the histograms f_#μ and g_#ν [49, 62, 63]. However this mapping contains no spatial information. If instead one uses minimizers to the Monge formulation of the $T{L}_{\lambda }^p$ distance (6) (λ < ∞) then one can include spatial information in the histogram specification. We explore this further in Section 4.4 and apply the method to the colour transfer problem.

It is well known that there exists a minimizer (when c is lower semi-continuous) for OT. Since the $T{L}_{\lambda }^p$ distance is closely related to an OT distance between measures in ℝ^d+m (i.e. measures supported on graphs) then there exists a minimizer to (3-4).

Proposition 3.4. [79] Let Ω ⊆ ℝ^d be open and bounded, f ∈ L^p(μ), g ∈ L^p(ν) where μ, ν ∈ 𝒫(Ω), λ ∈ [0, + ∞] and p ≥ 1. Under these conditions there exists an optimal plan π ∈ Π(μ,ν) realising the minimum in $d_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)$.

As in the OT case it is natural to set the $T{L}_{\lambda }^p$ distance in the Monge formulation (2). We can write

$$d_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)=\underset{T:T_{\#}\mu =\nu }{\text{inf}}{\int }_{\mathrm{\Omega }}c\lambda \left(x,T\left(x\right);f,g\right)\mathrm{d}\mu \left(x\right).$$ (6)

Minimizers to the above will not always exist. For example, consider when f = g then the $T{L}_{\lambda }^p$ distance is the OT distance between μ and ν. If one chooses $\mu =\frac{1}{3}{\delta }_{x_1}+\frac{1}{3}{\delta }_{x_2}+\frac{1}{3}{\delta }_{x_3}$ and $\nu =\frac{1}{2}{\delta }_{y_1}+\frac{1}{2}{\delta }_{y_2}$ where all of x_i, y_j are distinct then there are no maps T : {x₁, x₂, x₃} → {y₁, y₂} that pushforward μ to ν.

However, in terms of numerical implementation, an interesting and important case is when μ and ν are discrete measures (see also [83, pg 5, 14-15] for the following argument with the Monge OT problem). Let $\mu =\frac{1}{n}{\sum }_{i=1}^n{\delta }_{x_i}$ and $\nu =\frac{1}{n}{\sum }_{i=1}^n{\delta }_{y_i}$ then $\pi ={\left({\pi }_{ij}\right)}_{i,j=1}^n\in \Pi \left(\mu ,\nu \right)$ is a doubly stochastic matrix up to a factor of $\frac{1}{n}$, that is

$${\pi }_{ij}\ge 0\forall i,j,\sum _{i=1}^n{\pi }_{ij}=\frac{1}{n}\forall j\text{and}\sum _{j=1}^n{\pi }_{ij}=\frac{1}{n}\forall i,$$ (7)

and the $T{L}_{\lambda }^p$ distance can be written

$$d_{T{L}_{\lambda }^p}^p\left(\left(f,\mu \right),\left(g,\nu \right)\right)=min\sum _{i=1}^n\sum _{j=1}^{n}c\lambda \left(x_i,y_j;f,g\right){\pi }_{ij}$$ (8)

where the minimum is taken over π satisfying (7). It is known (by Choquet's Theorem, e.g. [67, Theorem 32.3]) that the solution to this minimisation problem is an extremal point in the matrix set Π(μ, ν). It is also known (by application of Birkhoff's Theorem, e.g. [6]) that extremal points in Π(μ, ν) are permutation matrices. This implies that there exists an optimal plan π* that can be written as ${\pi }_{ij}^{\ast }=\frac{1}{n}{\delta }_{j-\sigma \left(i\right)}$ for a permutation σ : {1,…,n} → {1,…,n}. Hence there exists an optimal plan to the Monge formulation of the $T{L}_{\lambda }^p$ distance.

Proposition 3.5. For any f ∈ L^p(μ) and g ∈ L^p(ν) where $\mu =\frac{1}{n}{\sum }_{i=1}^n{\delta }_{x_i}$ and $\nu =\frac{1}{n}{\sum }_{j=1}^n{\delta }_{yj}$ there exists a permutation σ : {1, 2, …, n} → {1, 2, …, n} such that

$$d_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)=\frac{1}{n}\sum _{i=1}^{n}c\lambda \left(x_i,x_{\sigma (i)};f,g\right).$$

The above theorem implies that in the uniform discrete case there exists optimal plans (which are matrices) which will be sparse. In particular, π* is an n × n matrix with only n non-zero entries. This motivates the use of numerical methods that can take advantage of expected sparsity in the solution (e.g. iterative linear programming methods such as [53]).

4 TL^p in Multivariate Signal and Image Processing

Written in the form (3) the $T{L}_{\lambda }^p$ distance is an OT distance between the measures μ and ν with the cost function c given by (4) and which depends upon f and g. Hence, to compute $T{L}_{\lambda }^p$ distances there are many algorithms for OT distances that we may apply, for example the multi-scale approaches of Schmitzer [71] and Oberman and Ruan [53], or the entropy regularized approaches of Cuturi [16] and Benamou, Carlier, Cuturi, Nenna and Peyre [5]. Our choice was the iterative linear programming method of Oberman and Ruan [53] for the multivariate signals which we find works well both in terms of accuracy and computation time. Our choice for the images was the entropy regularized solution due to Cuturi [5, 16]. Whilst this only produces an approximation of the TL^p distance we find it computationally efficient for 2D images. In particular this method regularizes the OT distance with εH(π) where H is entropy. We choose ε as small as possible whilst avoiding numerical instability. In practice this corresponds to a choice of ε ≈ 0.005. For convenience we include a review of the numerical methods in Appendix B.

With respect to choosing λ there are two approaches we could take. The first is to compute the TL^p distance for a range of λ and then use cross-validation. There are two disadvantages to this approach: we would still have to know the range of λ and computing the $T{L}_{\lambda }^p$ distance for multiple choices of λ would considerably increase computation time. The second approach, and the one we use for each example in this section, is to estimate λ by comparing length scales and desired behaviour. In particular we choose λ so that both horizontal and vertical transport make a contribution. For the applications in this section we want to stay away from the asymptotic regimes λ ≈ 0 and λ ≫. By balancing the vertical and horizontal length scale we can formally find an approximation of λ which in our results below works well. For example, if we expect a set of real valued time series to have range [f_min, f_max] and domain [t_min, t_max] then to balance the vertical and horizontal length scales we choose λ so that

$$\frac{{\left|t_{max}-t_{min}\right|}^p}{\lambda }\approx {\left|f_{max}-f_{min}\right|}^p.$$

We first consider two synthetic examples. Considering synthetic examples allows us to better demonstrate where the TL^p distance will be successful. In particular synthetic examples can simplify the analysis and allow us to draw attention to features that may be obscured in real world applications.

The first synthetic example considers three classes where we can analytically compute the within class distances and between class separation. This allows us to compare how well we expect TL^p distances to perform in a classification problem.

The second synthetic example uses simulated 2D data from one-hump and two-hump functions. We test how well the TL^p distance recovers the classes and compare with OT and L^p distances.

Our first real world application is in classifying multivariate times series and 2D images. We choose a multivariate time series data set where we expect transport based methods to be successful but OT cannot be immediately applied. That is, OT must be applied to measures that are real valued so cannot be directly applied to multi-channelled signals. As a benchmark we find the OT distance on each channel separately and take the average over all channels. This would seem reasonable when channels are independent but is not a good assumption in the AUSLAN dataset where we expect temporally correlated signals.

Our chosen data set consists of sequences of sign language data (we define the data set in more detail shortly) which contains the position of both hands (parametrised by 22 variables) at each time. The $T{L}_{\lambda }^p$ distance can treat these signals as functions f : [0,1] → ℝ²². We expect to see certain features in the signals however these may be shifted based on the speed of the speaker. The second data set contains 2D images that must be normalised in order to apply the OT distance, this distorts some of the features leading to a poor performance.

We repeat the classification experiment on the AT&T Database of Faces. This is a database of ten 2D greyscale images of forty subjects. Note that if the images were in colour then one cannot immediately apply the OT distance.

The second real world application is histogram specification and colour transfer. Histogram specification or matching, where one defines a map T that matches one histogram with another, is widely used to define a colour transfer scheme. In particular let $f:{\left(x_i\right)}_{i=1}^N\to {ℝ}^3$ represent a colour image by mapping pixels x_i to a colour f (x_i) (for example in RGB space), one defines a multidimensional histogram of colours on an image by $\phi \left(c\right)=\frac{1}{N}\#\left\{x_i:f\left(x_i\right)=c\right\}$. For colour images the histogram ϕ is a measure on ℝ³. For notational clarity we will call ϕ the colour histogram. One can equivalently define a histogram for grayscale images as a measure on ℝ.

Let ϕ and ψ be two colour histograms for images f and g respectively. The OT map T defines a rearrangement of ϕ onto ψ, that is ψ = T_#ϕ. In colour transfer the map T is used to colour the image f using the palette of g by $\widehat{f}\left(x\right)=g\left(T\left(x\right)\right)$.

The histogram contains only intensity information and in particular there is no spatial dependence. Using the $T{L}_{\lambda }^p$-optimal map we define spatially correlated histogram specification and explain how this can be applied to the colour transfer problem. We demonstrate that $T{L}_{\lambda }^p$ distance produces a visually more appealing solution than the OT solution when spatial information is important. We also observe that, for colour images, computing OT maps is a 3D problem (the domain and range of the transport maps is in colour space), whereas computing $T{L}_{\lambda }^p$ maps is a 2D problem (the transport maps pixels to pixels). By Proposition 3.3 when λ is large we can approximate the OT distance between histograms by the $T{L}_{\lambda }^p$ distance. For images that use the full spectrum of colours, i.e. the colour histograms are 256 × 256 × 256, the size of the discretisation is 256³ ≈ 16.8 × 10⁶. Hence the spatially correlated histogram specification method allows for a numerically efficient approximation of the OT induced histogram specification method when the size (in terms of number of pixels) of the images are less than 4096 × 4096.

4.1 1D Class Separation for Synthetic Data

Objective

We compare the expected classification power of TL^p, L^p and OT distances with three classes of 1D signals that differ by position (translations), shape (1 hump versus 2 hump) and frequency (hump versus chirp).

Data Sets

We consider data from three classes defined in Figure 4. The first class contains single hump function and the second class contains two hump functions. The third class consists of functions with one hump and one chirp, defined to be a high frequency perturbation of a hump. The classes are chosen to test the performance of the TL² distance with L² and OT distances with regards to identifying translations (where we expect the L² distance to do poorly) with a class containing high frequency perturbations (where we expect the OT distance to do poorly).

Figure 4.

For fixed α, β, γ ∈ (0,1) where β> α ≫ γ the definition of the classes 𝒞_i.

Methods

For a distance to have good performance in classification and clustering problems it should be able to separate classes. To be able to quantify this we use the ratio of ‘between class separation’ to ‘class coverage radius’ that we define now.

Let ${\mathcal{C}}_i^N={\left\{f_j^i\right\}}_{j=1}^N$ be a sample of N functions from class 𝒞_i. For a given radius r we let G_i(r) be the graph defined by connecting any two points in ${\mathcal{C}}_i^N$ with distance less than r. The distance will be defined using the $T{L}_{\lambda }^p$, L² and OT distances. Let $R_{T{L}_{\lambda }^p}\left({\mathcal{C}}_i^N\right)$ be the smallest r such that G_i(r) is a connected graph using the $T{L}_{\lambda }^p$ distance. Analogously we can define R_L^p and R_OT.

We define ‘between class separation’ as the Hausdorff distance between classes:

$$d_{H,\rho }\left({\mathcal{C}}_i^N,{\mathcal{C}}_j^N\right)=max\left\{\underset{f\in {\mathcal{C}}_i^N}{sup}\underset{g\in {\mathcal{C}}_j^N}{inf}\rho \left(f,g\right),\underset{g\in {\mathcal{C}}_j^N}{sup}\underset{f\in {\mathcal{C}}_i^N}{inf}\rho \left(f,g\right)\right\}$$

where we will consider p to be one of the TL², L² or OT distances. Large values of $d_{H,\rho }\left({\mathcal{C}}_i^N,{\mathcal{C}}_j^N\right)$ imply that the classes ${\mathcal{C}}_i^N$ and ${\mathcal{C}}_j^N$ are well separated.

When $R_{\rho }\left({\mathcal{C}}_i^N\right)\le d_{H,\rho }\left({\mathcal{C}}_i^N,{\mathcal{C}}_j^N\right)$ then we say that the class ${\mathcal{C}}_i^N$ is separable from class ${\mathcal{C}}_j^N$ since for any $f\in {\mathcal{C}}_i^N$ the nearest neighbour in $\left({\mathcal{C}}_i^N\cup {\mathcal{C}}_j^N\right)\backslash \left\{f\right\}$ is also in class ${\mathcal{C}}_i^N$. We define the pairwise property

$${\kappa }_{\mathit{\text{ij}}}\left(\rho ;N\right)=\frac{\mathbb{E}{d}_{H,\rho }\left({\mathcal{C}}_i^N,{\mathcal{C}}_j^N\right)}{max\left\{\mathbb{E}R\rho \left({\mathcal{C}}_i^N\right),\mathbb{E}R\rho \left({\mathcal{C}}_j^N\right)\right\}}$$

where we take the expectation over sample classes ${\mathcal{C}}_i^N$. We will assume that the distribution over each class is uniform in the parameter ℓ. When κ_ij (ρ; N) > 1 then we expect classes ${\mathcal{C}}_i^N$ and ${\mathcal{C}}_j^N$ to be separable from each other.

As a performance metric we use the smallest value of N such that κ_ij (ρ; N) ≥ 1. We let

$$N_{ij}^{\ast }\left(\rho \right)=min\left\{N:{\kappa }_{ij}\left(\rho ;N\right)\ge 1\right\}.$$

This measures how many data points we need in order to expect a good classification accuracy.

Results

We leave the calculation to the appendix but the conclusion is

$$\begin{array}{l}{N}_{12}^{\ast }\left(T{L}^2\right)<N_{12}^{\ast }\left(\text{OT}\right)<N_{12}^{\ast }\left(L^2\right)\\ N_{13}^{\ast }\left(T{L}^2\right)<N_{13}^{\ast }\left(\text{OT}\right)<N_{13}^{\ast }\left(L^2\right)\\ N_{23}^{\ast }\left(T{L}^2\right)<N_{23}^{\ast }\left(\text{OT}\right)<N_{23}^{\ast }\left(L^2\right).\end{array}$$

In each case the TL² distance outperforms the L² and OT distances.

In each class the L² distance has a larger value of R. This implies a larger data set is needed to accurately cover each class. This is due to the Lagrangian nature of signals within each class (translations) that is poorly represented by the L² distance. The OT distance has the lowest (and therefore best) value of R in each class. Since each class is Lagrangian then the OT distance is very small between functions of the same class.

When considering between class separation the TL² and L² distances coincide and give a bigger (and better) between class distance than the OT distance. Since the class 𝒞₃ can be written as a high frequency perturbation of functions in the class 𝒞₂ then, in the OT distance, the functions from class 𝒞₃ approximate functions from the class 𝒞₂. The distance $d_{H,\text{OT}}\left({\mathcal{C}}_2^N,{\mathcal{C}}_3^N\right)$ is therefore small so that one needs more data points in order to fully resolve these classes. We see a similar effect when considering d_H,OT for the other classes.

4.2 2D Classification for Synthetic Data

Objective

We use simulated data to illustrate better separation of the TL^p distance compared to L^p and OT distances for 2D data from two classes of 1-hump and 2-hump functions.

Data Sets

The data set consists of two dimensional images simulated from the following classes

$$\mathbb{P}=\left\{p{\left[0,1\right]}^2:p(x)=\alpha \varphi \left(x|\gamma ,\sigma \right),\gamma \sim \text{unif}\left({\left[0,1\right]}^2\right),\alpha \sim \text{unif}\left(\left[0.5,1\right]\right)\right\}$$

$$\mathbb{Q}=\left\{{[0,1]}^2:q\left(x\right)=\alpha \varphi \left(x|{\gamma }_1,\sigma \right)-\alpha \varphi \left(x|{\gamma }_2,\sigma \right),{\gamma }_1,{\gamma }_2\underset{\sim }{\text{iid}}\text{unif}\left({\left[0,1\right]}^2\right),\alpha \sim \text{unif}\left(\left[0.5,1\right]\right)\right\}$$

where ϕ (·|γ, σ) is the multivariate normal pdf with mean γ ∊ ℝ² and co-variance σ ∊ ℝ^2×2. We choose σ = 0.01 × Id where Id is the 2 × 2 identity matrix. The first class, ℙ, are the set of multivariate Gaussians restricted to [0,1]² with mean uniformly sampled in [0,1]² and weighted by a uniformly sampled in [0.5,1]. The second class, ℚ, are the set of weighted differences between two Gaussian pdf's restricted to [0,1]² with means γ₁, γ₂ sampled uniformly in [0,1]². Note that the second class contains non-positive functions. See Figure 5 for examples from each class.

Figure 5.

Example signals and results for the synthetic data in Section 4.2.

We simulate 25 from each set and denote the resulting set of functions by $\mathcal{F}={\left\{f_i\right\}}_{i=1}^N$ where N = 50.

Methods

Let $\left({\left\{f_i\right\}}_{i=1}^N,D_{T{L}_{\lambda }^2}\right)$ be a finite dimensional metric space where $D_{T{L}_{\lambda }^2}$ is the N × N matrix containing all pairwise $S_{L_{\lambda }^2}$ distances, i.e. $D_{T{L}_{\lambda }^2}\left(i,j\right)=d_{T{L}_{\lambda }^2}\left(f_i,f_j\right)$. Similarly for $\left({\left\{f_i\right\}}_{i=1}^N,D_{L^2}\right)$ and $\left({\left\{f_i\right\}}_{i=1}^N,D_{\text{OT}}\right)$ where the optimal transport distance is defined by $d_{\text{OT}}\left(f,g\right)=\sqrt{\text{OT}\left(f,g\right)}$ and OT is given by (1) for c(x, y) = |x − y|².

To apply the optimal transport distance we need to renormalise so that signals are all non-negative and integrate to the same value. We do this by applying the nonlinear transform $\mathcal{N}\left(f\right)=\frac{f-\beta }{\int \left(f-\beta \right)}$ where β = min _f∈ℱmin_x∈[0,1]² f(x). Neither the L² nor TL² distances require normalisation.

We use non-metric multidimensional scaling (MDS) [39] to represent the graph in k dimensions. More precisely the aim is to approximate $\left({\left\{f_i\right\}}_{i=1}^N,D.\right)$ by a metric space $\left({\left\{x_i\right\}}_{i=1}^N,D_{|\cdot |2}\right)$ embedded in ℝ^k (D_|.|2 is the matrix of pairwise distances using the Euclidean distance, i.e. D_|.|2 (i, j) = |x_i−x_j|₂). This is done by minimising the stress S defined by

$$S_{T{L}_{\lambda }^2}\left(k\right)=\frac{{\sum }_{i,j=1}^N{\left({\left|x_i-x_j\right|}_2^2-F\left(D_{T{L}_{\lambda }^2}\left(i.j\right)\right)\right)}^2}{{\sum }_{i,j=1}^N{\left|x_i-x_j\right|}_2^2}$$

over ${\left\{x_i\right\}}_{i=1}^N\subset {ℝ}^k$ and monotonic transformations F : [0, ∞) → [0, ∞), with $S_{L_{\lambda }^2}$, S_OT defined analogously. The classical solution to finding the MDS projection (for Euclidean distances) is to use the k dominant eigenvectors of the matrix of squared distances, after double centring, as coordinates weighted by the square root of the eigenvalue. More precisely, define $D^{\left(2\right)}=-\frac{1}{2}J{\left[{\left|f_i-f_j\right|}^2\right]}_{ij}$ where $J=\text{Id}-\frac{1}{N}\mathbb{I}$ and 𝕀 is the N × N matrix of ones. Let Λ_k be the matrix with the k largest eigenvalues of D⁽²⁾ on the diagonal and E_k to be the corresponding matrix of eigenvectors. Then $X=E_k{\mathrm{\Lambda }}_k^{\frac{1}{2}}$ is the MDS projection. Increasing the dimension of the projected space k leads to a better approximation. In Figure 5 we show the projection in L², TL² and the OT distances for k = 2 as well as the dependence of k on S for each choice of distance.

Results

Our results in Figure 5 show that the TL² distance is the better distance for this problem. There is no separation in either L² or OT distances whereas the TL² distance completely separates the data. It should not therefore be surprising that the 1NN classifier with the TL² distances outperforms the others. In fact, using 5 fold cross validation (CV) we get 100% accuracy with the TL² distance, compared to 72% in the L² distance and 86% in the OT distance. In addition we see that the stress S_p is much smaller and converges quickly to zero for the TL² distance which indicates that the TL² distance is, in this problem, more amenable to a low dimensional representation than either OT or L² distances.

4.3 Classification with Real World Data Sets

Objective

We evaluate $T{L}_{\lambda }^2$ as a distance to classify real world data sets where spatial and intensity information is expected to be important and compare with popular alternative distances. We choose one dataset which is of the type multivariate time series and a second data set consisting of images.

Data Sets

We use two data sets. The first is the AUS- LAN [30, 41] data set which contains 95 classes (corresponding to different words) from a native AUSLAN speaker (Australian Sign Language) using 22 sensors on a CyberGlove (recording position of x, y, z axis, roll, yaw, pitch for left and right hand). Therefore signals are considered as functions from {t₁,t₂,.. .t_N} to ℝ²². There are 27 signals in each class which give a total of 2565 signals.

We make two pre-processing steps. The first is to truncate each signal so it is 44 frames in length. Empirically we find that the signal is constant after the 44th frame and therefore there is no loss of information in truncating the signal. The second pre-processing step is to normalise each channel independently. This is because some channels are orders of magnitude greater than others and would otherwise dominate each choice of distance.

The second data set we use is the AT&T Database of Faces [70]. The dataset consists of ten greyscale facial images from forty subjects, see Figure 6b for examples. There were 400 images in total. In order to reduce the computation time we reduced the size of the images from 92 × 112 pixels to 50 × 50 pixels.

Figure 6.

Example signals and results for the data sets described in Section 4.3.

Methods

For the multivariate time series we compare the performance of a 1NN classifier using the L² and $T{L}_{\lambda }^2$ distances as well as the state-of-the-art method dynamic time warping [22] and the OT distance average over each channel:

$$d_{\text{MOT}}\left(f,g\right)=\frac{1}{22}\sum _{i=1}^{22}{d}_{\text{OT}}\left(\widehat{f},{\widehat{g}}_i\right)$$

where f̂_i is the i^thchannel of f after normalisation that is given by ${\widehat{f}}_i=\frac{f_i+c}{\int \left(f_i+c\right)}$ and where c is chosen so that each signal is non-negative. We use the OT distance with cost c(x, y) = |x – y|².

There are three common variations of dynamic time warping. One can apply dynamic time warping directly to the signals f and g (denoted by DTW), to the derivative f′ of the signals (denoted by DDTW) and to a weighted average of DTW and DDTW (denoted by WDTW). We define

$$\begin{array}{l}{d}_{\mathit{\text{DDTW}}}\left(f,g\right)=d_{\mathit{\text{DTW}}}\left(f^{\prime },g^{\prime }\right)\\ d_{\mathit{\text{WDTW}}}\left(f,g\right)=\alpha d_{\mathit{\text{DTW}}}\left(f,g\right)+\left(1-\alpha \right)d_{\mathit{\text{DDTW}}}\left(f,g\right).\end{array}$$

The parameter a is chosen by 5-fold 2nd depth cross validation. To be more precise the training data set is split into five partitions. One forms the testing data set (accounting for 20% of the data) and the other four form the training data set. To choose α we further divide the training data set into five partitions (each accounting for 16% of the data set). For each $\alpha =\frac{i}{100}$, where i = 0,1, …, 100, we compute how accurately one partition of the training set is classified using the remaining four parts. We then choose the value of α which produces the best classification accuracy on the training data. This value of α is then used to classify the testing data set.

The analogous distances for L², $T{L}_{\lambda }^2$ and multi-channelled OT are defined by

$$\begin{array}{l}{d}_{D{L}^2}\left(f,g\right)=d_{L^2}\left(f^{\prime },g^{\prime }\right)\\ d_{DT{L}_{\lambda }^2}\left(f,g\right)=d_{T{L}_{\lambda }^2}\left(f^{\prime },g^{\prime }\right)\\ d_{D{L}^2}\left(f,g\right)=d_{L^2}\left(f^{\prime },g^{\prime }\right)\\ d_{\text{DMOT}}\left(f,g\right)=d_{\text{MOT}}\left(f^{\prime },g^{\prime }\right)\\ d_{W{L}^2}\left(f,g\right)=\alpha d_{L^2}\left(f,g\right)+\left(1-\alpha \right)d_{D{L}^2}\left(f,g\right)\\ d_{WT{L}_{\lambda }^2}\left(f,g\right)=\alpha d_{T{L}_{\lambda }^2}\left(f,g\right)+\left(1-\alpha \right)d_{DT{L}_{\lambda }^2}\left(f,g\right)\\ d_{\text{WMOT}}\left(f,g\right)=\alpha d_{\text{MOT}}\left(f,g\right)+\left(1-\alpha \right)d_{D\text{MOT}}\left(f,g\right).\end{array}$$

We do not have to choose the same value of λ in the $T{L}_{\lambda }^2$ and $DT{L}_{\lambda }^2$ distances however considering that signals are normalised, we will use the same value. Note that DL², DTW, DDTW, WDTW $DT{L}_{\lambda }^2$ and DMOT are not metrics.

We remark that an alternative method for including derivatives in the $T{L}_{\lambda }^p$ distance would be to extend the signal to include the derivative. We briefly assume that f is defined over a continuous domain. Let f : ℝ →ℝ, and $\stackrel{\sim }{f}=\left(f,\frac{\mathrm{d}f}{\mathrm{d}x}\right)$, then we define

$$d_{T{W}_{\lambda }^{1,p}}\left(f,g\right)=d_{T{L}_{\lambda }^p}\left(\stackrel{\sim }{f},\stackrel{\sim }{g}\right).$$

We take our notation $T{W}_{\lambda }^{k,p}$ from the Sobolev space notation where W^k,p is the Sobolev space with k weak derivatives integrable in L^p. There is no reason to limit this to one derivative, and we may define $\stackrel{\sim }{f}=\left(f,\frac{\mathrm{d}f}{\mathrm{d}x},\dots ,\frac{{\mathrm{d}}^{k}f}{\mathrm{d}{x}^k}\right)$ and

$$d_{T{W}_{\lambda }^{k,p}}\left(f,g\right)=d_{T{L}_{\lambda }^p}\left(\stackrel{\sim }{f},\stackrel{\sim }{g}\right).$$

When the signals are discrete one should use a discrete approximation of the derivative. In order to be consistent with previous extensions of dynamic time warping we do not develop this approach here.

Dynamic time warping is only defined on time series so we are not able to apply it to the AT&T Database of Faces. We apply the optimal transport distance by normalising each image f ∈ ℝ² → {0, 1} $\widehat{f}\left(x\right)=\frac{f\left(x\right)}{{\int }_{{[0,1]}^2}f\left(y\right)\mathrm{d}y}$. There is no normalisation for either L² or TL² distances. We find the 1NN classifier using TL², L² and OT distances.

We will use λ =1 in AUSLAN and λ = 0.1 in the AT&T Database of Faces for the $T{L}_{\lambda }^2$ based distances. The underlying measure μ is chosen to be the uniform measure defined on [0,1] or [0,1]².

Results

We considered two methods for comparing the performance of each distance. The first is the 1NN classification accuracy in each distance. We use the 1NN classification accuracy as a measure as to how well each distance captures the underlying geometry. A higher accuracy implies closest neighbours are more likely to belong to the same class.

The results are given in Table 1 where we report error rates using 5-fold cross-validation. In terms of the 1NN classifier for the AUSLAN data set we see that TL² is better than L² and is a modest improvement over dynamic time warping.

Table 1.

Error rates for 1NN classification in AUSLAN. The top row corresponds to applying each method to the signal, the second row is each method applied to the derivative of the signal, and the third row is the method applied to a weighted average of the signal and the derivative of the signal.

	L2	DTW	$\mathit{\text{TL}}\frac{2}{\lambda }$	MOT
Signal	15.39%	11.45%	12.12%	61.71%
Derivative	22.15%	19.77%	12.63%	10.41%
Weighted Average	8.06%	7.33%	6.70%	10.41%

A rather surprising result is the difference between the MOT distance between the signals and the MOT distance between the derivative of signals. We believe this is most likely due to the length of the word being a good indicator of the word (this would also explain why the L² distance has reasonable performance). We can see from the example signal in Figure 6a only the first part of the signal contains information (the word being spoken), the remainder of the signal is noise. Because we need to renormalise in order to apply the MOT distance then, similar to the example in Figure 2a, the difference between the first part of the signal (containing information) and the latter part of the signal (containing noise) is reduced.

On the other hand the derivative of the signal will place a lot of mass at the end of the word, with smaller masses in other places where the signal is changing. In particular MOT is now able to identify the length of the signal, leading to a big improvement in performance. Furthermore, some channels are likely to contain more information than others. The decoupling of channels in the MOT distance could be an advantage as simultaneously matching across all channels, as in the $T{L}_{\lambda }^p$ distance, can mean the latter distance is corrupted by low quality channels. This artefact could be removed by weighting channels (this would require training the distance). Since we expect a temporally correlated distance to be a better model then, when weighting channels, we would expect to see an improved performance of the TL^p distance over the MOT distance.

Our results indicate that the $T{L}_{\lambda }^p$ distance better represents the geometry of the dataset than any of the other (psuedo) distances. However a 1NN classifier should not be expected to achieve the best classification results. We refer to [2] for a state-of-the- art neural network which produces a much better classification error than the 1NN method considered here (the smallest error rate in [2] for AUSLAN is 2.53%, this uses a training data set equal to $\frac{4}{9}\text{th}$ of the total data set). We stress that the aim of this paper is to introduce a distance that better models signals where both spatial and intensity is important, not to define a new classification method.

For the AT&T Database of Faces the 1NN classifier using the OT distance performs worse with an error of 3.3%. The L² distance does the second best with 2.5% and the TL² distance is the best with 2%.

In the same spirit as Section 4.1 we define the performance metric κ_ij(ρ) as the ratio of distance between class i and class j and the maximum class coverage radius of class i and class j. For the distance between classes we use the Hausdorff distance (see Section 4.1) and for the class coverage radius we use the minimum radius r such that connecting any two data points in class i closer than r defines a connected graph. We plot the results in Figures 6c and 6d. The x axis represents pairs of classes where for visual clarity we have ordered the pairs so that the κ(L²) is increasing. A large value of κ_ij indicates that it is easier to identify class i from class j whereas a small value indicates that identifying the two classes is a difficult problem.

For AUSLAN we see that the TL² distance has, for the majority of pairs of classes, a larger value of κ_ij than the L² distance and DTW and therefore better represents the class structure. The MOT distance does poorly, except in a few cases. We notice that all distances follow the trend that class separation is increasing with κ(L²).

For AT&T Database of Faces the L² and TL² distances perform very similarly. However both the TL² and L² distances are much more consistent than the OT distance, we can see that although between some classes the OT distance achieves the best results, with other classes the OT distance does extremely poorly (there are many more classes with a class separation close to 1).

4.4 Histogram Specification and Colour Transfer with the TL^p Distance

Histogram specification and colour transfer

Histogram specification concerns the problem of matching one histogram onto another. For a function f on a discrete domain X the histogram is given by f_#μ where μ is the uniform discrete measure supported on N points. We do not make any assumption on the dimension of the codomain of f (so that f may be multivalued and the histogram may be multidimensional). This coincides with the definition given in the introduction to the section, that is

$$f_{\#}\mu \left(y\right)=\frac{1}{N}\#\left\{x\in X:f\left(x\right)=y\right\}.$$

Given two functions f : X → ℝ^m and g : Y → ℝ^m, with histograms ϕ and ψ respectively, histogram specification is the problem of finding a map T : X → Y such that ψ = T_# ϕ.

The colour transfer problem is the problem of colouring one image f with the palette of an exemplar image g. A common method used to solve this problem is to use histogram specification where T is the minimizer to Monge's optimal transport problem (2) between ϕ and ψ [14, 17, 49, 62, 63]. Let our colour space be denoted by 𝒞 where for example if the colour space is 8 bit RGB then 𝒞 = {0,1,…, 255}³. The colour histogram then defines a measure over 𝒞. If we consider two such histograms ϕ and ψ corresponding to images f : X → 𝒞 and g : Y → 𝒞 respectively then a histogram specification is a map T : 𝒞 → 𝒞 that satisfies ψ = T_#ϕ. The recoloured image fˆ = g∘T has the same colour histogram as g. The solution fˆ is a recolouring of f using the palette of g.

If we consider grayscale images then 𝒞 = [0,1] and the optimal transport map (assuming it exists) is a monotonically increasing function. In particular this implies that if pixel x is lighter than pixel y (i.e. f (x) > f (y)) then in the recoloured image f ∘ = Tº f pixel x is still lighter than pixel y. In this sense the OT solution preserves intensity ordering. But note that no spatial information is used to define T; only the difference in intensity between pixels is used and not the distance between pixels.

Spatially correlated histogram specification

Let ϕ and ψ be the histograms corresponding to images f : X → ℝ^m and g : Y → ℝ^m respectively. If we recall Proposition 3.3 then ${lim}_{\lambda \to \infty }{d}_{T{L}_{\lambda }^p}\left(\left(f,\mu \right),\left(g,\nu \right)\right)=d_{\text{OT}}\left(f_{\#}\mu ,g_{\#}\nu \right)$ (where μ and ν are the discrete uniform measures over the sets X and Y). For λ < ∞ the $T{L}_{\lambda }^p$ distance includes spatial and intensity information. Hence the $T{L}_{\lambda }^p$ distance provides a generalization of OT induced histogram specification.

Analogously to the OT induced histogram specification method we define the spatially correlated histogram specification to be histogram specification using the map T : X → Y which is a minimizer to Monge's formulation of the $T{L}_{\lambda }^p$ distance (6). When the images are of the same size then, by Proposition 3.5 such a map exists. The recoloured image f̂ of f is given by f̂ = g∘T. Furthermore when the images are of the same size the map T is a rearrangement of the pixels in X and therefore the histograms are invariant under T. In particular the histogram of f̂ is the same as the histogram of g.

Although we propose the spatially correlated histogram specification as a method to incorporate spatial structure we recall from the discussion at the start of Section 4 the value of the method as a numerically efficient approximation to OT induced histogram specification for colour images that are not too large. Motivated by Proposition 3.3 one expects that for large λ the $T{L}_{\lambda }^p$ map is approximately the OT map between colour histograms. The OT problem is in the C space which, for colour images is 3 dimensional. However, the $T{L}_{\lambda }^p$ problem is in the domain of the images Ω, which is typically 2 dimensional. Hence one can use the $T{L}_{\lambda }^p$ distance to approximate OT induced histogram specification in a lower dimensional space when $O\left(n_c^3\right)=\left|\mathcal{C}\right|>\left|\mathrm{\Omega }\right|=O\left(n_s^2\right)$ where n_c is the size of discretisation in each colour channel and n_s is the size discretisation is each spatial dimension.

We briefly remark that histogram specification methods often include additional regularization terms. Such choices of regularization on the transport map include penalizing the gradients [17, 62, 63], sparsity [63], average transport [56] and rigidity [28]. One could apply any of the above regularizations to spatially correlated histogram specification.

Examples

First, let us consider the 128 × 128 grayscale images in Figure 7. The objective is to combine the shading of the first image with the geometry of the second image. We are motivated by the scenario where one wishes to combine information about a scene obtained by two different measurements: one where intensities (dynamical range) are well resolved, but the spatial resolution (geometry) is not well captured, and another where dynamical range is poorly captured, but the geometry is well resolved. We furthermore allow that the scenes captured may be somewhat different. The desire is to combine the images to obtain a single image with both good geometry and intensity. The solution we propose is to use spatially correlated histogram specification to re-shade the image with low quality intensity.

Figure 7.

Spatially correlated histogram specification of synthetic grayscale images. An OT induced solution does not exist since there are no transport maps between the image to be shaded and the exemplar image.

The result, as given in Figure 7, produces what we consider to be the desired output. The shading has been transferred and the geometry has not been lost. One is not able to apply histogram specification (induced by the OT map) due to the lack of existence of an optimal transport map from the histogram of the original image ϕ to the histogram exemplar image ψ. This is due to the histogram of the original image being a sum of two delta masses as in Figure 7d.

As a more challenging example we consider real world colour images. Images are 128 × 128. We compare our method with histogram specification using the OT mappings and the following state of the art methods for which code is freely available. Reinhard, Ashikhmin, Gooch and Shirley's renormalisation method (RAGS) [66] rescales the image so that the mean and standard deviation of the LAB channels match the exemplar image. Pitié and Kokaram (PK) [61] approximate colour histograms with a Gaussian and look for the best linear map between the two colour histograms. Essentially they look for couplings, as in the Monge formulation of optimal transport but they restrict to linear mappings. In general there may not be any linear mappings between two histograms, however it can be shown that when the histograms are Gaussians the set of mappings is not empty. The final method we compare to is the regularized transportation method due to Ferradans, Papadakis, Rabin, Peyré and Aujol (FPRPA) [17].

It is difficult to quantitatively assess the performance of each method objectively. Whether the output is satisfactory depends on content and artistic preferences. We refer to [86] for an objective quantitative measure of colour transfer results, however this explicitly marks against introducing colour artefacts. Whilst some colour artefacts are clearly undesirable introducing others, such as the northern lights in Figure 8 was the objective. Hence we no of no way to quantify the performance of our method and instead rely on qualitative assessments.

Figure 8.

Spatially correlated histogram specification of real colour images between palm trees and the northern lights.

In the first pair of images the exemplar image contains a few trees with the northern lights in the background, whilst the other image has a few trees with a mostly clear sky in the background. The challenge is to recreate the northern lights in the second image.

As one would expect, in Figure 8c we see that the histogram specification induced by OT loses the spatial structure. Indeed, it is hard to recognise the northern lights. Similar with each competing method in Figure 8g-8i, none of them successfully manage to reproduce the northern lights and the palm trees all pick up an unnatural reddish shade. The spatially correlated histogram specification solution does a much better job at preserving the ordering locally. As λ increases it becomes cheaper to match pixels that are further apart and therefore, for large λ, the matching does not preserve the local structure in the exemplar image.

In the second real world example we consider colour transfer between two images from Masson's trichrome staining procedure shown in Figures 9a and 9b. We manipulate the luminosity of the second images. The objective is to colour the second image using the palette of the first. In Figure 9 we compare the spatially correlated histogram specification method of $T{L}_{\lambda }^p$ to the other methods.

Figure 9.

Spatially correlated histogram specification of real colour images between two images of Masson's trichrome staining procedure, one of which the luminosity has been manipulated.

Since we know the true image we may compare the colourisation with the true image. We report the L² error computed by

$$\text{err}\left(f-g\right)=\frac{1}{N}\sqrt{\sum _{i=1}^3{\sum _j\left|f_i\left(x_j\right)-g_i\left(x_j\right)\right|}^2}$$

where f = (f₁, f₂, f₃) and g = (g₁,g₂,g₃) are images in RGB space and N = 128² is the number of pixels. The TL^p method (error 0.2885) gives a more accurate colourisation compared to the RAGS method (error 0.4040), the PK method (error 0.3568), the FPRPA method (error 0.4817), and the OT induced histogram specification method (error 0.4030). One can also see that the $T{L}_1^2$ based method does not have the same artefacts as the other methods. In particular, (a) the darker band is still evident in RAGS and PK, (b) FPRPA fails to accurately recolour the white band on the right hand side, and (c) OT places too much white on the left hand side and not enough on the right hand side.

5 Conclusions

In this paper we have developed and applied a distance that directly accounts for the intensity of the signal within a Lagrangian framework. This differs from OT distances that do not directly measure intensity and the L^p distance which measures intensity only. Through applications we have shown the potential of this distance in signal analysis.

The distance is widely applicable, unlike in classical OT distances, such as the Wasserstein distance or the earth mover distance, the $T{L}_{\lambda }^p$ distance does not require treating signals as measures. Treating a signal as a measure implies the following constraints: non-negativity, normalised mass and single channelled. None of these assumptions are necessary for the $T{L}_{\lambda }^p$ distance. Furthermore the distance is applicable to both discrete and continuous signals as well as allowing practitioners to emphasise features which in many cases should allow for a better representation of data sets, for example one could include derivatives.

Efficient existing methods, such as entropy regularized or multi-scale linear programming, for optimal transport are applicable to the $T{L}_{\lambda }^p$ distance. In fact any numerical method for optimal transport that can cope with arbitrary cost functions is immediately available. This includes the entropy regularised approach of Cuturi [16]. However, there are more efficient methods that are specific to the OT distance with quadratic cost that are unavailable here, e.g. [74].

Via the representation as an OT distance between measures supported on graphs we expect many other results for OT distances to carry through to $T{L}_{\lambda }^p$ distances. For example, one could extend the LOT method [85] for signal representation and analysis to the $T{L}_{\lambda }^p$ distance. This would allow pairwise distances of a data set to be computed with numerical cost that is linear in the number of images. We leave the development for future work.

We considered a few examples where we expect (and then showed) that $T{L}_{\lambda }^p$ will outperform OT and other distances. We expect the $T{L}_{\lambda }^p$ distance to give a better performance than OT distances when intensity information is important. On the other hand, we do not expect the $T{L}_{\lambda }^p$ distance to be robust to high frequency noise. In this case an OT distance would probably have superior performance.

The applications we considered were to classification and histogram specification in the context of colour transfer. For classification we chose data sets with a Lagrangian nature but were either multi-channelled or non-positive (so that in both cases one must apply ad-hoc methods in order to apply the OT distance). We showed the $T{L}_{\lambda }^p$ distance better represented the underlying geometry. The 1NN classifier is a very simple method and we expect our results here could be significantly improved by, for example, replacing the L² distance in the MDS projection approach of Weinberger and Chapelle [87] with the $T{L}_{\lambda }^p$ distance. For the colour transfer problem we defined a spatially correlated histogram specification method which produced more visually appealing results when combining the colour of one image with the geometry of another.

Although the main motivation was to develop a distance which better represents Lagrangian data sets we also note that the $T{L}_{\lambda }^p$ distance provides a numerically efficient (for images that are not too large) approximation for the OT induced histogram specification method by, for 2-dimensional images colour images, reducing the effective dimension of the problem from three for OT distances to two for the $T{L}_{\lambda }^p$ distance. We also observe that the effective dimension of multi-channelled time signals is one. In particular the effective dimension is independent of the number of channels.

The applications we have considered are for demonstration on the performance of the $T{L}_{\lambda }^p$ distance. A next step would be to consider a more detailed study of a specific problem. For example in the colour transfer application we could have considered regularization terms/constraints which would have improved the performance, e.g. [17, 28, 51, 56, 62, 63]. It was not the aim to propose a state-of-the-art method for each application, indeed each application would constitute a paper within its own right.

A Performance of $T{L}_{\lambda }^p$ in Classification Problems with Simple and Oscillatory Signals

We compare the performance of ${\mathit{\text{TL}}}_{\lambda }^2$, L² and OT distances with respect to classification/clustering for the three classes {𝒞_j}_i=1,2,3 of signals defined in Figure 4. We test how each distance performs by finding the smallest number of data points such that the classes ${\mathcal{C}}_i^N={\left\{f_i\right\}}_{i=1}^N\subset {\mathcal{C}}_i$ are separable. For sufficiently large N the approximation $d_{H,\rho }\left({\mathcal{C}}_i^N,{\mathcal{C}}_j^N\right)\approx d_{H,\rho }\left({\mathcal{C}}_i,{\mathcal{C}}_j\right)$ is used to simplify the computation. Similarly, as a proxy for $\mathbb{E}R\rho \left({\mathcal{C}}_i^N\right)$ we use $R\rho \left({\widehat{\mathcal{C}}}^N\right)$ where

$${\widehat{\mathcal{C}}}_i^N=\left\{f_{\ell }:\ell ={\ell }_{min}^i+\frac{n-1}{N-1}\left({\ell }_{min}^i-{\ell }_{max}^i\right),n\in \left\{1,2,\dots ,N\right\}\right\}$$

is the uniform sample from class 𝒞_i (recall that class 𝒞_i is parameterized by $\ell \in \left[{\ell }_{min}^i,{\ell }_{max}^i\right]$ and with an abuse of notation we use the subscript of f_ℓ to denote the dependence of ℓ).

It follows that the class separation distances and class coverage radius are approximated by

$$\begin{array}{l}\begin{array}{cc}{d}_{H,L^2}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_2^N\right)\approx \frac{\alpha }{2}& R_{L^2}^2\left({\mathcal{C}}_1^N\right)\approx \frac{2}{N}\end{array}\\ \begin{array}{cc}{d}_{H,L^2}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_3^N\right)\approx \frac{3\alpha }{4}& R_{L^2}^2\left({\mathcal{C}}_2^N\right)\approx \frac{1}{N}\end{array}\\ \begin{array}{cc}{d}_{H,L^2}^2\left({\mathcal{C}}_2^N,{\mathcal{C}}_3^N\right)\approx \frac{\alpha }{4}& R_{L^2}^2\left({\mathcal{C}}_3^N\right)\approx \frac{2\alpha }{N\gamma }\end{array}\\ \begin{array}{cc}{d}_{H,\text{OT}}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_2^N\right)\approx \frac{{\beta }^2\alpha }{4}& R_{\text{OT}}^2\left({\mathcal{C}}_1^N\right)\approx \frac{\alpha }{N^2}\end{array}\\ \begin{array}{cc}{d}_{H,\text{OT}}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_2^N\right)\approx \frac{{\beta }^2\alpha }{4}& R_{\text{OT}}^2\left({\mathcal{C}}_1^N\right)\approx \frac{\alpha }{N^2}\end{array}\\ \begin{array}{cc}{d}_{H,\text{OT}}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_3^N\right)\approx \frac{{\beta }^2\alpha }{4}& R_{\text{OT}}^2\left({\mathcal{C}}_2^N\right)\approx \frac{\alpha }{N^2}\end{array}\\ \begin{array}{cc}{d}_{H,\text{OT}}^2\left({\mathcal{C}}_2^N,{\mathcal{C}}_3^N\right)\approx \frac{\alpha {\gamma }^2}{8}& R_{\text{OT}}^2\left({\mathcal{C}}_3^N\right)\approx \frac{\alpha }{N^2}\end{array}\\ \begin{array}{cc}{d}_{H,T{L}_{\lambda }^2}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_2^N\right)\approx \frac{\alpha }{2}& R_{T{L}_{\lambda }^2}^2\left({\mathcal{C}}_1^N\right)\approx \frac{{\alpha }^2}{N}\end{array}\\ \begin{array}{cc}{d}_{H,T{L}_{\lambda }^2}^2\left({\mathcal{C}}_1^N,{\mathcal{C}}_3^N\right)\approx \frac{3\alpha }{4}& R_{T{L}_{\lambda }^2}^2\left({\mathcal{C}}_2^N\right)\approx \frac{4{\alpha }^2}{N}\end{array}\\ \begin{array}{cc}{d}_{H,T{L}_{\lambda }^2}^2\left({\mathcal{C}}_2^N,{\mathcal{C}}_3^N\right)\approx \frac{\alpha }{4}& R_{T{L}_{\lambda }^2}^2\left({\mathcal{C}}_3^N\right)\approx \frac{{\alpha }^2}{N}\end{array}.\end{array}$$

We have

$$\begin{array}{l}{\kappa }_{12}^2\left(L^2;N\right)\approx \frac{\alpha N}{4},{\kappa }_{13}^2\left(L^2;N\right)\approx \frac{3\gamma N}{8},\\ {\kappa }_{12}^2\left(\text{OT};N\right)\approx \frac{{\beta }^{2}N}{4},{\kappa }_{13}^2\left(\text{OT};N\right)\approx \frac{{\beta }^2{N}^2}{4},\\ {\kappa }_{12}^2\left(T{L}_{\lambda }^2;N\right)\approx \frac{N}{8\alpha },{\kappa }_{13}^2\left(T{L}_{\lambda }^2;N\right)\approx \frac{3N}{4\alpha },\\ {\kappa }_{23}^2\left(L^2;N\right)\approx \frac{\gamma N}{8},\\ {\kappa }_{23}^2\left(\text{OT};N\right)\approx \frac{{\gamma }^2{N}^2}{8},\\ {\kappa }_{23}^2\left(T{L}_{\lambda }^2;N\right)\approx \frac{N}{16\alpha }.\end{array}$$

Finally we can compute N*,

$$\begin{array}{l}{N}_{12}^{\ast }\left(L^2\right)\approx \frac{4}{\alpha },N_{13}^{\ast }\left(L^2\right)\approx \frac{8}{3\gamma },N_{23}^{\ast }\left(L^2\right)\approx \frac{8}{\gamma }\\ N_{12}^{\ast }\left(\text{OT}\right)\approx \frac{2}{\beta },N_{13}^{\ast }\left(\text{OT}\right)\approx \frac{2}{\beta },N_{23}^{\ast }\left(\text{OT}\right)\approx \frac{\sqrt{8}}{\gamma }\\ N_{12}^{\ast }\left(T{L}^2\right)\approx \frac{\alpha }{8},N_{13}^{\ast }\left(T{L}^2\right)\approx \frac{4\alpha }{3},N_{23}^{\ast }\left(T{L}^2\right)\approx 16\alpha \end{array}$$

which for $\beta >\frac{\alpha }{2},\beta >\frac{3\gamma }{4}$ and $\gamma <\frac{\sqrt{2\alpha }}{8}$ implies the ordering given Section 4.1.

B Numerical Methods

In principle any numerical method for computing OT distances capable of dealing with an arbitrary cost function can be adapted to compute $T{L}_{\lambda }^p$ the distance. Here we describe two numerical methods we used in Section 4.

B.1 Iterative Linear Programming

Here we describe the iterative linear programming method of Oberman and Ruan [53] which we abbreviate OR. Although this method is not guaranteed to find the minimum in (3) we find it works well in practice and is easier to implement than, for example, methods due to Schmitzer [71] that provably minimize (3) but require a more advanced refinement procedure. See also [46] and references therein for a multiscale descent approach.

The linear programming problem restricted to a subset $M\subseteq {\mathrm{\Omega }}_h^2$ is

$$\begin{array}{cc}\begin{array}{l}\text{minimize}:\sum _{\left(i,j\right)\in M}{c}_{\lambda }\left(x_i,x_j;f_h,g_h\right){\pi }_{ij}\text{over}\pi \\ \text{subject to}\sum _{i:\left(i,j\right)\in M}{\pi }_{ij}=q_j,\sum _{j:\left(i,j\right)\in M}{\pi }_{ij}=p_i,\end{array}& \left({\text{LP}}_h\right)\end{array}$$

where c_λ is given by (4). When $\mathcal{M}={\mathrm{\Omega }}_h^2$ then the $T{L}_{\lambda }^p$ distance between (f_h, μ_h) and (g_h, ν_h) is the minimum to the above linear programme. Furthermore if π_h is the minimizer in the $T{L}_{\lambda }^p$ distance then it is also the solution to the linear programme in (LP_h) for any ℳ containing the support of π_h. That is if one already knows (or can reasonably estimate) the set of nodes ℳ for which the optimal plan is non-zero then one need only consider the linear programme on ℳ. This is advantageous when ℳ is a much smaller set. Motivated by Proposition 3.5 we expect to be able to write the optimal plan as a map. This implies whilst π_h has n² unknowns we only expect n of them to be non-zero.

The method proposed by OR is given in Algorithm 1. An initial discretisation scale h₀ is given and an estimate π_h0 found for the linear programme (LP_h) with $\mathcal{M}={\mathrm{\Omega }}_{h0}^2$. One then iteratively finds ${\mathcal{M}}_r\subseteq {\mathrm{\Omega }}_{hr}^2$, where $h_r=\frac{h_{r-1}}{2}$, to be the set of nodes defined by the following refinement procedure. Find the set of nodes for which π_hr−1 is non-zero, add the neighbouring nodes and then project onto the refined grid ${\mathrm{\Omega }}_{hr}^2$. The optimal plan π_hr on ${\mathrm{\Omega }}_{hr}^2$ is then estimated by solving the linear programme (LP_h) with ℳ = ℳ_r.

The grid Ω_hr will scale as ${\left(2^{rd}{h}_0^{-1}\right)}^2$. If the linear programme is run N times then at the r^th step the linear programme has on the order of $2^{rd}{h}_0^{-1}$ variables. In particular on the last (and most expensive) step the number of variables is $O\left(2^{Nd}{h}_0^{-1}\right)$. This compares to size ${\left(2^{Nd}{h}_0^{-1}\right)}^2$ if the linear programme was run on the final grid without this refinement procedure.

Algorithm 1 An Iterative Linear Programming Approach [53]
Input: functions f,g ∈ Lp(Ω), measures μ,ν ∈ 𝒫(Ω) and parameters h0, N.
1:	Set r = 0.
2:	repeat
3:	Define ${\mathcal{S}}_r={\mathrm{\Omega }}_{hr}^2$ where Ωhr is the square grid lattice with distances between neighbouring points hr and discretise functions f, g and measures μ,ν on Ωh.
4:	if r = 0 then
5:	Solve (LPh) on So and call the output πh0.
6:	else
7:	Find the set of nodes on 𝒮r−1 for which πhr−1 is non-zero and call the set 𝒦r−1.
8:	To 𝒦r−1 add all neighbouring nodes and call this set 𝒩r−1.
9:	Define ℳr to be the set of nodes on 𝒮r that are children of nodes in 𝒩r−1.
10:	Solve (LPh) restricted to ℳr and call the optimal plan πhr.
11:	end if
12:	Set $h_{r+1}=\frac{h_r}{2}$ and r ↦ r +1.
13:	until r = N
Output: The optimal plan πhN−1 for (LPh).

B.2 Entropic Regularisation

Cuturi, in the context of computing OT distances, proposed regularizing the minimization in (3) with entropy [16]. This was further developed by Benamou, Carlier, Cuturi, Nenna and Peyré [5], abbreviated to BCCNP, which is the method we describe here. Instead of considering the distance $T{L}_{\lambda }^p$ we consider

$$S_{\varepsilon }=\underset{\pi \in \Pi \left(\mu ,\nu \right)}{inf}\left\{\sum _{i=1}^n\sum _{j=1}^n{c}_{\lambda }\left(x_i,x_j;f,g\right){\pi }_{ij}-\varepsilon H\left(\pi \right)\right\}$$

where $H\left(\pi \right)=-{\sum }_{i=1}^n{\sum }_{j=1}^n{\pi }_{ij}log{\pi }_{ij}$ is the entropy. In the OT case the distance S_ε is also known as the Sinkhorn distance. It is a short calculation to show

$$S_{\varepsilon }=\varepsilon \underset{\pi \in \Pi \left(\mu ,\nu \right)}{inf}\left\{\text{KL}\left(\pi |\kappa \right)\right\}$$

where ${\mathcal{K}}_{ij}=exp\left(-\frac{c_{\lambda }\left(x_i,x_j:f,g\right)}{\varepsilon }\right)$ (the exponential is taken pointwise) and KL is the Kullback-Leibler divergence defined by

$$\text{KL}\left(\pi |\mathcal{K}\right)=\sum _{i=1}^n\sum _{j=1}^n{\pi }_{ij}log\left(\frac{{\pi }_{ij}}{{\mathcal{K}}_{ij}}\right)$$

It can be shown that the optimal choice of π for S_ε can be written in the form π* = diag(u) 𝒦diag(v) where u, v ∈ ℝⁿ are limits, as r →∞, of the sequence

$${\upsilon }^{\left(0\right)}=\mathbb{I},u^{\left(r\right)}=\frac{\underset{\_}{p}}{\mathcal{K}{\upsilon }^{(r)}},{\upsilon }^{(r+1)}=\frac{\underset{\_}{q}}{{\mathcal{K}}^{⊺}{u}^{(r)}}$$

and p =(p₁,…p_n),q= (q₁,…q_n) (multiplication is the usual matrix-vector multiplication, division is pointwise and Τ denotes the matrix transpose). The algorithm given in 2 is a special case of iterative Bregman projections and also known as the Sinkhorn algorithm.

The stopping condition proposed in [16] is to let π^(r) = diag(u^(r)) 𝒦 diag(v^(r)) then stop when

$$\left|\frac{{\sum }_{i,j=1}^n{\mathcal{K}}_{ij}{\pi }_{ij}^{\left(r\right)}-\varepsilon H\left({\pi }^{(r)}\right)}{{\sum }_{i,j=1}^n{\mathcal{K}}_{ij}{\pi }_{ij}^{(r-1)}-\varepsilon H\left({\pi }^{(r-1)}\right)}-1\right|<{10}^{-4}.$$

Note that although as ε →0 we will recover the unregularised $T{L}_{\lambda }^p$ distance we also suffer numerical instability as 𝒦 → 0 exponentially in ε. These instabilities have been addressed in, for example, [14, 72].

For optimal transport with quadratic cost c(x, y) = |x − y|² the Sinkhorn algorithm can be more efficiently implemented using Gaussian convolutions [74]. The two numerical methods described so far use the formulation of $T{L}_{\lambda }^2$ given by (3-4) which interprets the $T{L}_{\lambda }^2$ as an OT distance between measures μ and ν for a (non-quadratic) cost function c_λ(·,·; f, g), hence one cannot make use of previous OT methods such as [74].

However, we also recall that we can define the $T{L}_{\lambda }^2$ distance as the optimal transport distance between measures (f × Id)_#μ and (g × Id)_#ν, see (5), in which case the entropy regularized approach can be implemented using Gaussian convolutions in dimension d + m (when p = 2), where f :Ω⊆ ∝^d → ∝^m. Although this means that the numerical method is based in a higher dimension we note the success of the bilateral grid method for bilateral filters that are also based on computing a Gaussian filter in a higher dimension [10, 57]. For colour images, where m = 3 this approach may not be efficient however for m =1 these ideas have the potential for an improved algorithm.

Algorithm 2 An Entropy Regularised Approach [5, 16]
Input: discrete functions f = (f1,…,fn), g = (g1, …, gn), discrete measures $\mu ={\sum }_{i=1}^n{p}_i\delta x_i,\nu ={\sum }_{j=1}^n{q}_j\delta x_j$, the parameter ε and a stopping condition.
1:	Set r = 0, $\mathcal{K}={\left(exp\left(-\frac{c\left(x_i,x_j;f,g\right)}{\varepsilon }\right)\right)}_{ij}$ and u(0) = 𝕀 ∈ ℝn.
2:	repeat
3:	Let r↦r+ 1, ${\upsilon }^{\left(r\right)}=\frac{\underset{\_}{q}}{{\mathcal{K}}^{⊺}{u}^{(r-1)}}\text{and }{u}^{(r)}=\frac{\underset{\_}{p}}{\mathcal{K}{u}^{(r)}}$ where p =(p1,…pn),q= (q1,…qn)
4:	until Stopping condition has been reached
5:	Set π = diag(u(r))𝒦diag(v (r)).
Output: An estimate π on the optimal plan for Sε where the accuracy is determined by the stopping condition.