Transformations Based on Continuous Piecewise-Affine Velocity Fields

Abstract

We propose novel finite-dimensional spaces of well-behaved ℝⁿ → ℝⁿ transformations. The latter are obtained by (fast and highly-accurate) integration of continuous piecewise-affine velocity fields. The proposed method is simple yet highly expressive, effortlessly handles optional constraints (e.g., volume preservation and/or boundary conditions), and supports convenient modeling choices such as smoothing priors and coarse-to-fine analysis. Importantly, the proposed approach, partly due to its rapid likelihood evaluations and partly due to its other properties, facilitates tractable inference over rich transformation spaces, including using Markov-Chain Monte-Carlo methods. Its applications include, but are not limited to: monotonic regression (more generally, optimization over monotonic functions); modeling cumulative distribution functions or histograms; time-warping; image warping; image registration; real-time diffeomorphic image editing; data augmentation for image classifiers. Our GPU-based code is publicly available.

1 Introduction

Diffeomorphisms are important in many fields such as computer vision, medical imaging, graphics, and robotics. Unfortunately, current representations of highly-expressive diffeomorphism spaces are overly complicated. Thus, despite their potential and mathematical beauty, their applicability is limited, especially in large datasets or when computing time is restricted. Moreover, in such spaces, owing to their complexity, using powerful inference tools, e.g., Monte Carlo Markov Chain (MCMC), still presents challenges, although some encouraging recent progress has been made in this active field of research (e.g., [1], [2], [3]). Lastly, seemingly-formidable mathematical preliminaries render these spaces accessible to only a small group of geometry experts. This hinders the exchange of ideas between communities and unnecessarily limits the potential impact of diffeomorphism-based methods; e.g., while certain machine-learning areas can benefit from such methods, little work has been done in this direction, partly since practical computational tools have yet to become available.

Motivated by practicalities of probabilistic modeling and statistical inference as well as a desire to make diffeomorphisms broadly accessible, in this work, which expands [4], we propose a representation that (as we will show in Section 4) combines simplicity, expressiveness, and efficiency. Particularly, we propose new spaces of transformations that are based on (fast, highly-accurate) integration of Continuous Piecewise-Affine (CPA) velocity fields. Importantly, as we will show, their benefits go beyond speed and accuracy.

Possible applications of the proposed representation are numerous, as we demonstrate here with: image editing and shape manipulation; unconstrained optimization over monotonic functions; modeling of Cumulative Distribution Functions (CDFs) and histograms with order-preserving geometry; time warping; image registration; landmark-based image warping/animation; “prettifying” results of (non-diffeomorphic) dense-correspondence tools. Moreover, we recently used the proposed representation as a key component in a learned data-augmentation scheme [5], improving the results of image classifiers. Finally, our code is available at https://github.com/freifeld/cpabDiffeo.

2 Related Work

Pattern Theory and Differential Geometry

Representing objects via transformations acting on them is a Pattern-Theoretic cornerstone [6]. Our work is influenced by many impressive works in this field, primarily in the geometry-oriented areas of computer vision and medical imaging. Due to space limits, we can mention only a few: [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19]. Most of these works are based on complicated, usually ∞-dimensional, spaces, and the associated representations and computations are, in practice, discretized and/or otherwise approximated. We take a more practical approach and start from a finite-dimensional space, in which discretizing the representation is unneeded, while computations require no approximations in the 1D case and almost no approximations in higher dimensions. The result is a simple, efficient, and practical machinery for working with a rich space of diffeomorphisms. The rapidness and high accuracy of Algorithm 1 (see Section 4), together with the fact that the machinery is easily implemented on GPU, let us evaluate transformations fast. This, in turn, together with the fact that the representation effortlessly supports coarse-to-fine analysis and the use of smoothness priors, allows the use of general-purpose powerful inference methods such as MCMC. See also other recent approaches for MCMC on diffeomorphisms, e.g., [1], [2], [3].

In applications involving landmark pairs, most methods above can deal with only a small number of landmarks as their inference complexity grows super-linearly with the number of landmarks. Consequently, they cannot leverage the success of popular tools for dense-correspondence extraction (e.g., [20], [21]), highlighting a disconnect with the larger computer-vision community. Our algorithms have linear complexity and are (embarrassingly) parallelizable, yielding sub-linear running times in practice.

Note that our representation, despite the fact it involves tessellations, is not not based on control points (such as, e.g., [2], [19]); e.g., in cases where we either impose volume preservation or use type-II tessellations (see Section 4), one cannot simply define arbitrary velocities at the tessellation vertices as there are too few degrees of freedom.

PA and CPA Affine Maps in Computer Vision

Cootes et al. [22] use PA transformations. The latter, while simple and efficient, are neither continuous nor invertible. Several other works use CPA transformations (e.g.,[23]; see also [24]), sometimes with additional constrains (e.g., [25]). Common to those works is the direct use of PA (or CPA) maps as transformations. Those are not differentiable (and are often not invertible), hence not diffeomorphisms. In contrast, by integrating CPA velocity fields, we obtain transformations–which themselves are not PA (hence not CPA)–that are (orientation-preserving) diffeomorphisms. Lin et al. [26] work with CPA velocity fields but, rather than integrating them, they use them to model motion pattens.

Closer to ours is the elegant log-Euclidean polyaffine method proposed by Arsigny et al. [27] (see also [28], [29]) that, similar to ours, uses finitely-many affine building blocks to build flexible velocity fields. They use a spatial averaging with smoothly-decaying weights of the blocks to ensure a smooth velocity field whose integration yields trajectories that define a diffeomorphism. However, as their integral has no closed form they must use an approximation throughout; i.e. they must numerically approximate the entire trajectory. This is computationally demanding and can cause substantial approximation errors. Their method is based on approximating the integral of a weighted sum of affine velocity fields via a weighted sum of the affine diffeomorphisms associated with these fields. To reduce errors, they divide the field by a large 2^#steps as the approximation holds only for near-zero velocity. To keep the number of steps not too large, they smartly generalize the scaling-and-squaring method. The result is, however, still exact only if a single affine component is used. For expressiveness, however, a larger number is needed, and accuracy drops. It is thus unsurprising they focus on a small number of affine components, and that to achieve reasonable timings, they use their method only in the last stage of inference; till then they resort to a non-diffeomorphic fusion. In contrast, we use affine building blocks in a different way; i.e., we use them in a piecewise manner, but, using linear constraints, force them to yield everywhere-continuous velocity fields. This seemingly-subtle difference has profound implications. In 1D, it yields a closed-form integration; in higher dimensions it lets us integrate almost the entire trajectory in closed form using large (hence few) steps which are exact, and only in small portions of the trajectory do we resort to a numerical solver. This virtually eliminates numerical issues and allows construction of substantially more expressive transformations; e.g., Arsigny et al. [27] report using 7 affine blocks, while we routinely use tens or hundreds while neither accuracy nor computational cost becomes an issue. Though this indirect comparison of integration methods (as their method is inapplicable to our velocity fields and vice versa, a direct comparison is impossible) favors ours, it is just part of the story. Our representation has additional advantages over theirs, including simplicity, better suitability for a GPU implementation, trivial handling of boundary constraints and volume preservation, and simpler encoding of statistical priors and coarse-to-fine analysis.

Discrete Representations and Approximations

Allassonnière et al. [13] efficiently approximate diffeomorphisms. Unlike our transformations, their non-differentiable transformations are not diffeomorphisms. While we focus on a general-purpose representation, diffeomorphic demons [16] is a registration method, popular due to its speed, based on discretely-defined fields. As its authors note, these may be inconsistent with a diffeomorphic framework and may not preserve orientation. They also cannot easily impose volume preservation, though some success was reported [30]. Also, a computer representation of a discrete-field sequence needs plenty of memory. These issues can be obviated by adapting their method to use our compact and continuously-defined fields. More generally, approximations based on discretely-defined fields and/or discrete diffeomorphisms are widely used, e.g., in medical imaging [16], [31], robotics [32], [33], [34], geometric modeling [35] and fluid dynamics [36]. Unlike these works, both our fields and transformations are continuously-defined and more compact.

Statistics on Manifolds and Tangent Spaces

Like many authors (including of some aforementioned works), we handle the nonlinearity of a space via the linearity of the tangent space at the identity [11], [27], [29], [37], [38], [39], [40], [41]. Other tools (not explored here) for statistics on manifolds that either use other tangent spaces or work on the manifold itself [42], [43], [44], [45], [46], [47], [48] may also be applied to our spaces. Particularly, parallel-transport tools may be especially relevant here [49], [50], [51], [52], [53], [54], [55].

CDF/Histogram Modeling

In modeling distributions, it is better to work with cummulative distribution functions over densities since the latter might not exist and inter-density L_p (or sphere-based) distances can be arbitrarily large even if their probability measures are essentially the same. One approach to CDF representation uses p-Wasserstein spaces, usually p ∈ {1, 2}, the p = 1 case is tied to Earth Mover’s Distance [56]. A limitation of this approach is that it needs bounded pth-moments and hard computations. 1-Wasserstein methods also lack easy synthesis of new points that are valid histograms/CDFs. While it is less of an issue for spherical methods [57], they suffer from two issues. 1) They do not respect the ordering of the bins or of ℝ. While, for histograms, bin ordering is sometimes immaterial, the ordering of ℝ matters. 2) Large moves on the sphere lead to CDFs/histograms with negative values. The problems above do not exist in our representation.

Image Warping and Shape Manipulation

Related to ours are works on image warping ([58], [59], [60], [61], [62]) and shape manipulation (e.g., [63]). Unlike most methods, ours is fast, invertible and handles constraints effortlessly.

Benefits of the Proposed Representation

To summarize, existing spaces of diffeomorphisms offer only subsets of the following list: 1) high expressiveness; 2) ease of implementation; 3) modest mathematical preliminaries (basic linear algebra and ODE); 4) ease of handling optional constraints (e.g., volume preservation); 5) convenient modeling choices (coarse to fine, easy-to-use smoothness priors); 6) finite dimensionality; 7) fast and highly-accurate computations. These benefits, especially the last three, render more tractable the use of inference tools that are usually too expensive in the context of rich diffeomorphisms.

3 High-Level Summary

The section provides a summary of the proposed representation, laying the ground for the formal treatment (Section 4). Let Ω be either ℝⁿ or a certain type, to be defined later, of a proper subset of ℝⁿ (i.e., Ω⊊ℝⁿ). A popular way to obtain a diffeomorphism, T : Ω → Ω, is via the integration of velocity fields; see Fig. 1a. The choice of velocity-field family affects the dimensionality, structure, and expressiveness of the space of the resulting diffeomorphisms, as well as the accuracy and computational complexity of the integration. Thus, this choice crucially affects which probabilistic models can be used and the tractability of the statistical inference.

Fig. 1.

(a) Integration of sufficiently-nice velocity fields is widely used to generate well-behaved nonlinear transformations. The choice of using CPA velocity fields, among other benefits, reduces computational costs, increases integration accuracy, and simplifies modeling and inference. A CPAB transformation, x ↦ ϕ^θ(x, t), is one that is based (via integration) on a CPA velocity field, v^θ. (b) A 1D example. (c–d) Two 2D examples, where in (d) there are also additional constraints. Top row: a continuously-defined v^θ in select locations. Middle: Visualizing the horizontal ( $v_h^{\theta }$, left) and vertical ( $v_v^{\theta }$, right) components as heat maps highlights the CPA property; blue = −λ, green=0, and red=λ where $\mathrm{\lambda }={max}_{x\in \mathrm{\Omega }}max(|v_h^{\theta }(x)|,|v_v^{\theta }(x)|)$. Bottom: $I_{\text{src}}\circ {\varphi }^{\theta }(\cdot ,1)$.

CPA Velocity Fields

We base our representation on spaces of Ω → ℝⁿ CPA velocity fields (Fig. 1). The term ‘piecewise’ is w.r.t. a certain tessellation (Section 4.1), denoted by $\mathcal{P}$. Let ${\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}$ be such a space. While ${\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}$ depends on Ω and $\mathcal{P}$, we will usually notationally suppress these dependencies, and will just write $\mathcal{V}$. One appeal of these spaces is that they are finite-dimensional and linear (although their elements, i.e., the velocity fields, are usually nonlinear). Let $d=\dim (\mathcal{V})$. The spaces ℝ^d and $\mathcal{V}$ are identified with each other (as we will explain in Section 4.2, Eq. (11)), where every θ ∈ ℝ^d is identified with exactly one element of $\mathcal{V}$, denoted by v^θ, and vise versa. Symbolically, we write

$$\theta \leftrightarrow v^{\theta }\text{where}{v}^{\theta }\in \mathcal{V},\theta \in {\mathrm{ℝ}}^d.$$ (1)

Likewise, $\theta +\theta \prime \leftrightarrow v^{\theta }+v^{{\theta }^{\prime }}\triangleq v^{\theta +{\theta }^{\prime }}$ and αθ ↔ αv^θ ≜ v^αθ where θ; θ′ ∈ ℝ^d and α ∈ ℝ. Note that d depends on $\mathcal{P}$ (and typically grows with n). A finer $\mathcal{P}$ implies a higher d and richer velocity fields and vice versa (Figs. 4, 2, and 3).

Fig. 4.

Samples from the prior (Section 4). (a–h) The top 3 rows echo those in Fig. 1c and Fig. 1d. The 4^th row shows a deformed grid overlaid on the image. The 5^th row shows select trajectories. Note that the trajectories and transformations are differentiable (hence continuous) but not piecewise affine. The tessellation in (e–h) is a refinement of the one in (a–d). (a) & (e): T^θ ∈ M. (b) & (f): T^θ ∈ M^vp. (c) & (g): T^θ ∈ M^∂. (d) & (h): T^θ ∈ M^∂,vp. See Section 4 for details.

Fig. 2.

Several type-I tessellations of a 2D region.

Fig. 3.

Several type-II tessellations of a 2D region.

Remark 1

There are many finite-dimensional linear spaces of continuous velocity fields (e.g., [27] or other spaces based on splines). We will show that CPA spaces, however, have additional useful properties in our context.

From CPA Velocity Fields to Trajectories

Modulo a detail (to be explained in Section 4.4) related to the case Ω⊊ℝⁿ, any continuous Ω → ℝⁿ velocity field, whether Piecewise-Affine (PA) or not, defines differentiable ℝ → Ω trajectories. If x ∈ Ω then $v^{\theta }\in \mathcal{V}$ defines a trajectory, t ↦ ϕ^θ(x, t), such that ϕ^θ(x, 0) = x and ϕ^θ(x, t) solves the integral equation

$${\varphi }^{\theta }(x,t)=x+{\displaystyle {\int }_0^t{v}^{\theta }({\varphi }^{\theta }(x,\tau ))d\tau \text{where}{v}^{\theta }\in \mathcal{V}}.$$ (2)

The equivalent ODE (with an initial condition x) is

$$d{\varphi }^{\theta }(x,t)/dt=v^{\theta }({\varphi }^{\theta }(x,t)).$$ (3)

Remark 2

Eq. (2), whose unknown ϕ^θ(x, ·) is both inside and outside the integral, should not be confused with the piecewise-quadratic Ω → ℝⁿ map, $y\mapsto {\displaystyle {\int }_{0_{n\times 1}}^y{v}^{\theta }(x)dx}$. The latter, a popular tool in computer-vision [24] and numerical analysis, is unrelated to our work. Particularly, both x ↦ ϕ^θ(x, t) and t ↦ ϕ^θ(x, t) are not piecewise quadratic.

CPA-Based (CPAB) Transformations

Modulo that detail, any continuous Ω → ℝⁿ velocity field, whether PA or not, defines a transformation; i.e., a map whose input and output are viewed as points, not vectors. Letting x vary and fixing t, x ↦ ϕ^θ(x, t) is an Ω → Ω transformation. Without loss of generality (Section 4), we may set t = 1 and define

$$T^{\theta }(\cdot )\triangleq {\varphi }^{\theta }(\cdot ,1),\theta \in {\mathrm{ℝ}}^d.$$ (4)

Since we integrate CPA velocity fields, we coin our transformations CPA-Based.

Remark 3

CPAB transformations are not CPA (unless the CPA field is affine): while T^θ is continuous, it is not PA.

The notation T^θ = exp(v^θ) indicates that T^θ is defined by integrating v^θ ↔ θ for t = 1 (see Section 4.9). We let

$$M\triangleq \exp (\mathcal{V})\triangleq \{\exp (v^{\theta }):v^{\theta }\in \mathcal{V}\}$$ (5)

denote the space of CPAB transformations, where we notationally suppressed that both M and $\mathcal{V}$ depend on both Ω and $\mathcal{P}$ (i.e., more formally we should write $M_{\mathrm{\Omega },\mathcal{P}}$). Note that M is nonlinear: i.e., both T^θ + T^θ′ and αT^θ are usually not in M. The linearity of $\mathcal{V}$, together with $\exp :M\to \mathcal{V}$, however, provides a convenient way to handle this nonlinearity.

CPAB Transformations are Well Behaved

E.g., they are diffeomorphisms; moreover, (T^θ)⁻¹ ∈ M and the inversion is simple. Appealingly, useful subsets of M (e.g., volume-preserving CPAB transformations) are easily obtained via linear subspaces of $\mathcal{V}$ (we will return to this point in Section 4).

Remark 4

x ↦ ϕ^θ(x, t = 1) should not be confused with (the non-diffeomorphism, parametric optical-flow-like representation) x ↦ x + v^θ(x), the latter being only a Taylor approximation of the former. Thus, the way we utilize flexible parametrized vector fields is different from, e.g., [64], [65].

Integration of CPA Velocity Fields

Integral equations usually lack analytic solutions. Since CPA velocity fields are Lipschitz continuous and almost-everywhere smooth, generic integration solvers are quite effective for them. However, we can do even better. One of our contributions is showing that integration of such fields is given in either closed form (n = 1) or almost closed form (n > 1). Besides its obvious pluses (accuracy, computing time), this analytic solution makes it easier to interpret the resulting trajectories/transformation and is key to the theorems in Section 4.

A Specialized Numerical Solver

There is a shortcoming to that solution: as we will explain after Theorem 2, it requires tedious bookkeeping (if n > 1) and invoking certain routines (easy if n = 1 but hard if n > 1). This is especially a hurdle with GPU. We thus propose a practical alternative, a specialized solver for integrating CPA velocity fields, which is faster and more accurate than non-specialized solvers.

Convenient Modeling, Tractable Inference

Smoothness priors on M are easy to build and use. As is common with nonlinear spaces of nonlinear transformations, the nonlinearity of (θ, x) ↦ T^θ(x) prohibits closed forms for the posterior or Maximum Likelihood (ML) estimation. However, since our transformations are evaluated fast, we rapidly evaluate θ↦p(data|T^θ). This, together with the relatively-modest dimensionality and ease of using priors, facilitates the use of inference methods that require multiple likelihood evaluations (e.g., MCMC). Lastly, spaces of CPA velocity fields support coarse-to-fine approaches.

4 The Mathematical Representation

Let Ω be a Cartesian product of n compact intervals; i.e., Ω is a closed n-orthotope, also called a (closed) hyperrectangle. The lemmas/theorems below are proved in our supplemental material, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TPAMI.2016.2646685, henceforth referred to as Sup. Mat., available online.

4.1 Tessellations

A (finite) tessellation, denoted by $\mathcal{P}={\left\{U_c\right\}}_{c=1}^{N_{\mathcal{P}}}$ (where $N_{\mathcal{P}}$ is a positive integer), is a set of $N_{\mathcal{P}}$ closed subsets of Ω, also called cells, such that their union is Ω and the intersection of any pair of adjacent cells is their shared border (see Figs. 2 and 3). Henceforth we will always assume the cells are convex n-polytopes (namely, a subset of ℝⁿ whose sides are flat). We define 3 cell types. Type-I cells have n + 1 vertices; i.e., intervals (if n = 1), triangles (n = 2), tetrahedral (n = 3), etc. Type-II cells are n-orthotopes (i.e., hyperrectangles). Note intervals fit both these types. Type-III includes all other convex n-polytopes. We say that $\mathcal{P}$ is of type I (respectively, II, III) if all its cells are of type I (II, III). A type-II tessellation is called regular if its cells are evenly spaced along each of the n dimensions. As we will show, type-I tessellations are special, having benefits possessed by neither type II nor III. Mathematically, there is no difference (for our purposes) between types II and III; all theoretical results below that apply to type-II cells also apply to type-III cells. Type-II cells, however, lead to faster computations and simpler implementations than both other types, especially when compared with type III. Thus, unless stated otherwise, we restrict our attention to types I and II. Let

$${\mathcal{P}}_{\text{vert}}=\{\xi :\xi \text{is}\mathrm{a}\text{vertex}\text{of}\text{some}{U}_c\in \mathcal{P}\}$$ (6)

and let N_v denote its cardinality; e.g., see Fig. 2 and Table 1. Henceforth we will view ${\mathcal{P}}_{\text{vert}}$ as an ordered N_v-tuple; the ordering is arbitrary, but assumed fixed. Let ${N_{\mathcal{P}}}^{\prime }>N_{\mathcal{P}}$. An ${N_{\mathcal{P}}}^{\prime }$-cell tessellation, ${\mathrm{P}}^{\prime }={\left\{U_{c^{\prime }}\right\}}_{c^{\prime }=1}^{N_{{\mathrm{P}}^{\prime }}}$, is a refinement of $\mathcal{P}$, a relation denoted $\mathcal{P}\prec {\mathrm{P}}^{\prime }$, if each $U_{c^{\prime }}\in {\mathrm{P}}^{\prime }$ is a (possibly-improper) subset of some $U_c\in \mathcal{P}$.

TABLE 1.

Values of $N_{\mathcal{P}}$, N_v, $D=\dim ({\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}^{\prime })$, and $d=\dim ({\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}})$ for the $\mathcal{P}$’s shown in Fig. 2. See also Section 4.

$\mathcal{P}$	$N_{\mathcal{P}}$	Nv	$D=6N_{\mathcal{P}}$	d = 2Nv
${\mathcal{P}}_1$	4	5	24	10
${\mathcal{P}}_2$	16	13	96	26
${\mathcal{P}}_3$	64	41	384	82
${\mathcal{P}}_4$	256	145	1536	290
${\mathcal{P}}_5$	1024	1025	6144	1090

Algorithm 1.

Integrating $v^{\theta }\in \mathcal{V}$. See Text for Details:

Input: $\mathcal{P}={\left\{C_c\right\}}_{c=1}^{N_{\mathcal{P}}}$, $(A_{1,\theta },\dots ,A_{N_{\mathcal{P}},\theta })$; U = a sequence of Npts points in Ω; t > 0; Nsteps, $n_{\text{steps}}\in {\mathrm{\mathbb{Z}}}^{+}$
Output: ${\varphi }^{\theta }(U,t)\triangleq {\{{\varphi }^{\theta }(x,t)\}}_{x\in U}\subset \mathrm{\Omega }$
1:	Δt ← t/Nsteps; δt ← Δt/nsteps
2:	for $c\in \{1,\dots ,N_{\mathcal{P}}\}$ do in parallel
3:	$T_{A_{c,\theta },{\mathrm{\Delta }}_t}\leftarrow \exp ({\mathrm{\Delta }}_t\stackrel{\sim }{A_{c,\theta }})$
4:	for x ∈ U do in parallel
5:	x0 ← x
6:	for i ∈ {1, …, Nsteps} do
7:	${\stackrel{\sim }{x}}_{\text{temp}}={[x_{\text{temp}}^{T}1]}^T\leftarrow T_{A_{\gamma (x_i-1),\theta },{\mathrm{\Delta }}_t}{\stackrel{\sim }{x}}_i-1$
8:	if γ(xi−1) == γ(xtemp) then
9:	xi ← xtemp // analytic update
10:	else
	// inter-cell bdry crossed
11:	xi ← genericODEsolver(ini.con.=xi−1, step size=δt, nsteps, $v(\cdot ):x\mapsto A_{\gamma (x),\theta }\stackrel{\sim }{x}$)
12:	(ϕ)θ(x, t) ← xi

4.2 CPA Velocity Fields

Fix $\mathcal{P}$, let x ∈ Ω, and define the membership function $\gamma :\mathrm{\Omega }\to \{1,\dots ,N_{\mathcal{P}}\}$, γ : x↦min{c : x ∈ U_c}. i.e, if x is not on an inter-cell border, then γ(x) = c⇔x ∈ U_c, while a border point is (arbitrarily) assigned to the cell of the lower index.

Definition 1

A map, f : Ω → ℝⁿ, is called $PA(w.r.t.\mathcal{P})$ if ${\left\{f{|}_{U_c}\right\}}_{c=1}^{N_{\mathcal{P}}}$ are affine; i.e., $f(x)=A_{\gamma (x)}\stackrel{\sim }{x}$ where

$$\stackrel{\sim }{x}\triangleq \left[\frac{x}{1}\right]\in {\mathrm{ℝ}}^{n+1},A_c\in {\mathrm{ℝ}}^{n\times (n+1)}\forall c\in \{1,\dots ,N_{\mathcal{P}}\}.$$ (7)

Definition 2

f is called CPA if it is continuous and PA.

Fact 1

If f is CPA w.r.t. $\mathcal{P}$ it is CPA w.r.t. any ${\mathrm{P}}^{\prime }\succ \mathcal{P}$.

A vector field v (on Ω) is an Ω → ℝⁿ map viewed as the mapping of points to vectors; i.e., if x ∈ Ω then v(x) is viewed as an n-dimensional “arrow”. The terms velocity field and vector field will be used interchangeably.

Definition 3

A vector-field space is a linear space whose elements are vector fields.

Let ${\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}^{\prime }$ and ${\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}$ be the spaces of PA and CPA velocity fields on Ω w.r.t. $\mathcal{P}$. Note ${\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}\subset {\mathcal{V}}_{\mathrm{\Omega },\mathcal{P}}^{\prime }$. We will often write ${\mathcal{V}}^{\prime }$ and $\mathcal{V}$, suppressing the dependencies on Ω and $\mathcal{P}$.

Lemma 1

${\mathcal{V}}^{\prime }$ and $\mathcal{V}$ are linear spaces, $D\triangleq \dim ({\mathcal{V}}^{\prime })=(n^2+n)\times N_{\mathcal{P}}$, $d\triangleq \dim (\mathcal{V})\le n{N}_v$ with equality if and only if $\mathcal{P}$ is of type I, and d ≤ D with equality if and only if $N_{\mathcal{P}}=1$.

The values of D and d for the tessellations in Fig. 2 appear in Table 1. A generic element of ${\mathcal{V}}^{\prime }$ is denoted by v_A where $A\triangleq (A_1,\dots ,A_{N_{\mathcal{P}}})$ consists of its associated matrices (see Eq. (7)). If A ∈ ℝ^n×(n+1), then $\text{vec}(A)\in {\mathrm{ℝ}}^{n^2+n}$ is its row-by-row flattening to a column vector. Likewise, $\mathbf{v}\text{ec}(A)\triangleq {[{(\text{vec}(A_1))}^T\cdots {(\text{vec}(A_{N_{\mathcal{P}}}))}^T]}^T\in {\mathrm{ℝ}}^D$. Both vec(·) and vec(·) are linear bijections. An inner product on ${\mathcal{V}}^{\prime }$ is defined by $\langle v_{A_1},v_{A_2}\rangle =\mathbf{v}\text{ec}{(A_1)}^T\mathbf{v}\text{ec}(A_2)$.

We now explain how to build an orthonormal basis for $\mathcal{V}$, how its elements are parametrized by θ ∈ ℝ^d, and how PA velocity fields are projected onto $\mathcal{V}$.

Lemma 2

An element of $\mathcal{V}$ is any $v_A\in {\mathcal{V}}^{\prime }$ such that vec(A) satisfies a linear system of constraints, denoted by L vec(A) = 0.

Proof

For concreteness, we prove it for n = 2 and type-I tessellations, the other cases being similar. Let $v_A\in {\mathcal{V}}^{\prime }$. While v_A is continuous on every cell, it is (usually) discontinuous on cell boundaries. Consider two adjacent cells, U_i and U_j. Let x_a and x_b be their 2 shared vertices and let A_i and A_j denote the corresponding 2 × 3 matrices. Continuity of v_A at x_a implies 2 (more generally, n) linear constraints on A_i and A_j. Similarly, continuity at x_b implies another constraint pair. Thus, the continuity at both x_a and x_b implies the following 4 linear constraints:

$$\underset{4\times 12}{\left[\begin{array}{cccc}{\stackrel{\sim }{x}}_a^T& 0_{1\times 3}& -{\stackrel{\sim }{x}}_a^T& 0_{1\times 3}\\ 0_{1\times 3}& {\stackrel{\sim }{x}}_a^T& 0_{1\times 3}& -{\stackrel{\sim }{x}}_a^T\\ {\stackrel{\sim }{x}}_b^T& 0_{1\times 3}& -{\stackrel{\sim }{x}}_b^T& 0_{1\times 3}\\ 0_{1\times 3}& {\stackrel{\sim }{x}}_b^T& 0_{1\times 3}& -{\stackrel{\sim }{x}}_b^T\end{array}\right]}\underset{12\times 1}{\left[\frac{\text{vec}(A_i)}{\text{vec}(A_j)}\right]}=0_{4\times 1}.$$ (8)

Continuity at x_a and x_b implies continuity throughout their join, i.e., $A_i{\stackrel{\sim }{x}}_a=A_j{\stackrel{\sim }{x}}_a$ and $A_i{\stackrel{\sim }{x}}_b=A_j{\stackrel{\sim }{x}}_b$ imply (∀λ ∈ [0, 1])

$$A_i(\mathrm{\lambda }{\stackrel{\sim }{x}}_a+(1-\mathrm{\lambda }){\stackrel{\sim }{x}}_b)=A_j(\mathrm{\lambda }{\stackrel{\sim }{x}}_a+(1-\mathrm{\lambda }){\stackrel{\sim }{x}}_b).$$ (9)

Any v_A whose A_i and A_j satisfy Eq. (9) is thus continuous on U_i ∪ U_j. Similar constraints for other pairs of adjacent cells can be stacked together in an analogous equation:

$$\underset{4N_e\times 6N_{\mathcal{P}}}{L}\underset{6N_{\mathcal{P}}\times 1}{\mathbf{v}\text{ec}(A)}=0_{4N_e\times 1}$$ (10)

where N_e is the number of shared line segments in $\mathcal{P}$ and L is the constraint matrix. Any v_A whose A satisfies Eq. (10) is thus everywhere continuous. We conclude that the null space of L, denoted by null(L), coincides with $\mathcal{V}$. □

Let the columns of B = [B₁ … B_d] ∈ ℝ^D×d denote the particular orthonormal basis of null(L) which is obtained via SVD of L. Let θ = [θ₁ … θ_d]^T ∈ ℝ^d. If vec(A) = Bθ then v_A : Ω → ℝⁿ is CPA. Particularly, for every j ∈ {1, …, d}, $v_{\mathbf{v}{\text{ec}}^{-1}(B_j)}:\mathrm{\Omega }\to {\mathrm{ℝ}}^n$ is CPA and ${\left\{v_{\mathbf{v}{\text{ec}}^{-1}(B_j)}\right\}}_{j=1}^d$ is a basis for $\mathcal{V}$. For a visualization of this basis, see Sup. Mat., available online Regardless of whether $v_A\in {\mathcal{V}}^{\prime }$ is CPA or not, $v_{\mathbf{v}{\text{ec}}^{-1}(B{B}^T\mathbf{v}\text{ec}(A))}$, its projection on $\mathcal{V}$, is CPA. When earlier we denoted a generic element of $\mathcal{V}$ by v^θ we meant that θ stands for the coefficients w.r.t. ${\left\{v_{\mathbf{v}{\text{ec}}^{-1}(B_j)}\right\}}_{j=1}^d$:

$$v^{\theta }(x)=A_{\gamma (x),\theta }\stackrel{\sim }{x}={\displaystyle {\sum }_{j=1}^d{\theta }_j{v}_{\mathbf{v}{\text{ec}}^{-1}(B_j)}}\stackrel{\sim }{x}$$ (11)

where $A_{\theta }\triangleq (A_{1,\theta },\dots ,A_{N_{\mathcal{P}},\theta })\triangleq \mathbf{v}{\text{ec}}^{-1}({\displaystyle {\sum }_{j=1}^d{\theta }_j{B}_j})$. This basis is orthonormal w.r.t. the inner product

$${\langle \cdot ,\cdot \rangle }_B:\mathcal{V}\times \mathcal{V}\to \mathrm{ℝ},(v^{{\theta }_1},v^{{\theta }_2})\mapsto {\theta }_1^T{\theta }_2.$$ (12)

We refer to it as a global basis since each $v_{\mathbf{v}{\text{ec}}^{-1}(B_j)}$ impacts v^θ on the entirety of Ω. We will soon describe another orthonormal basis which is more local in nature.

Let $v_{\text{vert}}^{\theta }\triangleq {\{v^{\theta }(\xi )\}}_{\xi \in {\mathcal{P}}_{\text{vert}}}$ denote the (ordered) N_v-tuple of n-dimensional values v^θ takes at the N_v points in ${\mathcal{P}}_{\text{vert}}$.

Lemma 3

Let $L_{\theta \mapsto v_{\text{vert}}^{\theta }}$ denote the map that sends θ to $v_{\text{vert}}^{\theta }$. $L_{\theta \mapsto v_{\text{vert}}^{\theta }}$ is a linear injection and its associated d × (nN_v) matrix is given in closed form. If $\mathcal{P}$ is of type I, the map is invertible. In which case, we denote its inverse by $L_{v_{\text{vert}}^{\theta }\mapsto \theta }$.

Let ∂Ω be the border of Ω and n : ∂Ω → Sⁿ⁻¹ be the unit normal to it. We now discuss linear subspaces of $\mathcal{V}$.

Lemma 4

Optional constraints such as 〈v^θ(x), n(x)〉 = 0 on ∂Ω and/or ${\{\text{tr}(A_c)=0\}}_{c=1}^{N_{\mathcal{P}}}$, are linear and can thus extend L (from Lemma 2) to have more rows. The null space of the extended L is a linear subspace (whose dimension is denoted by d′ < d) of the null space of the original L. Reusing the symbol $B=[B_1\dots B_{d^{\prime }}]\in {\mathrm{ℝ}}^{D\times d^{\prime }}$ to denote the particular orthonormal basis of this new null space which is obtained via SVD of the extended L, we get a d′-dimensional basis of CPA velocity fields, ${\left\{v_{\mathbf{v}{\text{ec}}^{-1}(B_j)}\right\}}_{j=1}^{d^{\prime }}$, whose elements satisfy the new constraint(s).

The corresponding linear subspaces of $\mathcal{V}$ are denoted ${\mathcal{V}}^{\partial }$ and ${\mathcal{V}}^{\text{tr}}$. We also define their intersection, ${\mathcal{V}}^{\partial ,\text{tr}}\triangleq {\mathcal{V}}^{\partial }\cap {\mathcal{V}}^{\text{tr}}$. We reuse the symbol v^θ to denote an element of any of these subsets, with the understanding that then θ ∈ ℝ^d′ for some d′ < d. We now define, similarly to Eq. (5), subsets of M:

$$M^{\partial }\triangleq \{\exp (v^{\theta }):v^{\theta }\in {\mathcal{V}}^{\partial }\}\subset M;$$ (13)

$$M^{\text{vp}}\triangleq \{\exp (v^{\theta }):v^{\theta }\in {\mathcal{V}}^{\text{tr}}\}\subset M;$$ (14)

$${\mathrm{M}}^{\partial ,\text{vp}}\triangleq \{\exp (v^{\theta }):v^{\theta }\in {\mathcal{V}}^{\partial ,\text{tr}}\}\subset M.$$ (15)

If $\mathcal{P}$ is of type I (so d = nN_v) then, by viewing $v_{\text{vert}}^{\theta }$ as a point in ℝ^d, the standard basis for ℝ^d defines another basis for $\mathcal{V}$ (there is a linear bijection between the (n + 1)-tuple of the n-dimensional velocities at the vertices of a type-I U_c and the n × (n + 1) matrix A_c). Naturally, we refer to this vertex-based basis as a localized basis of $\mathcal{V}$. It is orthonormal w.r.t. the following inner product, obtained by summing the dot products between pairs of velocities at the vertices:

$${\langle \cdot ,\cdot \rangle }_{\text{vert}}:(v^{{\theta }_1},v^{{\theta }_2})\mapsto {\displaystyle {\sum }_{\xi :\xi \in {\mathcal{P}}_{\text{vert}}}{v}^{{\theta }_1}(\xi )}\cdot v^{{\theta }_2}(\xi ).$$ (16)

For ${\mathcal{V}}^{\partial }$, we can build a similar (d′-dimensional) localized basis; however, for ${\mathcal{V}}^{\text{tr}}$ (and ${\mathcal{V}}^{\partial ,\text{tr}}$) this is not possible. We conclude Section 4.2 with the following fact.

Algorithm 2.

A Greedy Algorithm for (Landmark-based) Inference over a Transformation Obtained by Integrating a Non-Stationary Field where the Field is Piecewise-Constant w.r.t. Time. See Text for Details:

Input : Npts pairs of Ω-valued points: ${\{(x_i,y_i)\}}_{i=1}^{N_{\text{pts}}}$, ε; kmax
1:	for i ∈ {1, …, Npts} do in parallel
2:	$x_i^{[0]}\leftarrow x_i$
3:	for k ∈ {1, …, kmax} do
4:	${\text{obs}}^k\leftarrow {\{(x_i^{k-1},y_i)\}}_{i=1}^{N_{\text{pts}}}$ // observations
5:	θk ← arg maxθp(θk)Lv(vθ; obsk) where $L_v(v^{{\theta }^k};{\text{obs}}^k)\propto e^{-\frac{1}{{\sigma }^2}{\displaystyle {\sum }_{i=1}^{N_{\text{pts}}}{\Vert x_i^{k-1}+v^{\theta }(x_i^{k-1})-y_i\Vert }^2}}$ // closed-form optimization; see text
6:	${\alpha }_k\leftarrow {\mathit{argmax}}_{\alpha }p(\alpha {\theta }^k)L_T(T^{\alpha {\theta }^k};{\text{obs}}^k)$ where $L_T(T^{\alpha {\theta }^k};{\text{obs}}^k)\propto e^{-\frac{1}{{\sigma }^2}{\displaystyle {\sum }_{i=1}^{N_{\text{pts}}}{\Vert T^{\alpha {\theta }^k}(x_i^{k-1})-y_i\Vert }^2}}$ /* the 1D optimization can be done by, e.g., a line search; ${\{T^{\alpha {\theta }^k}(x_i^{k-1})\}}_{i=1}^{N_{\text{pts}}}$ is computed by Algorithm 1
7:	if ‖θkαk‖ < ε then break
Output $\widehat{T}\triangleq (T^{{\alpha }_k{\theta }^k}\circ \cdots \circ T^{{\alpha }_1{\theta }^{[1]}})$ // $\widehat{T}$ is a concatenation of points in M

Fact 2

〈·,·〉_B and Bvary with translations of Ω while 〈·,·〉_vert and the localized basis are invariant to them.

4.3 Smoothness Priors on CPA Velocity Fields

Adopting a Bayesian approach, we now present two complementary methods to construct priors on CPA fields.

I: Project a prior from ${\mathcal{V}}^{\prime }$ onto $\mathcal{V}$

On ${\mathcal{V}}^{\prime }$, it is easy to build a smoothness prior that also penalizes large values; e.g., we use a zero-mean Gaussian with a D × D covariance, Σ_PA, whose correlations decay with inter-cell distances, and write $\mathbf{v}\text{ec}(A)~\mathcal{N}(0_{D\times 1},{\sum }_{\text{PA}})$. Next, we use the ${\mathcal{V}}^{\prime }\to \mathcal{V}$ projection, A↦θ = B^Tvec(A), to induce a prior on $\mathcal{V}$:

$$p(\theta )=\mathcal{N}(0_{d\times 1},{\sum }_{\text{CPA}})$$ (17)

where Σ_CPA = B^TΣ_PAB. See Fig. 4 for samples from p(θ). By Fact 2, p(θ) depends on the origin of Ω.

II: Let a prior on $v_{\text{vert}}^{\theta }$ induce a prior on $v^{\theta }\in \mathcal{V}$

If the prior on $v{}_{\text{vert}}^{\theta }$ is invariant to the origin of Ω then so is the resulting $p(\theta =L_{v_{\text{vert}}^{\theta }\mapsto \theta }{v}_{\text{vert}}^{\theta })$; e.g., such is the case for either a zero-mean Gaussian whose covariance is fully determined by inter-vertex distances, or for an MRF of the form ${\displaystyle {\prod }_{i,j:{\xi }_i\underset{~}{\text{nbrs}}\xi }\psi (v^{\theta }({\xi }_i),v^{\theta }({\xi }_j))}$ where ψ is some pairwise potential. Method II does not require the computation of B. Unlike method I, however, it is applicable only if $\mathcal{P}$ is of type Iand volume preservation is not imposed. Another advantage of method I over method II (or, for that matters, over most regularizers/priors people often place on velocity fields in the context of diffeomorphisms) is that by defining the prior in terms of the A_c’s (as opposed to the $v_{\text{vert}}^{\theta }$ values they generate), we avoid introducing fictitious variance. For example, if v^θ is purely affine then the A_c’s are all the same. Thus, w.r.t. the prior defined using the global basis (i.e., method I), v^θ is considered very smooth (in the machine-learning sense, not calculus one). However, w.r.t. the localized basis (i.e., method II), v^θ is less smooth since the values of $v_{\text{vert}}^{\theta }$ vary with the vertex location.

4.4 CPA-Based Transformations

Recall from Section 3 that, via Eq. (2), $v^{\theta }\in \mathcal{V}$ implies T^θ ∈ M, a relation denoted T^θ = exp(v^θ) (see Eq. (5)). The detail alluded to in Section 3 is that for Eq. (2) to be well defined, ϕ^θ(x, τ) must always be in Ω, the domain of v^θ. This trivially holds if Ω = ℝⁿ. It also holds if Ω⊊ℝⁿ and $v^{\theta }\in {\mathcal{V}}^{\partial }$. If Ω⊊ℝⁿ and $v^{\theta }\in \mathcal{V}\backslash {\mathcal{V}}^{\partial }$, it might not. Also, some of the results below require both the continuity and PA properties of v^θ. This brings us to the following lemma.

Lemma 5

If Ω⊊ℝⁿ, we can extend $\mathcal{P}$ and constrain the velocity fields such that we will obtain a linear subspace of $\mathcal{V}$ whose elements extend to CPA fields on the entirety of ℝⁿ. In which case, we also redefine Ω to be equal to (the whole of) ℝⁿ.

Henceforth, if Ω⊊ℝⁿ and we use $\mathcal{V}$ (or ${\mathcal{V}}^{\text{tr}}$) and not ${\mathcal{V}}^{\partial }$ (or ${\mathcal{V}}^{\partial ,\text{tr}}$), we will assume the procedure from Lemma 5 has been applied. Thus, Eq. (2) is well defined, T^θ is always an Ω → Ω map and v^θ is indeed CPA. Either way, ϕ^θ(x, 0) = x for every x ∈ Ω while {x : v^θ(x) = 0} are fixed points of T^θ.

Fact 3

If A ∈ ℝ^{n (n+1)}, let $\stackrel{\sim }{A}=\left[\genfrac{}{}{0ex}{}{A}{0_{1\times (n+1)}}\right]\in {\mathrm{ℝ}}^{(n+1)\times (n+1)}$. The last row of $\mathbf{exp}\mathbf{m}(\stackrel{\sim }{A})$ (where expm is the matrix exponential) is [0_1×n 1]. Also, det $\mathbf{exp}\mathbf{m}(\stackrel{\sim }{A})>0$.

If n = 1 (so $\stackrel{\sim }{A}$ is 2-by-2) or if A has a special structure (e.g., if A_1:n,1:n+1 is either diagonalizable, idempotent, or a 3-by-3 skew-symmetric matrix), then $\mathbf{exp}\mathbf{m}(\stackrel{\sim }{A})$ has a closed form. Otherwise, since n is usually small and due to the structure of $\stackrel{\sim }{A}$, a generic expm routine typically approximates it well. Let t ∈ ℝ and let ${\psi }_{\theta ,c}^t:\mathrm{\Omega }\to {\mathrm{ℝ}}^n$ be the solution to an ODE with an ℝⁿ → ℝⁿ affine velocity field, $\xi \mapsto A_{c,\theta }\stackrel{\sim }{\xi }$, and an initial condition, x:

$$\left[\genfrac{}{}{0ex}{}{{\psi }_{\theta ,c}^t(x)}{1}\right]\triangleq T_{A_{c,\theta },t}\stackrel{\sim }{x},T_{A_{c,\theta },t}\triangleq \mathbf{exp}\mathbf{m}(t\stackrel{\sim }{A_{c,\theta }}).$$ (18)

We note that the solution to Eq. (2) is the composition of a finite number, denoted by m, of such solutions:

$${\varphi }^{\theta }(x,t)=({\psi }_{\theta ,c_m}^{t_m}\circ \dots \circ {\psi }_{\theta ,c_2}^{t_2}\circ {\psi }_{\theta ,c_1}^{t_1})(x).$$ (19)

As mentioned earlier, T^θ is defined via T^θ(x ≜ ϕ^θ(x, 1). The compact form of Eq. (19) hides a difficulty: the number of the trajectory segments, m, their durations, ${\left\{t_i\right\}}_{i=1}^m$, and the indices of the cells involved, ${\left\{c_i\right\}}_{i=1}^m$ (where a cell may appear more than once), all depend on x. Except the index of the first cell, c₁ = γ(x), they also depend on θ and t. Thus, a more precise and cumbersome notation would be:

$$\left({\psi }_{\theta ,c_{m_{x,\theta ,t}}(x,\theta ,t)}^{t_{m_{x,\theta ,t}(x,\theta ,t)}}\circ \dots \circ {\psi }_{\theta ,c_2(x,\theta ,t)}^{t_2(x,\theta ,t)}\circ {\psi }_{\theta ,c_1(x)}^{t_1(x,\theta ,t)}\right)(x).$$

Remark 5

While invertible affine matrices form a group, ϕ^θ(·, t) : Ω → Ω is not affine, exactly because of the x-dependencies mentioned above. In fact, since the above quantities vary with x even within a cell, the continuous x↦ϕ^θ(x, t) is not PA, hence not CPA. the term CPA-based.

A map, T : Ω → Ω, is a called a (C¹) diffeomorphism (on Ω) if T⁻¹ exists and both T and T⁻¹ are differentiable. Let G be the space of (orientation-preserving) diffeomorphisms on Ω. Let H ⊂ G be its restriction to those diffeomorphisms that can be obtained, via integration, from uniformly-continuous stationary velocity fields; i.e., T(·) = ϕ(·,t) where $\varphi (x,t)=x+{\displaystyle {\int }_0^{t}v(\varphi (x,\tau ))d\tau }$ for a uniformly-continuous v : Ω → ℝⁿ. Both G and H are ∞–dimensional nonlinear spaces.

Theorem 1

(i) If α ∈ ℝ then v^θ = v^αθ/α and ϕ^θ(·, t) = ϕ^αθ(·, t/α). (ii) (T^θ)⁻¹ exists, is in M, and equals to T^−θ. (iii) M ⊂ G. (iv) If T^θ ∈ M^vp then T^θ is volume-preserving (hence the superscripted vp). (v) M is a d-dimensional nonlinear space; similar statements hold for M^∂, M^vp and M^∂,vp, with their respective values of d′. (vi) If ${\mathcal{P}}_1\prec {\mathcal{P}}_2\prec \dots$ is a tessellation sequence such that eventually all the cells become arbitrarily small, then $M_{\mathrm{\Omega },{\mathcal{P}}_k}\underrightarrow{k\to \mathrm{\infty }}H$.

Remark 6.3

Since M is nonlinear, it is easier to work in the linear $\mathcal{V}$ and then, via exp, move from $\mathcal{V}$ to M. This (tangent-space-based) approach is popular in the context of nonlinear differentiable manifolds and, particularly, is used extensively in various spaces of diffeomorphisms (e.g., [27]).

Corollary 1

If n = 1, T^θ is increasing. Let J ⊂ ℝ be an interval. If F₀ : J → Ω is either non-decreasing, increasing, right-continuous, continuous, differentiable, a diffeomorphism, or a step function (7 non-mutually-exclusive cases), then so is F^θ ≜ T^θ ∘ F₀ : J → Ω. If, in addition, Ω = [0, 1], $v^{\theta }\in {\mathcal{V}}^{\partial }$ and F₀ is a CDF, then F^θ is a CDF. The (trivial) proof is omitted.

This gives an unconstrained parametrization of large classes of increasing functions and CDFs/histograms via a finite-dimensional linear space (Figs. 5 and 6). In the CDF case, F₀ need have neither a density nor finite moments.

Fig. 5.

Monotonic regression on synthetic data (top) by inferring CPA fields (bottom), defined over 100 equal-length cells. We used a squared loss in the 4 leftmost columns and a robust one (Geman-McClure) in the 2 others. Col. 1, 3, and 5 show ML solutions. Col. 2, 4 and 6 show MAP solutions with a smoothness prior. In all cases, the function is increasing, as obtained effortlessly from the representation.

Fig. 6.

CDF/histogram representation. 1^st row: $N_{\mathcal{P}}=10$. 2^nd row: $N_{\mathcal{P}}=100$. F₀, not shown, is a CDF of a uniform distribution on J = [−3, 3]. (a) $v^{\theta }\in {\mathcal{V}}^{\partial }([0,1],\mathcal{P})$. (b) F = T^θ ∘ F₀. (c) $\frac{d}{dx}F$. Note it is not piecewise constant. H₀, not shown, is a cumsum of a 20-bin uniform histogram. (e) H = T^θ ∘ H₀, a cumsum of a new histogram h (f) h.

4.5 Integration Details

Equation (19) justifies and simplifies the proof of Theorem 1, provides an insight into the structure of t↦ϕ^θ(x, t), and, for n = 1, yields a closed-form solution for x↦T^θ(x). If n > 1, it suggests a practical, essentially-exact solution.

Theorem 2

If n = 1 then m_x,θ,t, ${\{t_i(x;\theta ,t)\}}_{i=1}^{m_{x,\theta ,t}}$, and ${\{c_i(x;\theta ,t)\}}_{i=2}^{m_{x,\theta ,t}}$ have closed forms. Thus, so does T^θ(x).

Proof

The notation below drops the θ-dependencies. Without loss of generality, let t > 0. For $c\in {\{c_i(x;t)\}}_{i=2}^{m_{x,t}}$, let A_c = [a_c, b_c] and let $U_c=\left[x_c^{min},x_c^{max}\right]$. It can be shown that

$${\psi }_c^t(x)=\{\begin{array}{ll}{e}^{t{a}_c}x+\frac{b_c(e^{t{a}_c}-1)}{a_c}\hfill & \text{if}{a}_c\ne 0\hfill \\ x+t{b}_c\hfill & \text{if}{a}_c=0\hfill \end{array}.$$ (20)

Note c₁ = γ(x) and suppose v(x) > 0. Let t₁(x) be the hitting time of the right boundary of $U_{c_1}$:

$$t_1\triangleq min\{t:{\psi }_{c_1}^t(x)=x_{c_1}^{max}\}$$ (21)

where, by convention, the minimum of an empty set is ∞. If $v(x_{c_1}^{max})\le 0$ then t₁ = ∞. Otherwise,

$$t_1=\{\begin{array}{cc}\frac{1}{a_{c_1}}\log \left(\frac{x_{c_1}+\frac{b_{c_1}}{a_{c_1}}}{x+\frac{b_{c_1}}{a_{c_1}}}\right)& \text{if}{a}_{c_1}\ne 0\\ \frac{x_{c_1}-x}{b_{c_1}}& \text{if}{a}_{c_1}=0\end{array}.$$ (22)

If t₁ > t, we are done: ${\varphi }^{\theta }(x,t)={\psi }_{c_1}^t(x)$. Otherwise, we entered the next interval to the right, $U_{c_1+1}$ (so c₂ = c₁ + 1), and the process (i.e., solving for t₂, but this time with $x_{c_1}^{max}$ as the new starting point instead of x, with t − t₁ instead of t, and c₂ instead of c₁) is redone iteratively till convergence (the number of such steps, m_x,t, is finite since t is finite). A loose upper bound on m_x,t is $N_{\mathcal{P}}-c_1+1$. The case where v(x) < 0 is handled similarly. Taken together,

$$\varphi (x;t)=({\psi }_{c_m}^{t_m}\circ \dots \circ {\psi }_{c_2}^{t_2}\circ {\psi }_{c_1}^{t_1})(x)$$ (23)

where for 2 ≤ i ≤ m_x,_t, c_i = c_i−1 + sign(v(x)), while m_x,_t and ${\left\{t_i\right\}}_{i=1}^{m_{x,t}}$ are found as describe above. □

If n > 1 then T^θ(x) is given essentially in closed from in the sense that Eq. (19) still holds, but there is the mild issue that a generic expm is needed (see also the matrix decomposition suggested in [27]) and, much more importantly, to find the quantities from Theorem 2 one needs, for every x, to sequentially solve, an x-dependent number of problems of the form $\arg {min}_{t>0}\gamma ({[\mathbf{exp}\mathbf{m}(t\stackrel{\sim }{A})]}_{1:n,1:n+1}\stackrel{\sim }{\xi })\ne \gamma (\xi )$ where the pair (ξ, A) ∈ Ω×ℝ^n×(n+1) is (in general) different in each problem. Doing this for n = 1 is easy, fast, and accurate. But for n ≥ 2, partly since ϕ^θ(x, ·) may reenter a cell (prohibiting, e.g., a binary-search method), this requires tedious bookkeeping, multiple expm calls, and multiple invocations of a numerical solver. This yields slow approximated solutions that are also hard to implement in GPU.

Inference over diffeomorphisms (CPAB included) often requires evaluating T^θ for multiple values of θ (e.g., 10,000), and a high N_pts, where N_pts is number of points to be transformed (e.g., the number of image pixels). Preferring a more practical alternative, we propose a specialized solver for integrating CPA fields. This solver, summarized in Algorithm 1, alternates between the analytic solution (with neither bookkeeping nor need to solve explicitly for m_x,θ,1, ${\{c_i(x;\theta ,1)\}}_{i=2}^{m_{x,\theta }}$ and ${\{t_i(x;\theta ,1)\}}_{i=1}^{m_{x,\theta }}$) and a generic solver. Thus, the proposed specialized solver is both faster and more accurate than a generic solver since most of the trajectory is solved exactly, while only in small portions of the trajectory does our solver resort to the generic one. Importantly, in Algorithm 1, the required number of expm calls is constant, $N_{\mathcal{P}}$, and grows with neither N_pts nor t. In fact, since Algorithm 1 is highly accurate, fast, and simple, and since usually $N_{\mathcal{P}}\ll N_{\text{pts}}$, we prefer to use it even when n = 1. Lower/upper bounds on the algorithm’s complexity are O(C₁) + O(C₂ × N_steps) and O(C₁) + O(C₂ × N_steps × n_steps) where C₂ = N_pts/#cores and $C_1=\text{expm}\prime \mathrm{s}\text{cost}\times N_{\mathcal{P}}/\#\text{cores}$. Usually, the O(C₁) term is negligible. $N_{\mathcal{P}}$ and θ affect inter-cell moves hence which bound is tighter; usually it is the lower one.

Remark 7

Consecutive analytic updates (for the same ${\mathrm{\Delta }}_t\stackrel{\sim }{A_c}$) amount to taking powers of expm ( ${\mathrm{\Delta }}_t\stackrel{\sim }{A_c}$). While ${(\mathbf{exp}\mathbf{m}({\mathrm{\Delta }}_t\stackrel{\sim }{A_c}))}^k=\mathbf{exp}\mathbf{m}(k{\mathrm{\Delta }}_t\stackrel{\sim }{A_c})$, on a computer numerical errors accrue and the equality may not hold [66], [67]. This issue is most noticeable when the matrices are large (and dense) and when k is large. Here, however, this issue does not arise in practice since both the matrices (which also have zeros in the last row) and k are small.

In Algorithm 1, γ is called often (though far less often than when using a generic solver throughout). In our implementation, we decided against a lookup-table approach since, once the trajectories start evolving, the argument to γ takes values in the continuum. Besides their simplicity, a key advantage of type-II cells, most notably when compared with type-III cells, is that evaluating γ is easy and fast, especially when $\mathcal{P}$ is also regular (since we can then use rounding and modulo operations). Thus, e.g., we construct our type-I tessellations as a refinement of regular type-II tessellation. If n = 2, we split each rectangle into 4 triangles while if n = 3 we split each box into 5 tetrahedra. When evaluating γ, we first find the “parent” hyperrectangle and then pick the correct type-II cell among those within that hyper-rectangle. For n > 3, we avoided implementing type-I cells; rather, in such cases we use only type-II tessellations.

4.6 Adaptive Tessellations

It is possible to use CPAB transformations with a data-adaptive $\mathcal{P}$; e.g., placing more cells in certain regions. The potential expressiveness gain, however, needs to be compared against: 1) the slowdown in evaluating γ and thus in evaluating x ↦ T^θ(x); 2) the need to recompute B (unless one is content with a localized basis and priors obtained via method II). In any case, while for high values of n (say, n > 3) this may be a reasonable approach to mitigate the curse of dimensionality, we prefer, to keep computing time low and implementation simple, to use fixed tessellations and handle scarce-data regions via the smoothness prior.

4.7 Computing Derivatives of T^θ(x) w.r.t. θ

One of our main motivations was facilitating tractable MCMC inference over latent diffeomorphisms. While we empirically found MCMC methods to be easier to tune and to produce better results than gradient-based optimization, for completeness we provide the details for computing dT^θ(x)/dθ. While one can also use finite differences, the derivation below leads to better accuracy and some computational savings (e.g., no additional expm calls are needed). Let θ_j denote the jth component of θ. Define $y(x,t)\triangleq \frac{d}{d{\theta }_j}{\varphi }^{\theta }(x,t)$. It can be shown that y(x, t) satisfies

$$\underset{n\times 1}{\underbrace{y(x,t)}}={\displaystyle {\int }_0^t\underset{n\times (n+1)}{\underbrace{B_{c,j}}}\underset{(n+1)\times 1}{\underbrace{{\varphi }^{\theta }(\stackrel{\sim }{x,\tau })}}+\underset{n\times (n+1)}{\underbrace{A_{c,\theta }}}\underset{(n+1)\times n}{\underbrace{y(\stackrel{\sim }{x,\tau })}}d\tau }$$ (24)

where c = γ(ϕ^θ(x, τ)) and B_c,j is the cth matrix in $\mathbf{v}{\text{ec}}^{-1}(B_j)=(B_{1,j},\dots ,B_{N_{\mathcal{P}},j})$. Setting t = 1, the solution of these d equations (for j ∈ {1, …; d}) yields $\frac{d{T}^{\theta }(x)}{d\theta }$. Any generic numerical solver for integral equations can be used to solve these equations, where the required values of ϕ^θ(x, τ) can be computed using Algorithm 1. We omit the details, but our implementation includes this derivative.

4.8 Time-Dependent Fields and Their Inference

This paper focuses on stationary CPA fields but, more generally, θ may vary with t. In which case, however, Algorithm 1 and Eq. (19) no longer apply (but note that most of the other advantages of the proposed approach will still apply; e.g., relatively-low dimensionality, compact representation, etc.). While we are unaware of a way to exploit the structure of general non-stationary CPA fields for better integration, we can still apply any generic integration technique to them. That said, one useful case of non-stationary fields is special: if θ(t) is piecewise constant w.r.t. t, then we (trivially) apply Algorithm 1 consecutively, once within each time interval. Here is a usage example. Given N_pts pairs of Ω-valued points, ${\{(x_i,y_i)\}}_{i=1}^{N_{\text{pts}}}$, consider an optical-flow-like representation, x↦x + v^θ(x), together with the following Gaussian likelihood model,

$$y_i\stackrel{\mathrm{I}.\mathrm{I}.\mathrm{D}.}{~}\mathcal{N}(x_i+v^{\theta }(x_i),{\sigma }^2{I}_{n\times n})$$ (25)

(σ > 0), and a Gaussian prior for θ as in, e.g., Eq. (17). The following Ω × ℝ^d → ℝⁿ map,

$$(x,\theta )\mapsto x+v^{\theta }(x)\stackrel{\text{Eqn}.(11)}{=}x+{\displaystyle {\sum }_{j=1}^d{\theta }_j{v}_{\mathbf{v}{\text{ec}}^{-1}(B_j)}\stackrel{\sim }{x}},$$ (26)

is affine w.r.t. θ = [θ₁, …, θ_d]^T. It follows that there is a closed-form MAP solution for θ (obtained by applying the standard result of computing the conditional mean of one random variable given the other, when both are jointly Gaussian). Let ${\widehat{\theta }}_{\text{MAP}}$ denote this solution. Importantly, the Ω → ℝⁿ map,

$$x\mapsto x+v^{{\widehat{\theta }}_{\text{MAP}}}(x),$$ (27)

is (usually) not a diffeomorphism; however, for any real scalar α, the Ω → Ω map,

$$x\mapsto T^{\alpha {\widehat{\theta }}_{\text{MAP}}}(x)=\exp (v^{\alpha {\widehat{\theta }}_{\text{MAP}}})(x),$$ (28)

is a (CPAB) diffeomorphism. This suggests a practical greedy approach for a quick-and-dirty inference over (a transformation obtained by the integral of) a CPA velocity field which is piecewise-constant w.r.t. time. This method, summarized in Algorithm 2, may be adjusted to the case of no-correspondences image registration setting.

4.9 Some Differential-Geometric Aspects

Readers less interested in differential geometry may safely skip this section. As many readers must have recognized, $\mathcal{V}$ is the tangent space at the identity to M; i.e., $\mathcal{V}=T_{I}M$. Like the transformation space from [27], M is not a Lie group (lack of closure under composition); equivalently, $\mathcal{V}$ is not a Lie algebra (lack of Lie-bracket closure). Note that $\mathcal{V}$ is a finite-dimensional linear subspace of the (∞-dimensional) Lie algebra, T_IG (the Lie group G was defined before Theorem 1), and that exp : $\mathcal{V}\to M$ is the restriction of the Lie-group exponential map, exp : T_IG → G, to $\mathcal{V}\to T_{I}G$. This approach, of restricting a Lie group exponential map to a linear subspace (of the Lie algebra) lacking a Lie-bracket closure, was found useful, e.g., in the context of Symmetric Positive-Definite (SPD) matrices (SPD is not closed under the standard binary operation of matrix Lie groups; likewise, T_ISPD is not closed under the Lie Bracket.) [39]. The inner product (on $\mathcal{V}=T_{I}M$) in Eq. (16) is close in spirit to the oft-used inner product (on T_IG):

$$\langle v_1,v_2\rangle ={\displaystyle {\int }_{\mathrm{\Omega }}{v}_1(\xi )\cdot v_2(\xi )d\xi }{v}_1,v_2\in T_{I}G.$$ (29)

As we pointed out, one of the advantages of Eq. (12) over Eq. (16), is related to eliminating fictitious noise. We are unaware of works (including those using Sobelev spaces) suggesting an inner product on T_IG with a similar property. While here we adopt an approach exploiting (the restriction of) the Lie group exponential of G ⊃ M, future work should explore Riemannian geometries of M (with their Riemannian exponential maps) and their relations to those of G. While one can define M-specific Riemannian metrics, note that M may also inherit a Riemannian metric from G.

5 Experiments

Sections 5.1, 5.2 and 5.3 focus on synthesis, Section 5.4 pertains to analysis by synthesis, Section 5.5 addresses large numbers of landmarks, Section 5.6 is related to Algorithm 2, and Section 5.7 illustrates representation of CDFs.

5.1 Integration Accuracy and Timings

The two parameters in Algorithm 1, N_steps and n_steps, imply two step sizes, one large, Δ_t = t/N_steps, used for exact updates, and one small, δ_t = Δ_t=n_steps, used for the generic-solver subroutine. The reason we can make large (hence few) steps is that we use an exact analytic update when possible. In our experiments, the generic solver was the modified Euler Method, which requires (i) the ability to evaluate a velocity field at a given location and (ii) specifying the step size and number of steps. By construction, Algorithm 1 is more accurate (due to the analytic updates) and faster (due to the smaller number of steps) than any generic solver used as its subroutine. We also verified this empirically, comparing the algorithm against simply running the generic solver step size δ_t and N_steps × n_steps steps. For accuracy, in 1D, we compared the integration error (that we can compute this error at all, is by the virtue of our closed-form solution; see Theorem 2) averaged over random CPA velocity fields; i.e., for each value of $N_{\mathcal{P}}$, we drew 100 CPA fields form the prior, ${\left\{v^{{\theta }_i}\right\}}_{i=1}^{100}$, sampled (uniformly) 1000 points in Ω, ${\left\{x_j\right\}}_{j=1}^{1000}$ and then computed the error:

$$\epsilon =\frac{1}{100}{\displaystyle {\sum }_{i=1}^{100}\frac{1}{1000}}{\displaystyle {\sum }_{i=j}^{1000}|T_{\text{exact}}^{\theta }(x_j)-T_{\text{solver}}^{\theta }(x_j)|}.$$ (30)

Table 2 shows the relative improvements, $\frac{{\epsilon }_{\text{generic}}-{\epsilon }_{\text{algorithm}}}{{\epsilon }_{\text{generic}}}$, for different values of $N_{\mathcal{P}}$ (the number of intervals). For higher dimensions, where closed form is unavailable, such error analysis is impossible. The results of our timing experiments appear in the Sup. Mat., available online. Note that our implementation can be further optimized. e.g., use a C+ + wrapper instead of a Python one. Moreover, most of the computing time is spent on evaluating γ (γ is continuously-defined, so discretely-defined lookup tables will not do); with some tricks, the number of such calls can be drastically reduced.

TABLE 2.

Accuracy: Improvement, in %, of Algorithm 1 (N_steps = 10, n_steps = 10) Over the Generic Solver (100 Steps). Ω = [0, 1] Was Tessellated into $N_{\mathcal{P}}$ Equal-Length Intervals

$N_{\mathcal{P}}$	5	25	45	65	85	105
improvement [%]	2.99	12.56	11.26	9.98	3.29	3.73

5.2 Real-Time Diffeomorphic Image Editing

To highlight the speed at which CPAB transformations are computed, our code includes a GUI where a user chooses velocities at some/all vertices of ${\mathcal{P}}_4$. Using Lemma 3, the Gaussian priors on v^θ, and the fast integration, we compute, in real time, the conditional warp, exp(E(v^θ|user’s choice)). For a demo, please see the video in the Sup. Mat., available online.

5.3 Expressiveness

While simple, CPAB transformations capture a wide range of diffeomorphisms (obviously, like any other work on diffeomorphisms, we make no claims about capturing non-diffeomorphisms such as occlusions). One way to see this is by inspecting samples from p(θ) (Fig. 4). M also contains what is sometimes called large deformations (an overloaded term: used in some texts for elements of G outside its identity component). One way to show a nominal space captures these is a rectangle-to-horseshoe morphing, achieved easily here (see Sup. Mat., available online). Expressiveness is also exemplified by the experiments below (Figs. 7 and 8) and the demo (Section 5.2). As another example, Fig. 9 shows a volume-preserving transformation sampled from the prior applied to a 3D shape. Here, T^θ ∈ M^∂,vp, where $\mathcal{P}$ consists of $N_{\mathcal{P}}=320$ cells (a layout of 4×4×4 cubes, each divided into 5 tetrahedra) and N_v = 125. The dimension, dim(M^∂,vp), is 20 (and not 375 = 3N_v) due to the additional constraints.

Fig. 7.

Landmark-based warping. One example in each row. (a) I_src (and src landmarks). (b) I_src (and src/dst landmarks). (c) The inferred v^θ. (d) A new image is synthesized by warping I_src using T^θ (and dst & T^θ_(src) landmarks). (d) Animation.

Fig. 8.

Nonlinear time warping in real MoCap Data. Left: a reference signal, s₀, 5 other signals, ${\left\{s_i\right\}}_{i=1}^5$, and their mean, $\overline{s}=\frac{1}{5}{\displaystyle {\sum }_{i=1}^5{s}_i}$. The mean is smeared since ${\left\{s_i\right\}}_{i=1}^5$ are misaligned. Center: having inferred the warps, ${\left\{T^{{\theta }_i}\right\}}_{i=1}^5$, we unwarp the signals, setting $u_i=s_i\circ T^{-{\theta }_i}$. Note ${\left\{u_i\right\}}_{i=1}^5$ are better aligned, as is evident by the details preserved in their mean, $\overline{u}=\frac{1}{5}{\displaystyle {\sum }_{i=1}^5{u}_i}$. Right: warping $\overline{u}$ by $T^{\theta +j{\sigma }_1{\xi }_1}$ where $\theta =\frac{1}{5}{\displaystyle {\sum }_{i=1}^5{\theta }_i}$, j ∈ {−6, −4, −2, 0, 2, 4, 6}, ξ₁ is the first principal component of ${\{{\theta }_i-\overline{\theta }\}}_{i=1}^5$, and σ₁ is the corresponding standard deviation.

Fig. 9.

A 3D example. Left: Source shape (taken from the Tosca Dateset: http://tosca.cs.technion.ac.il). Right: Result of applying a volume-preserving transformation, sampled from the prior, to the source.

5.4 Inference: Analysis-by-Synthesis

Real-world signal analysis requires 3 oft-confounded parts: a representation; a probabilistic/deterministic model (e.g., a likelihood); inference. While the first affects the second which in turn affects the third, when judging a new choice for any of these, it is crucial to distinguish between them. For example, as noted in [68], Horn and Schunk’s representation+model of optical flow had been regarded highly inaccurate, till modern inference rendered it almost state-of-the-art. Since the present work is focused on representations, we avoid advocating a particular likelihood model or inference method, and instead use, in a coarse-to-fine manner (recall that if v^θ is CPA w.r.t. $\mathcal{P}$ it is CPA w.r.t. ${\mathrm{P}}^{\prime }\succ \mathcal{P}$), simple choices such as Gaussian likelihood and either conjugate-gradient optimization or the MCMC Metropolis algorithm (where, in the proposal step, we use either global or local moves, utilizing Lemma 3). Without claiming these choices are optimal, these turned out to be effective.

Latent transformations arise in two typical settings: with and without known correspondences. In the first, also called (exact/inexact) diffeomorphic point matching [10], [14], one has a set of landmark pairs, ${\{(x_i,y_i)\}}_{i=1}^{N_{\text{pts}}}$ and seeks T : Ω → Ω such that ${\{T(x_i)\approx y_i\}}_{i=1}^{N_{\text{pts}}}$. We observe that for n = 1, by Corollary 1, our ability to easily solve this problem using CPAB transformations lets us solve a monotonic regression problem, as Fig. 5 demonstrates. Particularly, by adding boundary constraints and simulated annealing, we can well fit a CPAB transformation to any reasonable CDF. More generally, but still for n = 1, our method can be used to solve statistical/optimization problems over monotonic functions, CDFs and (cumulative sums of) histograms/filters while respecting bin ordering. The case of n > 1 appears, e.g., in landmark-based image warping. As shown in Fig. 7, using real images and landmarks from an online hands dataset (http://www.imm.dtu.dk/pubdb/p.php?403), we infer a transformation between different hands in different poses. We then apply it to the whole source image, creating new hands, whose appearance belongs to the source hands, with geometry akin to the destination hands. We also animate the source image by evaluating ϕ^θ for different t’s (including outside of [0, 1]); e.g., Fig. 7e shows animated frames for the example in the second row of Fig. 7a–7d.

In the no-known-correspondences case, there are two subsets of Ω, denoted U_src and U_dst, and two feature maps, $I_{\text{src}}:U_{\text{src}}\to \mathcal{F}$ and $I_{\text{dst}}:U_{\text{dst}}\to \mathcal{F}$, where $\mathcal{F}$ is some feature space (e.g., color), and one seeks T : Ω → Ω such that I_dst ∘ T⁻¹ ≈ I_src (or such that I_src ∘ T ≈ I_dst). This is, e.g., the case for time warping (when n = 1) and landmark-free image registration (n > 1). Fig. 8 (left and middle) shows inferred time warps for motion-capture data from [69]. A similar procedure can be applied to 2D images, in which case it is known as image registration. Fig. 10 shows some results for registrations of images of digits using a pixelwise Gaussian likelihood model and MCMC inference over CPAB transformations. We used 10K iterations for each single registration which took 10 [sec] (i.e., ~ 0.001 [sec] per each MCMC iteration).

Fig. 10.

Example registrations of images from the MNIST dataset.

As an example for statistics on the inferred transformations, Fig. 8 (right) shows a 1D warping of $\overline{u}$ (defined in the figure’s caption) along the 1st eigen warp (computed using PCA on inferred velocity fields); i.e., we use euclidean statistics in the linear $\mathcal{V}$ and map the results to the nonlinear M. For a 2D example in the same spirit, we first performed thousands of within-class pairwise registrations of images of digits as described above. Overall we estimated ~ 30, 000 transformations i.e., ~ 3, 000 per class (class of 1’s, class of 2’s, etc.). Next, we computed PCA on the inferred CPA velocity fields; Fig. 11 shows the first eigen wrap of each class. The Sup. Mat., available online includes a video showing the result of applying the first 3 eigen warps (per class) applied to 100 random images from the training set.

Fig. 11.

The 1st eigen wrap of class. Center row: the mean (identity) transform. Top row: the mean plus 3 standard devidations. Bottom row: the mean minus 3 standard deviations.

5.5 Handling a Large Number of Landmarks

In the hand experiment we showed inference for landmark-based warping using a relatively small set of landmarks on real data. The inference technique used there was MAP estimation using the Metropolis algorithm. Here we show, using synthetic data, that the framework can easily utilize a large set of landmark for inference over the latent transformation between the landmark sets; see Fig. 12. The technique used here is ML estimation using the Metropolis algorithm. It is not surprising that with more data the results improve, suggesting empirical evidence that the estimation procedure is consistent. However, our point here is that most other methods cannot easily utilize a large set of landmarks.

Fig. 12.

Landmark-based Maximum Likelihood (ML) estimates of latent transformations. The dimension of the transformation space in this example is d = 38. Rows differ in the number of landmarks, N_pts. Top row: N_pts = 30. Middle: N_pts = 100. Bottom: N_pts = 10, 000. Colors: green=src, blue=dst, red=T^θ(src), where T^θ is the ML transformation, found by Metropolis’ algorithm. (a) original grid lines + src landmarks; (b) grid lines deformed by the Ground-Truth (GT) transformation + dst landmarks. (c) grid lines deformed by the GT transformation. (d) grid lines deformed by the ML transformation. (e) grid lines deformed by the GT transformation (blue) and grid lines deformed by the ML transformation (red). (f) dst landmarks (blue) and the src landmarks transformed by the ML transformation (red).

5.6 Inference over a CPA Field Piecewise w.r.t. t

As a proof-of-concept for the utility of Algorithm 2, we used it to “prettify” the output of a dense-correspondence tool. We extracted dense correspondences, ${\{(x_i,y_i)\}}_{i=1}^{N_{\text{pts}}}$, between two images using SIFT-Flow [20]. Then, we discarded the images and treated the correspondences as noisy observations fed as input to Algorithm 2. Finally, for visualization purposes, we used the output of Algorithm 2 to warp the source image. Fig. 14 shows a typical result.

Fig. 14.

“Prettifying” the result of a dense-correspondence tool. From left to right: I_src; I_dst; warping I_src according to SIFT-Flow [20]; warping I_src using the output of Algorithm 2 where the SIFT-Flow was the input.

5.7 Representing a CDF Using a CPAB Transformation

Fig. 13 shows results (using Metropolis’ algorithm) for fitting F^θ ≜ T^θ ∘ F₀, where F₀ is the CDF of the uniform distribution on J = [−7, 7], to a 3-component Gaussian mixture. Note F^θ well approximates the target CDF.

Fig. 13.

CDF Representation. The source is the CDF of the uniform distribution. The target is the CDF of a 3-component Gaussian mixture. The setting and the inference procedure, where we used a Maximum Likelihood (ML) solution obtained using MCMC, are virtually the same as in the monotonic-regression experiment, except here we also used simulated annealing, gradually reducing the standard deviation of the “noise” to zero. Typical to an ML solution, the inferred v^θ is non-smooth. MAP estimation (not shown) leads to a smoother field, analogously to the results in Fig. 5.

6 CONCLUSION

We proposed new classes of diffeomorphisms that are simple, expressive, efficient, and have many applications. We showed these objects are evaluated fast and with high accuracy (exact in 1D) and that they are highly expressive. While we required Ω to be a hyperrectangle it is not hard to see that, more generally, Ω may be any finite intersection of closed half spaces of ℝⁿ. Like the space in [27], our spaces lack closure under composition; but since $T^{{\theta }_2}\circ T^{{\theta }_1}\in G$, our representation can still be used within frameworks that apply consecutive transformations [16]. In fact, Algorithm 2 deals exactly with such scenarios. While this work focused on representation it was primarily motivated by modeling-and-inference considerations: e.g., the ability to quickly run MCMC inference over diffeomorphisms is a hallmark of the method, showing that simplicity pays off.

Biographies

Oren Freifeld received the BSc and MSc degrees in biomedical engineering from Tel-Aviv University, in 2005 and 2007, respectively. He received the ScM and PhD degrees in applied mathematics from Brown University, in 2009 and 2013, respectively. Later, he was a postdoc with MIT Computer Science & Artificial Intelligence Laboratory. He is currently an assistant professor in the Department of Computer Science, Ben-Gurion University. His main fields of research are computer vision, statistical inference, and machine learning.

Søren Hauberg received the PhD degree in computer science from the University of Copenhagen, in 2011. He has been a visiting scholar with UC Berkeley (2010), and a post doc in the Perceiving Systems Department, Max Planck Institute for Intelligent Systems (2012–2014). He is currently a post doc in the Section for Cognitive Systems, Technical University of Denmark. He works in the span of geometry and statistics.

Kayhan Batmanghelich received the PhD degree in electrical and system engineering from the University of Pennsylvania, in 2012. He is currently a postdoctoral associate with MIT Computer Science & Artificial Intelligence Laboratory. He works in the span of machine learning, medical imaging, and computational biology.

John W. Fisher III received a PhD degree in electrical and computer engineering from the University of Florida, Gainesville, Florida in 1997. He is a Senior Research Scientist in the Computer Science and Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. Prior to joining the Massachusetts Institute of Technology he was a member of the research staff with the Electronic Communications Laboratory at the University of Florida from 1987 to 1997, during which time he conducted research in the areas of ultra-wideband radar for ground and foliage penetration applications, radar signal processing, and automatic target recognition algorithms. His current area of research focus includes information theoretic approaches to signal processing, multi-modal data fusion, machine learning and computer vision.