Deep Learning of Warping Functions for Shape Analysis

Abstract

Rate-invariant or reparameterization-invariant matching between functions and shapes of curves, respectively, is an important problem in computer vision and medical imaging. Often, the computational cost of matching using approaches such as dynamic time warping or dynamic programming is prohibitive for large datasets. Here, we propose a deep neural-network-based approach for learning the warping functions from training data consisting of a large number of optimal matches, and use it to predict optimal diffeomorphic warping functions. Results show prediction performance on a synthetic dataset of bump functions and two-dimensional curves from the ETH-80 dataset as well as a significant reduction in computational cost.

1. Introduction

A key ingredient in rate or parameterization-invariant matching of shapes of one-dimensional functions or curves is a cost function (either an L₁ or an L₂ norm) that includes a nonlinear one-one differentiable function with a differentiable inverse (diffeomorphism) that controls the rate of change of one function (shape) with respect to the other. This transformation can be conveniently described by a group action if an appropriate representation or mapping of shapes or functions is defined, or by a direct non-linear transformation of the coordinates of the curves or graphs of functions. In both cases, the transformation results in a composition of a shape with a diffeomorphic warping function. While the full problem of shape analysis for curves also involves an invariant matching over a set of translations, scalings, and rotations, the main computation for both curves and one-dimensional functions requires an optimization over a set of diffeomorphic warping functions. There have been several algorithms proposed to solve this optimization problem. A classical, widely-used algorithm is dynamic time warping (DTW), which is amenable to a global solution using dynamic programming [13]. This algorithm has a computational complexity of O(T²) for T samples or points along the shape. In this paper, instead of solving the dynamic programming (DP) problem explicitly, we outline a deep learning (DL) framework for predicting the warping functions by learning from pairwise matches from a training dataset.

1.1. Motivation

Our motivation for learning to predict reparameterizations is multifold.

Computational cost:

Direct computation of alignment, either exclusively using dynamic time warping or by optimizing a cost function to find geodesics as in [3, 4, 14], incurs a computational cost. This cost is negligible for small-sized functional or curve-shape datasets with sizes on the order of a few thousands. However, for an order increase of tenfold, hundredfold, or larger (data sizes > 100K or > 1 million), the computational cost becomes prohibitive. In real-world applications, such data arise from biological shapes [2, 5], clinical time-series data of heart rates [11], or functional magnetic resonance imaging signals (fMRI) [8]. Thus, we suggest that a prediction or an evaluation-type of an approach has the potential to yield sizable cost savings as compared to a direct computation-type of an approach. In spirit, this motivation is similar to the idea proposed by Yang et al. who predict the momentum-parameterization of diffeomorphisms using a deep learning approach to achieve a speedup in image registration [17].

Shape learning and classification:

As proposed by Thompson [15], the task of comparing the deformations of the objects, rather than a precise definition of the object itself, often yields more interesting information. The temporal warps or spatial reparameterizations (diffeomorphisms) can be analyzed under a statistical learning framework including tangent-space principal component analysis or linear discriminant analysis. Under a deep-learning framework for warping functions, one would not only have the predicted warping function, but also the weights associated with the underlying model training from the population. Thus the goal of shape learning or shape understanding may potentially benefit from having more information about the population. We also note that Lohit et al. [10] aims to achieve this by jointly learning the time warping functions associated with temporal human activity. There, the approach is classification-focused and is trained to give the best performance with respect to classification.

Novel descriptors:

Finally, the output of the intermediate layers can serve as a rich feature descriptor for the shape of the population. This can be achieved by following the process of feature extraction and obtaining the projection of a novel shape as a mapping of the intermediate layers of the neural network, or by modeling the ensembles of the activation layer for a population level shape descriptor.

1.2. Background and related work

Several ideas have been proposed for learning-based prediction of the warping functions. Here, we outline them in brief. Kazlauskaite et al. proposed a method for automatically learning functional alignments based on a probabilistic model built on non-parametric priors [6]. Another idea, deep canonical time warping (DCTW) for learning warping functions (for multiple time-series) uses deep learning to achieve simultaneous temporal alignment and maximal correlation of time-series in a common subspace [16]. The work by Oh et al. [12] uses a sequence transformer network to learn linear transformations along the temporal axis (stretch, compress, flip and/or shift the signal) to identify and account for invariances in clinical time-series data. A more recent approach by Lohit et al. [10] uses a temporal transformer network (TTN) to learn warping functions in the context of classification. Their network performs learning as well as class-aware discriminative alignment jointly for time-series classification including action trajectories by reducing the intra-class variability and also increasing the inter-class separation. Notably, in their framework the prediction of the warping functions is achieved without an explicit template for the class.

1.3. Contributions

The contributions of this paper are as follows. We propose a deep learning based framework for predicting diffeomorphic warps giving rise to invariant matching of one-dimensional functions and two-dimensional curves. The network architecture is simple and consists of a convolutional layer followed by three dense layers. We propose a choice of three loss functions that measure discrepancies between i) warping functions, ii) linear combination of the coordinates (graphs in case of functions) and warping functions, and a iii) linear combination of the shape representations and warping functions, and demonstrate that the latter yields the best performance. We train the network on warping functions obtained using dynamic programming and show prediction results for a large set (∼500K) of data for synthetic bump functions and two-dimensional curves from the ETH-80 dataset [9].

Our architecture differs from [10] in the following ways: i) we permit negative inputs into the fully connected layers by using leaky reLU activation functions with parameter 0.1, ii) we train our network directly on the shapes and warping functions to minimize loss functions that penalize shape matching differences as opposed to minimizing loss functions that penalize classification errors, and iii) we do not enforce a positive monotonicity in the network output and instead try to learn this constraint.

Additionally, different from the approach in [10], we achieve reparameterization-invariant matching for pairs of curves by integrating loss functions that aim to find the shortest distance between points in a Hilbert sphere.

This paper is organized as follows. Section 2 outlines the preliminaries for the shape representation and matching problem. Section 3 outlines the deep learning architecture including the choice of loss functions, followed by results in section 4 and discussion in section 5.

2. Shape representation preliminaries

Throughout this paper, we will consider a parameterized representation for two-dimensional curves and functions. We will let p denote a parameterized curve such that $p:D\equiv [0,2\pi ]\to {ℝ}^2.$ In this paper we will only consider those ps that are differentiable and their first derivative is in $L^2(D,{ℝ}^n).$ For a one-dimensional function, we assume that $p:D\to ℝ$ with a slight abuse of notation. Though the following discussion is for two-dimensional curves, the theory holds for one-dimensional functions. For the purpose of studying the shape of p, we will represent it using the square-root velocity function (SRVF) [3, 4, 14] defined as $q:D\to {ℝ}^n,$ where $q(t)=\dot{p}(t)/\sqrt{\dot{\Vert p(t)\Vert }}.$ We note that for every $q\in L^2(D,{ℝ}^n)$ there exists a curve p (unique up to a translation) such that the given q is the SRVF function of that p. This curve is recoverable using the equation: $p(t)={\int }_0^{t}q(s)\Vert q(s)\Vert ds.$

To achieve invariance to scale, we re-scale all curves to be of length 2π. This causes the SRVF functional representation for these curves to be identified as elements of a hypersphere in the Hilbert manifold $L^2(D,{ℝ}^2).$ In this paper, we will use the notation ${\mathcal{C}}^o$ to denote this hypersphere.

We impose a standard L² metric on the tangent space of this hypersphere as follows. Since ${\mathcal{C}}^o$ is a sphere in $L^2([0,2\pi ],{ℝ}^n),$ its tangent space at a point q is given by: $T_q({\mathcal{C}}^o)=\{v\in L^2([0,2\pi ],{ℝ}^n)\mid \langle v,q\rangle =0\}.$ Here, $\langle v,q\rangle$ denotes the inner product in $L^2([0,2\pi ],{ℝ}^n):\langle v,q\rangle ={\int }_0^{2\pi }(v(t)\cdot q(t){)}_{{ℝ}^n}dt.$ In this paper, we deal with one-dimensional functions (n = 1) and two-dimensional curves (n = 2). This standard metric on $L^2([0,2\pi ],{ℝ}^n)$ restricts to one on ${\mathcal{C}}^o$ and is used to compute geodesics between shapes.

Representing a parameterized curve p(t) by its SRVF function q(t), and imposing the constraint ${\int }_D\langle q(t),q(t)\rangle dt=2\pi ,$ makes it invariant to translation and scaling. Further, the rotation and the reparameterization variability is accounted for as follows.

For two-dimensional curves, a rotation is an element of SO(2), the special orthogonal group of 2 × 2 matrices, and a re-parameterization is an element of Γ, the set of all orientation-preserving diffeomorphisms of D. The rotation and re-parameterization of a curve p are both denoted by the actions of SO(2) and Γ on its SRVF. While the action of SO(2) is denoted by multiplication: SO(n) × ${\mathcal{C}}^o$ → ${\mathcal{C}}^o$, $(O,q(t))=Oq(t),$ the action of Γ is derived as follows. For a γ ∈ Γ, the composition p ◦ γ denotes its re-parameterization; the SRVF of the re-parameterized curve is given by $q(\gamma (t))\sqrt{\dot{\gamma (t)}},$ where q is the SRVF of p. This gives us the action $\mathrm{\Gamma }\times {\mathcal{C}}^o\to {\mathcal{C}}^o,(q,\gamma )=(q◦\gamma )\sqrt{\dot{\gamma }}.$

Next, we enable comparisons between functions by computing the shortest path between them. Since the space $\mathcal{Q}$ is a Hilbert sphere, the shortest path between two points (shapes) q₁ and q₂ can be expressed analytically as,

$${\chi }_t(q_1;v)=\cos (t{\cos }^{-1}\langle q_1,q_2\rangle )q_1+\sin (t{\cos }^{-1}\langle q_1,q_2\rangle )v,$$ (1)

where t ∈ [0, 2π] and the initial tangent vector $v\in T_{q_1}(\mathcal{Q})$ is given by $v=q_2-\langle q_1,q_2\rangle q_1.$ Then the shortest distance between the two shapes q₁ and q₂ in $\mathcal{Q}$ is given by

$$d\left(q_1,q_2\right)={\int }_0^{2\pi }\sqrt{\langle {\dot{\chi }}_t,{\dot{\chi }}_t\rangle }dt.$$ (2)

The distance in Eqn. 2 can be made invariant by searching over all reparaemterizations γ as

$$d_{\gamma }\left(q_1,q_2\right)=\underset{\gamma }{\min }d(q_1,\sqrt{\dot{\gamma }}{q}_2◦\gamma ).$$ (3)

We use dynamic programming to minimize Eqn. 3 to find the optimal reparameterization γ_DP as the minimizer ${\gamma }_{DP}=\underset{\gamma }{\arg \min }d(q_1,\sqrt{\dot{\gamma }}{q}_2◦\gamma ).$

In the following discussion with a slight abuse of notation, we refer to the γ_DP obtained using dynamic programming as γ without the subscript DP and will refer to the predicted warping function using deep learning as $\widehat{\gamma }.$ Next, we outline the framework for learning this warping function by considering a training dataset of pairwise matchings.

3. Deep Learning Architecture

In an effort to minimize training time, we constructed our network to be relatively simple with sufficient complexity to be applicable to varying datasets. Indeed, we use the same architecture across various datasets as discussed further in section 4.

Drawing inspiration from TTN [10], our network also consists of a convolutional layer. However, this is then followed by three dense layers as illustrated in Figure 1.

Figure 1.

Deep neural architecture of the prediction network.

As presented, this network operates on two-dimensional curves; that is, given $p_1:D\to {ℝ}^2$ and $p_2:D\to {ℝ}^2,$ the network aims to find the optimal diffeomorphism mapping p₂ to p₁. The curves p₁ and p₂ are first downsampled by selecting T evenly spaced points in the interval [0, 2π] and then evaluating the curves at these points. As such, p₁ and p₂ can be regarded as matrices each of size T × 2. With a slight abuse of notation, we refer to these matrix representations of p₁ and p₂ as p₁ and p₂. Concatenating the two matrices yields $[p_1,p_2]\in {ℝ}^{T\times 4}.$ This is what is then fed through the network.

The input is first passed through a convolutional layer with 32 filters of size 3 × 3 and a unit stride. The output of this layer is then fed through the standard reLU activation function. After flattening this output, it is then fed through two successive fully-connected layers of sizes 256 and 128, respectively. Both of these layers have a leaky reLU activation function with parameter 0.1 and both are followed by dropout layers with a drop probability of 0.25. The output layer has T neurons and, to restrict the output to be in D, the activation function is given by $\mathrm{\Omega }(x)=2\pi \sigma (x)$ where $\sigma (x)=\frac{1}{1+e^{-x}}$ is the sigmoid function.

For $p_1,p_2:D\to ℝ,$ the network remains the same except the convolution layer now has filters of size 2 × 2 and the input to the network is $\left[p_1,p_2\right]\in {ℝ}^{T\times 2}.$

In contrast to [10], we do not enforce a non-decreasing output but rather try to learn it from the data. Moreover, our network aims to reproduce warps obtained from DTW rather than warps that yield the best classification results.

3.1. Choice of loss functions

Given discretized curves p₁ and p₂, our network outputs $\widehat{\gamma },$ which is an estimate of the γ obtained by solving

$$\gamma =\underset{\overline{\gamma }}{\arg \min }{\Vert q_1-\sqrt{\dot{\overline{\gamma }}}(q_2◦\overline{\gamma })\Vert }_2^2$$ (4)

where q₁ and q₂ are the SRVF representations of p₁ and p₂, respectively.

Since we desire $\widehat{\gamma }$ to be as close to γ as possible, it is natural to consider their squared ${\mathcal{l}}_2$ difference as a measure of similarity. As such, one possible loss function is

$${\mathcal{L}}_1(\gamma ,\widehat{\gamma })=\Vert \gamma -\widehat{\gamma }{\Vert }_2^2.$$ (5)

After warping p₂, we expect the distance between p₁ and this warped p₂ to be smaller than that of p₁ and p₂. As such, we can also aim to minimize ${\Vert p_1-p_2◦\widehat{\gamma }\Vert }_2^2$ with respect to $\widehat{\gamma }.$ However, this loss does not make use of the true value γ, rendering the use of deep learning moot. As such, we can simply consider a linear combination of this and ${\mathcal{L}}_1$:

$${\mathcal{L}}_2(\gamma ,\widehat{\gamma })=a{\Vert p_1-p_2◦\widehat{\gamma }\Vert }_2^2+b\Vert \gamma -\widehat{\gamma }{\Vert }_2^2.$$ (6)

where $a,b\in {ℝ}_{>0}.$

Since the true γ is obtained by solving (4), it makes sense to include this function in our loss. As such, the final loss function we consider is again a linear combination of this function and ${\mathcal{L}}_1$

$${\mathcal{L}}_3(\gamma ,\widehat{\gamma })=a{\Vert q_1-\sqrt{\dot{\widehat{\gamma }}}\left(q_2◦\widehat{\gamma }\right)\Vert }_2^2+b\Vert \gamma -\widehat{\gamma }{\Vert }_2^2.$$ (7)

Because we do not impose a nonnegativity constraint on $\dot{\widehat{\gamma }},$ instead of considering loss (7), we will consider the equivalent (8) for numerical stability

$${\mathcal{L}}_3(\gamma ,\widehat{\gamma })=a\Vert q_1-\frac{\left({\dot{p}}_2◦\widehat{\gamma }\right)\dot{\widehat{\gamma }}}{\sqrt{|({\dot{p}}_2◦\widehat{\gamma })\dot{\widehat{\gamma }}|}}\Vert +b\Vert \gamma -\widehat{\gamma }{\Vert }_2^2.$$ (8)

4. Results

4.1. Data Generation

We consider two separate applications of time warping: to one and two-dimensional curves. One-dimensional curves were synthesized by appending sinusoidal waves of varying phase and amplitudes. We refer to these curves as bumps and characterize them by their number of peaks. Examples of one, two, three, and four-bump curves is given in figure 2.

Figure 2.

Examples of synthesized bumps.

Each curve was discretized to T = 300 points. Amplitudes were sampled uniformly on the interval [0.15, 3] and wave lengths were chosen to be a percentage of T chosen uniformly on the interval [5,10].

Two-dimensional curves were constructed by first modifying the contours in the ETH-80 [9] dataset. In particular, each contour was manually inspected and modified so that the curve did not exhibit any holes. Figure 3 illustrates this process on an example of a cow contour from the dataset. This process yields closed curves in ${ℝ}^2$ with no holes.

Figure 3.

Hole removal from ETH-80 contour dataset.

Since the dataset consists of PNG images, we first binarize the image and then apply the Moore-Neighbor tracing algorithm to extract the curve boundary. This process gives a set of points in ${ℝ}^2$. Using linear interpolation, the curve is downsampled to T = 300 points so that the curve is an element of ${ℝ}^{300\times 2}.$ This process is applied to each of the modified ETH-80 curves, yielding 3280 curves belonging to one of eight evenly distributed classes: apple, car, cow, cup, dog, horse, pear, and tomato. Within each class, each curve is enumerated according to its original file name; afterwards, every possible image pair (i, j), j > i is considered. Since each class has 410 curves, there are $\frac{410*409}{2}=83845$ such pairs. Across all 8 classes, this gives a total of 670760 curve pairs.

For each curve pair (i, j), the optimal diffeomorphism that warps curve j to curve i was computed using DTW. With the curves and diffeomorphisms at hand, the data is split into training/validation/testing in a 70/10/20 split, respectively, for a total of 469532 training pairs.

In the one-dimensional case, for each bump curve we generate 650000 random pairs so as to be consistent with the size of the two-dimensional dataset. The optimal alignment diffeomorphism is then found using DTW for each pair and the same training/validation/test split is applied. This is repeated for one, two, three, and four-bump curves so that we have 450000 training pairs for each bump class.

4.2. Loss function performance

Using the one-bump dataset, we trained the network using the ${\mathcal{L}}_1$, ${\mathcal{L}}_2$ and ${\mathcal{L}}_3$ losses and used a = 1 and $b=\frac{1}{4}$ for ${\mathcal{L}}_2$ and ${\mathcal{L}}_3$. Figure 4 depicts the average value of the ${\mathcal{L}}_1$ loss on the validation data when trained using each loss. We see that the ${\mathcal{L}}_3$ loss gives outputs that are most similar, in the ${\mathcal{l}}_2$ sense, to the desired output. Consequently, we use the ${\mathcal{L}}_3$ loss to train our network on the remaining bump datasets. The ${\mathcal{L}}_3$ loss also gives the best performance on the two-dimensional dataset as depicted in figure 5.

Figure 4.

Average ${\mathcal{L}}_1$ loss on 1-d validation data.

Figure 5.

Average ${\mathcal{L}}_1$ loss on 2-d validation data.

4.3. Model performance

Figure 6 illustrates the performance of our network trained on one, two, three, and four-bump datasets separately using the ${\mathcal{L}}_3$ loss

Figure 6.

Performance for the one, two, three, and four-bump cases. Top: Warping functions from DP (γ) and DL $(\widehat{\gamma })$. Matching by an identity warp (2^nd row), DP γ (3^rd row), and deep learning $\widehat{\gamma }$ (4^th row).

The first row plots γ (in green), $\widehat{\gamma }$ (in red) and γ_I (in blue) where γ_I(t) = t. The second row is a correspondence plot between p₁ and p₂ as determined by γ_I. The third row is a correspondence plot between p₁ and p₂ as determined by γ and the fourth row is a correspondence plot between p₁ and p₂ as determined by $\widehat{\gamma }$.

Similarly, figure 7 illustrates the performance of our network when trained on the full 2d-dataset consisting of all classes.

Figure 7.

Performance for 2D curves. Top: Warping functions from DP (γ) and DL $(\widehat{\gamma })$. Matching by an identity warp (2^nd row), DP γ (3^rd row), and deep learning $\widehat{\gamma }$ (4^th row).

Figures 8 and 9 show the average error ${\mathcal{L}}_1$ for the one-dimensional (bump) data and the two-dimensional curves for different bump and curve types. It is observed that in the case of bumps, the lowest error is obtained for the one-bump case, whereas the highest error is obtained for the three-bump case, with the two and three-bump cases giving similar errors. For curves, rotund shapes such as apples, cups, pears, and tomatoes yielded lower errors compared to shapes with articulated features such as cows, dogs, and horses.

Figure 8.

Average ${\mathcal{L}}_1$ loss on test data for all bumps.

Figure 9.

Average ${\mathcal{L}}_1$ loss on test data for shapes.

4.4. Implementation and computational cost

The network was trained with a batch size of 32 for 200 epochs using an Adam optimizer [7] with a learning rate of 0.001, and exponential decay rates of 0.9 and 0.999 for the first and second moment estimates, respectively.

Table 1 shows comparisons of computational costs between the dynamic programming approach and the deep learning prediction approach. All experiments were performed on an Intel i7-7700K CPU @ 4.20GHz. The machine was equipped with 2 TITAN Xp GPUs for deep learning. The network was implemented and trained using TensorFlow [1]. The dynamic programming was executed on the same machine. It is observed that the deep learning warping prediction approach was approximately 3000 times faster for one-dimensional functions (bumps) and 900 times faster for two dimensional curves.

Table 1.

Time (in seconds) to obtain 100,000 warps on an i7-7700K CPU @ 4.20GHz (dynamic programming) and two NVIDIA GP102 TITAN Xp GPUs (deep learning).

	Dynamic Programming	Deep Learning
Bumps	45130.011	14.047
Shapes	5519.849	5.639

5. Discussion

We presented a deep learning approach for predicting warping functions that achieve rate-invariant alignment in the case of functions and reparameterization-invariant matching for two-dimensional curves. While we listed shape learning and novel shape representation as potential applications, in this paper, our primary motivation was demonstrating reduced computational cost. The network architecture was simple to construct and has similarities with the approaches in [10] and [12]. We experimented with three loss functions, the first of which only plenalized the cost between warping functions. The second penalized the cost between the coordinates of curves and functions, and the third penalized the cost between their SRVFs; both of these penalized the cost between warping functions as well. We showed that the latter yielded the best results. While, visually, the predicted warping, and consequently the ensuing matching, appear close to each other, we also observed cases where the predicted function failed to achieve an optimal warping. We also noted that, occasionally, the dynamic programming algorithm partially failed to achieve a good match.

In the one-dimensional case, figure 6 suggests our network is able to perform reasonably well in aligning curves when the curves are relatively close to one another. Figure 10 (a) is an example where the alignment of the curves would require significant stretching and we see that both the DP and DL solutions fail to achieve this. Because the training data is derived from the DP warps, performance of our model must be measured relative to the DP performance.

Figure 10.

(a) A two-bump example where both DP and DL fail to match bumps. (b) A four-bump example where DP successfully matches all bumps but DL fails to match the third.

While our framework offers similar performance to DP in relatively simple curves, its architectural simplicity limits its flexibility and performance on difficult curves. For example, figure 10 (b) depicts a curve where DP succeeds but DL fails to match every bump correctly. Future work can aim to enforce positive monotonicity in the predicted warps as in [10]. However, more complicated architectures should be explored. In particular, the convolution filter size should be examined and chosen so as to span the dimension of the curve and be large enough to capture variations in the curve throughout its entire domain.

When trained on the two-dimensional shape data, we see that the performance is comparable to DP for simple shapes. In figure 7 we see that DL is able to perform alignments similar to DP in both the cup and pear shapes. For more complicated shapes like dogs and horses, detailed artifacts such as legs, tails, and ears can be matched reasonably well but performance is not as strong as in the DP case. This, again, may be attributed to the simplicity of the network and may possibly improve under a more complex network.