Non-Rigid Point Set Registration by Preserving Global and Local Structures

Abstract

In previous work on point registration, the input point sets are often represented using Gaussian mixture models and the registration is then addressed through a probabilistic approach, which aims to exploit global relationships on the point sets. For non-rigid shapes, however, the local structures among neighboring points are also strong and stable and thus helpful in recovering the point correspondence. In this paper, we formulate point registration as the estimation of a mixture of densities, where local features, such as shape context, are used to assign the membership probabilities of the mixture model. This enables us to preserve both global and local structures during matching. The transformation between the two point sets is specified in a reproducing kernel Hilbert space and a sparse approximation is adopted to achieve a fast implementation. Extensive experiments on both synthesized and real data show the robustness of our approach under various types of distortions such as deformation, noise, outliers, rotation and occlusion. It greatly outperforms state-of-the-art methods, especially when the data is badly degraded.

I. Introduction

Finding an optimal alignment between two sets of points is a fundamental problem in computer vision, image analysis and pattern recognition [1], [2], [3], [4]. Many tasks in these fields – such as stereo matching, image registration and shape recognition – can be formulated as a point set registration problem, which requires addressing two issues: find point-to-point correspondences and estimate the underlying spatial transformation which best aligns the two sets [5], [6], [7].

The registration problem can be roughly categorized into rigid or non-rigid registration, depending on the application and the form of the data. With a small number of transformation parameters, rigid registration is relatively easy and has been widely studied [1], [2], [8], [9], [10]. By contrast, non-rigid registration is more difficult because the true underlying non-rigid transformations are often unknown and modeling them is a challenging task [3]. But non-rigid registration is very important because it is required for many real world tasks, including shape recognition, deformable motion tracking and medical image registration.

Existing non-rigid registration methods typically consider the alignment of two point sets as a probability density estimation problem and more specifically, they use Gaussian Mixture Models (GMMs) [11], [12]. This makes the reasonable assumption that points from one set are normally distributed around points belonging to the other set. Hence, the point-to-point assignment problem can be recast into that of estimating the parameters of a mixture distribution. In general, these probabilistic methods are able to exploit global relationships in the point sets, since the rough structure of a point set is typically preserved; otherwise, even people cannot find correspondences reliably under arbitrarily large deformations. However, even for non-rigid shapes, the local structures among neighboring points are strong and stable. For example, most neighboring points on a non-rigid shape (e.g., a human face) cannot move independently under deformation due to physical constraints [13], [14]. Furthermore, preserving local neighborhood structures is also very important for people to detect and recognize shapes efficiently and reliably [15]. As a major contribution of this paper, we propose a uniform framework, which exploits both the global and the local structures in the point sets.

More precisely, we formulate point registration as the estimation of a mixture of densities: A GMM is fitted to one point set, such that the centers of the Gaussian densities are constrained to coincide with the other point set. Meanwhile, we use local features such as shape context to assign the membership probabilities of the mixture model, so that both global and local structures can be preserved during the matching process. The non-rigid transformation is modeled in a functional space, called the reproducing kernel Hilbert space (RKHS) [16], in which the transformation function has an explicit kernel representation.

The rest of the paper is organized as follows. Section II describes background material and related work. In Section III, we present the proposed non-rigid registration algorithm which is able to preserve both global and local structures among the point sets. Section IV illustrates our algorithm on some synthetic registration tasks on both 2D and 3D datasets with comparisons to other approaches, and then tests it for feature matching on real images involving non-rigid deformations. Finally, we conclude this paper in Section V.

II. Related Work

The iterated closest point (ICP) algorithm [2] is one of the best known point registration approaches. It uses nearest-neighbor relationships to assign a binary correspondence and then uses the estimated correspondences to refine the transformation. Efficient versions of ICP use sampling processes, either deterministic or based on heuristics [8]. The nearest point strategy of ICP can be replaced by soft assignments within a continuous optimization framework, e.g., the TPS-RPM [17], [3]. In the recent past, the point registration is typically solved by probabilistic methods [18], [19], [11], [12]. The Kernel Correlation Based Method [18] models each one of the two point sets as two probability distributions and measures the dissimilarity between the two distributions. It was later improved in [19], which represents the point sets using GMMs. In [11] and [12], the GMM is used to recast the point-to-point assignment problem into that of estimating the parameters of a mixture. This is done within the framework of maximum likelihood and the Expectation-Maximization (EM) algorithm [20]. In addition, point registration for articulated shapes has also been investigated using similar methods and shows promising results [21], [12], [22]. The methods mentioned above mainly focus on exploiting the global relationships in the point sets.

To preserve local neighborhood structures, Belongie et al. [23] introduced a registration method based on the shape context descriptor, which incorporates the neighborhood structure of the point set, and helps to establish better point correspondences. Using shape context as a lever, Zheng and Doermann [13] proposed a matching method, called the Robust Point Matching-preserving Local Neighborhood Structures (RPM-LNS) algorithm, which was later improved in [24]. Tu and Yuille [25] presented a shape matching method, based on a generative model and informative features. Wasserman et al. used local structure information to achieve point cloud registration for registering medical images [26], [27]. Recently, Ma et al. introduced Robust Point Matching methods based on local features and a robust L₂E estimator (RPM-L₂E) [28], [29] or Vector Field Consensus [30], [31], which can exploit both global and local structures. However, they find the correspondences and the transformation separately. In this paper, we propose a method which estimates these two variables jointly under the GMM formulation.

III. Method

We start by introducing a general methodology for non-rigid point set registration using Gaussian mixture models. We next present the layout of our model. We then explain the way we search for an optimal solution, followed by a description of our fast implementation. We also state the differences between our algorithm and several related state-of-the-art algorithms in detail. Finally, we provide the implementation details.

A. Point Registration and Gaussian Mixtures

We denote by X = (x₁, ⋯, x_N) ∈ IR^D×N the coordinates of a set of model points and by Y = (y₁, ⋯, y_M) ∈ IR^D×M the coordinates of a set of observed data points, where each point is represented as a D × 1 dimensional column vector. The model points undergo a non-rigid transformation 𝒯 : IR^D → IR^D, and the goal is to estimate 𝒯, which warps the model points to the observed data points, so that the two point sets become aligned. In this paper, we will formulate the point registration as the estimation of a mixture of densities, where a Gaussian mixture model (GMM) is fitted to the observed data points Y, such that the GMM centroids of the Gaussian densities are constrained to coincide with the transformed model points 𝒯(X) [11], [19], [12].

Let us introduce a set of latent variables 𝒵 = {z_m ∈ IN_N+1 : m ∈ IN_M}, where each variable z_m assigns an observed data point y_m to a GMM centroid 𝒯 (x_n), if z_m = n, 1 ≤ n ≤ N, or to an additional outlier class, if z_m = N + 1. The GMM probability density function then can be defined as

$$P({\mathbf{y}}_m)=\sum _{n=1}^{N+1}P(z_m=n)P({\mathbf{y}}_m|z_m=n).$$ (1)

Typically, for point registration, we can use equal isotropic covariances σ²I for all GMM components and the outlier distribution is supposed to be uniform $\frac{1}{a}$. Note that a more general assumption of anisotropic Gaussian noise in the data, which uses the full Gaussian model, has also been investigated in [12]. We denote by θ = {𝒯, σ², γ} the set of unknown parameters, where γ ∈ [0, 1] is the percentage of outliers. Let π_mn be the membership probability of the GMM, such that ${\sum }_{n=1}^N{\pi }_{\mathit{\text{mn}}}=1$. The mixture model then takes the form

$$P({\mathbf{y}}_m|\mathit{\theta })=\gamma \frac{1}{a}+(1-\gamma )\sum _{n=1}^N\frac{{\pi }_{\mathit{\text{mn}}}}{{(2\pi {\sigma }^2)}^{D/2}}{e}^{-\frac{{\Vert {\mathbf{y}}_m-𝒯({\mathbf{x}}_n)\Vert }^2}{2{\sigma }^2}}.$$ (2)

We want to recover the non-rigid transformation 𝒯. The slow-and-smooth model [32], which has been shown to account for a range of motion phenomena, suggests that the prior of 𝒯 has the form: $P(𝒯)\propto e^{-\frac{\lambda }{2}\varphi (𝒯)}$, where ϕ(𝒯) is a smoothness functional and λ is a positive real number (we will discuss the details of ϕ(𝒯) later). Using Bayes rule, we estimate a MAP solution of θ, i.e., θ* = arg max_θ P(θ|Y) = arg max_θ P(Y|θ)P(𝒯). This is equivalent to minimizing the negative log posterior

$$ℒ(\mathit{\theta }|\mathbf{Y})=-\sum _{m=1}^M\text{ln }P({\mathbf{y}}_m|\mathit{\theta })-\text{ln }P(𝒯),$$ (3)

where we make the i.i.d. data assumption. The transformation 𝒯 will be obtained from the optimal solution θ*.

B. Registration Using Global and Local Structures

The above method treats the point set as an instance of a GMM distribution, which aims to exploit the global structures in the point sets. For non-rigid point set registration, the local structures among neighboring points are also very strong and stable and can be used to improve the point correspondences. Next, we show how to use the local structures.

Consider the mixture model in Eq. (2), where we need to assign the membership probability π_mn that the observed data point y_m belongs to the GMM centroid 𝒯 (x_n). It is a prior and is typically assumed to be equal for all GMM components [11], [12], i.e., ${\pi }_{\mathit{\text{mn}}}=\frac{1}{N}$, ∀m ∈ IN_M, n ∈ IN_N. In this paper, we initialize π_mn by incorporating the local neighborhood structure information. More specifically, we compute a local feature descriptor for each point [23], [33], and then initialize π_mn based on the matching of the feature descriptors of the two sets.

We use shape context [23] as the feature descriptor in the 2D case, using the Hungarian method for matching, with the χ² test statistic as the cost measure. In the 3D case, the Fast Point Feature Histograms (FPFH) [34] can be used as a feature descriptor and the matching is performed by a Sample Consensus Initial Alignment method. After we obtain the correspondences between the two point sets by matching their feature descriptors, we use them to initialize π_mn according to the following two rules:

1. For a data point y_m, we denote by ℐ the set of model points, which it corresponds to¹. We let

   $${\pi }_{\mathit{\text{mn}}}=\{\begin{array}{cc}\hfill \frac{𝒯}{|ℐ|},\hfill & \hfill \forall {\mathbf{x}}_n\in ℐ,\hfill \\ \hfill \frac{1-\tau }{N-|ℐ|},\hfill & \hfill \forall {\mathbf{x}}_n\notin ℐ,\hfill \end{array}$$ (4)

   where parameter τ, 0 ≤ τ ≤ 1, could be considered as the confidence of a feature correspondence, and | · | denotes the cardinality of a set.

2. If a data point y_m does not have a corresponding model point, we use equal membership probabilities for all GMM components, i.e. ${\pi }_{\mathit{\text{mn}}}=\frac{1}{N}$, ∀n ∈ IN_N.

Note that in our formulation the membership probability π_mn is not assigned as a prior, since it depends on the data.

C. The EM Algorithm

There are several ways to estimate the parameters of the mixture model, such as the EM algorithm, gradient descent and variational inference. The EM algorithm [20] is a technique for learning and inference in the context of latent variables. It alternates between two steps: an expectation step (E-step) and a maximization step (M-step).

We follow standard notation [35] and omit some terms that are independent of θ. Considering the negative log posterior function, i.e., Eq. (3), the complete-data log posterior (also referred to as the posterior expectation of the complete-data log likelihood) is then given by

$$𝒬(\mathit{\theta },{\mathit{\theta }}^{\text{old}})=\frac{1}{2{\sigma }^2}\sum _{m=1}^M\sum _{n=1}^{N}P(z_m=n|{\mathbf{y}}_m,{\mathit{\theta }}^{\text{old}}){\Vert {\mathbf{y}}_m-𝒯({\mathbf{x}}_n)\Vert }^2+\frac{M_{\mathbf{P}}D}{2}\text{ln }{\sigma }^2-M_{\mathbf{P}}\text{ ln}(1-\gamma )-(M-M_{\mathbf{P}})\text{ ln }\gamma +\frac{\lambda }{2}\varphi (𝒯),$$ (5)

where $M_{\mathbf{P}}={\sum }_{m=1}^M{\sum }_{n=1}^{N}P(z_m=n|{\mathbf{y}}_m,{\mathit{\theta }}^{\text{old}})\le M$.

E-step

We use the current parameter values θ^old to find the posterior distribution of the latent variables. To this end, we first renew the membership probability π_mn based on the feature correspondence between the data points Y and the transformed model points 𝒯(X), as described in Section III-B. We denote by P the posterior probability matrix of size M × N, where the (m, n)-th element p_mn = P(z_m = n|y_m, θ^old) can be computed by applying Bayes rule:

$$p_{\mathit{\text{mn}}}=\frac{P({\mathbf{y}}_m|z_m=n,{\mathit{\theta }}^{\text{old}})P(z_m=n|{\mathit{\theta }}^{\text{old}})}{P({\mathbf{y}}_m|{\mathit{\theta }}^{\text{old}})}=\frac{{\pi }_{\mathit{\text{mn}}}e-\frac{{\Vert {\mathbf{y}}_m-𝒯({\mathbf{x}}_n)\Vert }^2}{2{\sigma }^2}}{{\sum }_{k=1}^N{\pi }_{\mathit{\text{mk}}}{e}^{-\frac{{\Vert {\mathbf{y}}_m-𝒯({\mathbf{x}}_k)\Vert }^2}{2{\sigma }^2}}+\frac{\gamma {(2\pi {\sigma }^2)}^{D/2}}{(1-\gamma )a}}.$$ (6)

Note that the local structure is used throughout the EM process. The posterior probability p_mn is a soft assignment, which indicates to what degree the observed data point y_m coincides with the model point x_n under the current estimated transformation 𝒯.

M-step

We compute the revised parameters as²: θ^new = arg max_θ 𝒬(θ, θ^old). Taking derivatives of 𝒬(θ) with respect to γ and σ², and setting them to zero, we obtain

$$\gamma =1-M_{\mathbf{P}}/M,$$ (7)

$${\sigma }^2=\frac{{\sum }_{m=1}^M{\sum }_{n=1}^N{p}_{\mathit{\text{mn}}}{\Vert {\mathbf{y}}_m-𝒯({\mathbf{x}}_n)\Vert }^2}{M_{\mathbf{P}}D}.$$ (8)

The estimate of the variance σ² with a large initial value is conceptually similar to deterministic annealing [3], which uses the solution of an easy (e.g., smoothed) problem to recursively provide the initial conditions to increasingly harder problems, but differs in several aspects (e.g., by not requiring any annealing schedule).

Next we consider the terms of 𝒬(θ) that are related to 𝒯. We define the transformation as the initial position plus a displacement function v: 𝒯(X) = X + v(X), and then obtain a regularized risk functional as:

$$ℰ(\mathbf{v})=\frac{1}{2{\sigma }^2}\sum _{m=1}^M\sum _{n=1}^N{p}_{\mathit{\text{mn}}}{\Vert {\mathbf{y}}_m-{\mathbf{x}}_n-\mathbf{v}({\mathbf{x}}_n)\Vert }^2+\frac{\lambda }{2}\varphi (\mathbf{v}).$$ (9)

This is a particular form of Tikhonov regularization [36], and the first term can be seen as a weighted empirical error. Thus the maximization of 𝒬 with respect to 𝒯 is equivalent to minimizing the regularized risk functional (9). To proceed, we model v by requiring it to lie within a vector-valued reproducing kernel Hilbert space (RKHS) ℋ defined by a diagonal Gaussian matrix kernel Γ : IR^D × IR^D → IR^D×D [37], where $\mathrm{\Gamma }({\mathbf{x}}_i,{\mathbf{x}}_j)=\text{exp}\{-\frac{{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2}{2{\beta }^2}\}\mathbf{I}$. A brief introduction of the vector-valued RKHS is given in Appendix A. By using the square norm for the smoothness functional ϕ(v), i.e., $\varphi (\mathbf{v})={\Vert \mathbf{v}\Vert }_{\mathscr{H}}^2$, we then have the following representer theorem [38], and the proof is given in Appendix B.

Theorem 1

The optimal solution of the regularized risk functional (9) has the form

$$\mathbf{v}(\mathbf{x})=\sum _{n=1}^N\mathrm{\Gamma }(\mathbf{x},{\mathbf{x}}_n){\mathbf{c}}_n.$$ (10)

with the coefficient set {c_n : n ∈ IN_N} determined by the following linear system:

$$\mathbf{C}(\mathbf{\Gamma }d({\mathbf{1}}^{\mathrm{T}}\mathbf{P})+\lambda {\sigma }^2\mathbf{I})=\mathbf{Y}\mathbf{P}-\mathbf{X}d({\mathbf{1}}^{\mathrm{T}}\mathbf{P}),$$ (11)

where C = (c₁, ⋯, c_N) ∈ IR^D×N is the coefficient set, Γ ∈ IR^N×N is the so-called Gram matrix with the (i, j)-th element ${\mathbf{\Gamma }}_{\mathit{\text{ij}}}=\text{exp}\{-\frac{{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2}{2{\beta }^2}\}$, 1 is a column vector of all ones, and d(·) is the diagonal matrix.

Algorithm 1.

The PR-GLS algorithm

	Input: Two point sets X and Y, parameters γ, β, λ
	Output: Optimal transformation 𝒯
1	Construct the Gram matrix Γ;
2	Initialization: C = 0, ${\sigma }^2=\frac{1}{\mathit{\text{DMN}}}{\sum }_{m,n=1}^{M,N}{\Vert {\mathbf{y}}_m-{\mathbf{x}}_n\Vert }^2$;
3	Compute feature descriptors for point set Y;
4	repeat
5		E-step:
6		Compute feature descriptors for point set 𝒯 (X);
7		Establish correspondence between 𝒯 (X) and Y;
8		Initialize πmn based on the feature correspondence;
9		Update posterior probability P by Eq. (6);
10		M-step:
11		Update C by solving the linear system (11);
12		Update 𝒯 (X) = X + CΓ;
13		Update γ and σ2 by Eqs. (7) and (8);
14	until 𝒬 converges;
15	The transformation 𝒯 is determined as 𝒯 (X) = X + CΓ.

Note that the M-step is different from the standard EM algorithm, where the free parameters are the means and the covariances of the Gaussian mixture, and the estimation of these parameters is quite straightforward. Our problem requires multiple conditional maximization steps and thus the algorithm is referred as the Expectation Conditional Maximization (ECM), which was first introduced into the point registration problem by Horaud et al. [12]. Since our non-rigid Point set Registration algorithm exploits both Global and Local Structures, we call it PR-GLS and summarize it in Algorithm 1.

D. Fast Implementation

Solving for the transformation 𝒯 requires solving the linear system (11). But for large values of N, this may pose a serious problem because this has O(N³) computational complexity. Even if this is tractable, we may prefer a suboptimal, but faster method. In this section, we provide a fast implementation based on an idea related to the Subset of Regressors method [39], [37].

Instead of searching for the optimal solution (10), we use a sparse approximation and randomly pick a subset containing K (K ≪ N) model points {x̃_k : k ∈ IN_K}. Only points in this subset are allowed to have nonzero coefficients in the expansion of the solution, as in [39], [37]. Our experiments showed that simply selecting an arbitrary subset of the training inputs at random does not perform any worse than more sophisticated methods. Therefore, we seek a solution of the form:

$$\mathbf{v}(\mathbf{x})=\sum _{k=1}^K\mathrm{\Gamma }(\mathbf{x},{\stackrel{\sim }{\mathbf{x}}}_k){\mathbf{c}}_k,$$ (12)

with the coefficients C̃ = (c₁, ⋯, c_K) ∈ IR^D×K determined by a linear system

$$\stackrel{\sim }{\mathbf{C}}({\mathbf{U}}^{\mathrm{T}}d({\mathbf{1}}^{\mathrm{T}}\mathbf{P})\mathbf{U}+\lambda {\sigma }^2\stackrel{\sim }{\mathbf{\Gamma }})=\mathbf{\text{YPU}}-\mathbf{X}d({\mathbf{1}}^{\mathrm{T}}\mathbf{P})\mathbf{U},$$ (13)

where Γ̃ is the Gram matrix Γ̃ ∈ IR^K×K, where ${\stackrel{\sim }{\mathbf{\Gamma }}}_{\mathit{\text{ij}}}=\text{exp}\{-\frac{{\Vert {\stackrel{\sim }{\mathbf{x}}}_i-{\stackrel{\sim }{\mathbf{x}}}_j\Vert }^2}{2{\beta }^2}\}$, and U ∈ IR^N×K with the (i, j)-th element ${\mathbf{U}}_{\mathit{\text{ij}}}=\text{exp}\{-\frac{{\Vert {\mathbf{x}}_i-{\stackrel{\sim }{\mathbf{x}}}_j\Vert }^2}{2{\beta }^2}\}$. By using this sparse approximation, the time and space complexities for solving the linear system (11) are reduced from O(N³) and O(N²) to O(K²N) and O(KN), respectively.

E. Related Non-Rigid Registration Methods

The most relevant non-rigid point sets registration algorithm to ours is coherent point drift (CPD) [11], as both algorithms use the GMM formulation and Gaussian radial basis functions to parameterize the transformations. However, our PR-GLS has three major advantages compared to CPD. Firstly, CPD ignores the local structure information in the point sets; it simply uses equal membership probabilities in the mixture model, while our PR-GLS tries to preserve local structures by encouraging the matching of points with similar neighborhood structures, e.g., shape features. Secondly, the outlier percentage γ is fixed and set manually in CPD. This may be problematic in many real-world problems, where the amount of outliers is not known in advance. By contrast, our PR-GLS treats γ as an unknown parameter and estimates it for each EM iteration. Thirdly, CPD can not handle large rotations, e.g., rotations with angles larger than 60°. We solve this problem by choosing rotation invariant features in our formulation.

The two other most relevant non-rigid point sets registration algorithms to ours are RPM-LNS [13] and RPM-L₂E [28]. More specifically, they also use local structure information. RPM-LNS formulates point matching as a graph matching problem. RPM-L₂E establishes point correspondences by first constructing a large putative set of correspondence and then uses a robust L₂E estimator to get rid of the outliers. Both of the two algorithms use shape context to initialize the correspondences. One major drawback of RPM-LNS and RPM-L₂E is that they find the correspondence and the transformation separately, while our method estimates these two variables jointly using the GMM formulation. Moreover, RPM-LNS is not applicable to the 3D case.

In summary, our main contribution is that we propose an efficient framework for preserving both global and local structures during matching. Compared to other related methods, the efficient use of global and local information in our PR-GLS can lead to big improvements under various degenerations (see Fig. 3).

Fig. 3.

Comparison of PR-GLS and its fast version (PR-GLS-F) with SC [23], TPS-RPM [3], RPM-LNS [13], GMMREG [19], CPD [11] and RPM-L₂E [28] on the fish (top) and Chinese character (bottom) [3], [13]. The error bars indicate the registration error means and standard deviations over 100 trials. Note that the variance bar can reach the negative y-axis, where large variance indicates that a method sometimes works well and sometimes totally fails.

F. Implementation Details

The performance of point matching algorithms typically depends on the coordinate system in which the points are expressed. We use data normalization to control this. More specifically, we perform a linear scaling of the coordinates, so that the points in both of the two sets both have zero mean and unit variance.

To assign the membership probability π_mn, we need to establish initial correspondences which is usually time consuming. Fortunately, the deformation of the model shape is gradually reduced during the EM iterations and as such, the correspondences between the two shapes will not change too fast. Therefore, to speed-up our algorithm, we update π_mn in every 10 iterations rather than in each iteration.

Parameter settings

There are three main parameters in our PR-GLS algorithm: γ, β and λ. Parameter γ reflects our assumption on the amount of outliers in the point sets. Parameters β and λ reflect the amount of the smoothness constraint, where β determines the width of the range of the interaction between samples and λ controls the trade-off between the closeness to the data and the smoothness of the solution. In general, we found that our method was robust to parameter changes. Parameter τ is used to initialize the membership probabilities, which we will discuss in the experiments. Moreover, the uniform distribution parameter a is set to be the volume of the bounding box of the data.

Analysis of convergence

The objective function (3) is not convex, so it is unlikely that any algorithm can find its global minimum. However, a stable local minimum is often enough for many practical applications. To this end, our strategy is to initialize the variance σ² with a large initial value and then use the EM algorithm. If the value of σ² is large enough, the objective function becomes convex in a large region which can filter out a lot of unstable shallow local minima. Hence we are likely to find a good minimum for large variance. As σ² decreases, the objective function tends to change smoothly, which makes it likely that using the old minimum as the initial value could be helpful to converge to a new good minimum. Therefore, as the iterations continue, we have a good chance of reaching a stable local minimum. This is conceptually similar to deterministic annealing [3], which uses the solution of an easy problem to recursively give initial conditions to increasingly harder problems.

IV. Experimental Results

In order to evaluate the performance of our algorithm, we conducted three types of experiments: i) non-rigid point set registration for 2D shapes; ii) non-rigid point set registration for 3D shapes; iii) sparse feature correspondence on real images. The experiments were performed on a laptop with 2.5 GHz Intel^® Core™2 Duo CPU, 8 GB memory and Matlab Code.

A. Results on 2D Non-Rigid Shape Matching

We tested our PR-GLS algorithm on the same synthesized data as in [3] and [13]. The data consists of two models with different shapes, where the first model consists of 96 points representing a fish and the second model consists of 108 points, representing a Chinese character. For each model, there are five sets of data designed to measure the robustness of registration algorithms with respect to different degrees of deformation, noise, outliers, rotation and occlusion. In the deformation test, the deformation is generated by using Gaussian radial basis functions with coefficients sampled from a Gaussian distribution with zero mean and a standard deviation ranging from 0.02 to 0.08. In the noise test, Gaussian noise is added with standard deviation ranging from 0 to 0.05. In the outlier test, random outliers are added with the outlier to original data ratio ranging from 0 to 2. In the rotation test, six rotation angles are used: 0, 30, 60, 90, 120, and 180. In the occlusion test, some parts of the data are removed with the removed data to original data ratio ranging from 0 to 0.5. In each test, one of the above distortions is applied to the model set to create an observed data set, and 100 samples are generated for each degradation level. We use shape contexts as the feature descriptor to establish the initial correspondences. It is easy to make the shape context feature descriptors translation and scale invariant and in some applications, rotation invariance is also required. We use the rotation invariant shape context as in [13].

Generally, the parameter τ in Eq. (4) reflects the accuracy of the initial correspondences, which to some extent depends on the degree of the distortion. In these experiments we determine τ adaptively, based on the data. To this end, we first define a registration error ε*, which is independent of the ground truth:

$${\epsilon }^{\ast }={\left(\frac{1}{|𝒮|}\sum _{(n,m)\in 𝒮}{\Vert 𝒯({\mathbf{x}}_n)-{\mathbf{y}}_m\Vert }^2\right)}^{1/2},$$ (14)

where 𝒮 consists of the min(N, M) matches with the largest posterior matching probabilities, i.e., p_mn. For each set of data, we randomly choose 5 of the 100 samples and compute their average registration error with τ = 0.1, 0.3, 0.5, 0.7 and 0.9. Afterwards, we set τ to be the value which achieves the smallest registration error ε*. Typically, τ ≈ 0.9 works well for small distortions and 0.3 for large distortions. For the parameters γ, β and λ, we set them to 0.1, 2 and 3, respectively, throughout the experiments.

The registration progress on the fish shape is illustrated schematically in Fig. 1. The columns show the iterative progress, and each row provides a different type of distortion. The goal is to align the model point set (blue pluses) onto the observed data point set (red circles). From the results, we see that our RPM-GLS registration is robust and accurate, and it typically converges in about 30 iterations. More qualitative results of our method on the two shape models are presented in Fig. 2. We organize the results in every two rows: the upper row presents the data, while the lower row shows the registration result. For the fish model, the shape is relatively simple and discriminative and as such, both the global and the local structures are preserved well, even under large degradations. Our method in this case produces almost perfect alignments, as shown in the left of Fig. 2. For the Chinese character model, the points are spread out on the shape and the local structure is not preserved as well under large degradations. The matching performance then degrades gradually, but it remains acceptable, even for large degradations. Let us consider the rotation test in (a4, b3) of Fig. 2. It is interesting that our method can handle the fold-back contained in the data shape, which is a desirable property in real-world applications. Moreover, in some extreme cases, such as the deformation test in (a1, b3), as well as the outlier test in (a3, b1), it is hard to establish visual correspondences even by human beings, however, our method can still achieve accurate alignments. The average runtime of our method on 2D shape matching with about 100 points is about 0.4 seconds.

Fig. 1.

Schematic illustration of our method for non-rigid point set registration. The goal is to align the model point set (blue pluses) onto the observed data point set (red circles). The columns show the iterative alignment progress. From top to bottom: results on data degraded by deformation, noise, outliers, rotation and occlusion, respectively. Our RPM-GLS registration is robust and accurate in all experiments.

Fig. 2.

Point set registration results of our method on the fish (left) and Chinese character (right) shapes [3], [13], with deformation, noise, outliers, rotation and occlusion presented in every two rows. The goal is to align the model point set (blue pluses) onto the observed data point set (red circles). For each group, the upper figure is the model and data point sets, and the lower figure is the registration result. From left to right, increasing degree of degradation.

To provide a quantitative comparison, we report the results of six state-of-the-art algorithms, such as shape context (SC) [23], TPS-RPM [3], RPM-LNS [13], GMMREG [19], CPD [11], and RPM-L₂E [28] which were implemented using publicly available codes. The registration error of a pair of shapes is quantified as the average Euclidean distance between a point in the warped model shape and the ground truth corresponding point in the observed data shape (note that it is different from ε* defined in Eq. (14)). Then the registration performance of each algorithm is compared by the mean and standard deviation of the registration error of all the 100 samples for each distortion level. The statistical results, error means, and standard deviations for each setting are summarized in Fig. 3.

In the deformation test results, seven algorithms achieve similar performance in both the fish and the Chinese character datasets at low deformation levels. But our PR-GLS outperforms them, especially for large degrees of deformation. In the noise test results, we observe that all algorithms are affected by this type of distortion. Still, our PR-GLS achieves the best registration performance. In the outlier test results, GMMREG is not robust to outliers. RPM-L₂E is easily confused and starts to fail once the outlier level becomes relatively high. This is due to that it solves the correspondence and the transformation problems separately. When the outlier level becomes too high, it cannot establish good correspondences in the first step and hence, the final solution will be badly degraded. In contrast, our PR-GLS shows much more robustness, regardless of the outlier level. Note that the original CPD is not robust to changes in the outlier ratios, as it involves a fixed outlier ratio parameter, i.e., parameter γ in Eq. (2). To make a fair comparison, we set it adaptively similar to the setting of parameter τ in our PR-GLS. In the rotation test, TPS-RPM, GMMREG and CPD degenerate rapidly when the rotation angle is larger than 60°, while RPM-L₂E and our PR-GLS are not affected, because they are using the rotation invariant shape context. In the occlusion test, the registration error of TPS-RPM is the largest, and our PR-GLS again has the best results in most cases once again. We have also tested the fast version of PR-GLS on this dataset and achieved similar performance, as shown in the last column of each test in Fig. 3.

In our PR-GLS, we assign the membership probability π_mn based on shape context feature, so that the local structure information can be used to achieve better performance. To demonstrate the benefits for doing this, we perform experiments on the fish and Chinese character datasets with different degrees of deformations. The results are given in Tables I and II, where CPD assumes equal GMM components (π_mn = 1/N), PR-GLS assigns π_mn based on local shape features, and GT assigns π_mn by using the ground truth correspondences. Clearly, a good initialization of π_mn can improve registration performance, especially when the data is badly degraded.

TABLE I.

Average registration error on the fish datasets with different degrees of deformations (DoD) over 100 trials. The last row (GT) uses the ground truth correspondences to initialize π_ij.

DoD	0.02	0.035	0.05	0.065	0.08
CPD	2.6×10−5	1.3×10−4	1.5×10−3	8.1×10−3	1.6×10−2
PR-GLS	2.5×10−5	7.3×10−5	3.6×10−4	1.5×10−3	4.0×10−3
GT	2.5×10−5	7.3×10−5	2.1×10−4	5.1×10−4	1.0×10−3

TABLE II.

Average registration error on the Chinese character datasets with different degrees of deformations (DoD) over 100 trials. The last row (GT) uses the ground truth to initialize π_ij.

DoD	0.02	0.035	0.05	0.065	0.08
RPM-L2E	4.3×10−4	1.9×10−3	5.9×10−3	1.5×10−2	3.8×10−2
PR-GLS	4.3×10−4	1.4×10−3	3.6×10−3	7.0×10−3	1.8×10−2
GT	4.3×10−4	1.4×10−3	3.5×10−3	5.9×10−3	9.1×10−3

In conclusion, our PR-GLS is effective for various non-rigid point set registration problems, regardless of large deformation, noise, outliers, rotation or occlusion.

B. Results on 3D Non-Rigid Shape Matching

We next test our PR-GLS algorithm for 3D shape matching. We adopt the FPFH [34] to determine the initial correspondence. For the dataset, we use two pairs of point sets, representing a wolf in different poses from a surface correspondence benchmark³ [40], as shown in the left two columns of Fig. 4. The ground truth correspondences are supplied by the dataset. Some other challenging benchmark datasets with known ground truth correspondences have also been developed [41], [42]. To assign the membership probability π_mn, parameter τ can be determined adaptively according to Eq. (14); here, we fix it to 0.9 for efficiency. Moreover, 3D datasets typically contain thousands of points (e.g., about 5, 000 in the wolf shape), which may pose a serious problem due to heavy computational or memory requirements; here, we use the fast version of our method with K = 50 in Eq. (12).

Fig. 4.

Point set registration results of our method on the 3D wolf shape [40], with deformation, occlusion and outliers presented in every two columns. The goal is to align the model point set (blue dots) onto the observed data point set (red circles). For each group of experiments, the upper figure is the model and data point sets, and the lower figure is the registration result.

Fig. 4 presents the registration results of PR-GLS. In the left two columns, the data contains large rotations and slight pose changes, and our method gives almost perfect alignments. In the middle two and right two columns, we also test our method on occlusion and outliers. In each group, the first set contains a degradation on only one set of points, while the second set contains degradations on both the two point sets. The last column demonstrates a challenging example, where we biased the outliers to the different sides of point sets, and the outliers were generated randomly from a normal distribution rather than a uniform distribution. The results demonstrate that our method is quite robust and is not affected by all these degradations. The average runtime of our method with the fast implementation on 3D shape matching with about 5, 000 points is about 1 minute. Note that in this experiment the FPFH is quite efficient and requires only 1.5 seconds approximately to establish the initial correspondences.

A quantitative comparison of our method with CPD and RPM-L₂E is reported in Table III. In addition, we also compare with a recent method using local structure information based on Tensor Fields (TF) [27]. In the case of CPD, we rotate the data, so that the two point sets have roughly similar orientations, since CPD is not robust to large rotations. As shown in Table III, our PR-GLS consistently outperforms the other three algorithms in every case. In conclusion, PR-GLS is effective for non-rigid registration, in both the 2D and the 3D cases.

TABLE III.

Comparison of the registration errors of PR-GLS with respect to CPD [11], RPM-L₂E [28] and TF [27] on the 3D wolf shape in Fig. 4. From left to right: deformation (Def1, Def2), occlusion (Occ1, Occ2), and outlier (Out1, Out2).

	Def1	Def2	Occ1	Occ2	Out1	Out2
CPD	0.80	0.82	1.05	0.72	3.58	3.37
RPM-L2E	1.03	1.18	1.13	0.96	1.22	1.35
TF	1.12	1.36	2.53	3.13	3.87	5.85
PR-GLS	0.50	0.71	0.58	0.45	0.74	0.92

C. Results on Image Feature Correspondence

In this section, we test our PR-GLS on real image data, focusing on establishing visual correspondences between two images. Here the images contain some deformable objects and consequently, the underlying relationships between the images are non-rigid. We use SIFT [33] for feature detection and description. The goal is to find correspondences/matches between two sets of sparse feature points {x_n : n ∈ IN_N} and {y_m : m ∈ IN_M}, with corresponding feature descriptors. To assign the membership probability π_mn, we determine the initial correspondences by searching the nearest neighbors and fix parameter τ to 0.9. Note that since the feature descriptor here is computed based on an image patch rather than the neighboring points, the value of π_mn does not need to be recalculated in every iteration.

Fig. 5 contains a T-shirt with different amount of spatial warps, where we aim to establish SIFT matches in each image pair. We consider (x_n, y_m) to be a correspondence/match if its posterior probability p_mn is larger than 0.5. Performance is characterized by the number of correct matches and the matching score (i.e., the ratio between the number of correct matches and the number of total matches). Since the image relations are non-rigid and not known in advance, it is impossible to establish accurate ground truth. We evaluate the matching results by manually checking. Though the judgment of correct match or false match seems arbitrary, we make the benchmark to ensure objectivity.

Fig. 5.

Results of image feature correspondence on image pairs of deformable objects. From left to right, increasing degree of deformation. The lines indicate matching results (blue = correct matches, red = false matches).

In feature correspondence, the image typically contains hundreds or thousands of feature points, thus, for efficiency, we use the fast version of our method with K = 15 in Eq. (12). The results of our method are presented in Fig. 5. In the left columns, where the deformations are relatively small, our method obtains a lot of correspondences, whereas in the right columns, the deformations are large and the number of correct matches decreases. Still, even in these cases, the matching scores remain quite high, e.g., around 95% or higher. For comparison, we also report three other matching methods as shown in Fig. 6. The SIFT method can be considered as a baseline. To eliminate as many outliers as possible, while preserving the inliers, SIFT compares the distance ratio between the nearest and the second nearest neighbors against a predefined threshold to decide whether they are considered a match or not. From these results, we see that the matching score of SIFT is not good, especially in the case of large deformations. This suggests that using local image features alone is insufficient. CPD is not efficient for such problems, since it ignores the local image features’ information. However, RPM-L₂E and PR-GLS consider both global structures and local features and hence, they are able to find more correct matches and achieve high matching scores. The average runtime of our method with the fast implementation on the six image pairs is about 0.5 seconds per pair.

Fig. 6.

Comparison of SIFT [33], CPD [11], RPM-L₂E [28] and PR-GLS for feature correspondence. Left: number of correct matches; right: matching score. The horizontal axis corresponds to the six image pair in Fig. 5 (from left to right, top to bottom).

V. Conclusion

In this paper, we presented a new approach for non-rigid registration. A key characteristic of our approach is the preservation of both global and local structures during matching. We also provide an efficient implementation of our method for handling large datasets, which significantly reduces the computational complexity without reducing the quality of the matching significantly. Experiments on public datasets for 2D and 3D non-rigid point registration and on real images for sparse image feature correspondence demonstrate that our approach yields superior results to those of the state-of-the-art methods, when there are significant deformations, noise, outliers, rotations and/or occlusions in the data.

Biographies

Alan L. Yuille received his B.A. in mathematics from the University of Cambridge in 1976, and completed his Ph.D. in theoretical physics at Cambridge in 1980 studying under Stephen Hawking. Following this, he held a postdoc position with the Physics Department, University of Texas at Austin, and the Institute for Theoretical Physics, Santa Barbara. He then joined the Artificial Intelligence Laboratory at MIT (1982–1986), and followed this with a faculty position in the Division of Applied Sciences at Harvard (1986–1995), rising to the position of associate professor. From 1995–2002 Alan worked as a senior scientist at the Smith-Kettlewell Eye Research Institute in San Francisco. In 2002 he accepted a position as full professor in the Department of Statistics at the University of California, Los Angeles. He has over two hundred peer-reviewed publications in vision, neural networks, and physics, and has co-authored two books: Data Fusion for Sensory Information Processing Systems (with J. J. Clark) and Two- and Three- Dimensional Patterns of the Face (with P. W. Hallinan, G. G. Gordon, P. J. Giblin and D. B. Mumford); he also co-edited the book Active Vision (with A. Blake). He has won several academic prizes and is a Fellow of IEEE.

Jiayi Ma received the B.S. degree from the Department of Mathematics, and the Ph.D. Degree from the School of Automation, Huazhong University of Science and Technology, Wuhan, China, in 2008 and 2014, respectively. From 2012 to 2013, he was with the Department of Statistics, University of California at Los Angeles. He is now a postdoctoral research associate with the Electronic Information School, Wuhan University. His current research interests include in the areas of computer vision, machine learning, and pattern recognition.

Ji Zhao received the B.S. degree in automation from Nanjing University of Posts and Telecommunication in 2005. He received the Ph.D. degree in control science and engineering from HUST in 2012. Since 2012, he is a postdoctoral research associate at the Robotics Institute, Carnegie Mellon University. His research interests include image classification, image segmentation and kernel-based learning.

Appendix A

Vector-Valued Reproducing Kernel Hilbert Space

We review the basic theory of vector-valued reproducing kernel Hilbert space, and for further details and references we refer to [38], [43].

Let 𝒳 be a set, for example, 𝒳 ⊆ IR^P, 𝒴 a real Hilbert space with inner product (norm) 〈·, ·〉, (‖ · ‖), for example, 𝒴 ⊆ IR^D, and ℋ a Hilbert space with inner product (norm) 〈·, ·〉_ℋ, (‖ · ‖_ℋ), where P = D = 2 or 3 for point matching problem. Note that a norm can be induced by an inner product, for example, $\forall \mathbf{f}\in \mathscr{H},{\Vert \mathbf{f}\Vert }_{\mathscr{H}}=\sqrt{{\langle \mathbf{f},\mathbf{f}\rangle }_{\mathscr{H}}}$. And a Hilbert space is a real or complex inner product space that is also a complete metric space with respect to the distance function induced by the inner product. Thus a vector-valued RKHS can be defined as follows.

Definition 1

A Hilbert space ℋ is an RKHS if the evaluation maps eυ_x : ℋ → 𝒴 (i.e., eυ_x(f) = f (x)) are bounded, i.e., if ∀x ∈ 𝒳 there exists a positive constant C_x such that

$$\Vert e{\upsilon }_{\mathbf{x}}(\mathbf{f})\Vert =\Vert \mathbf{f}(\mathbf{x})\Vert \le C_{\mathbf{x}}{\Vert \mathbf{f}\Vert }_{\mathscr{H}},\forall \mathbf{f}\in \mathscr{H}.$$ (15)

A reproducing kernel Γ: 𝒳 × 𝒳 → ℬ(𝒴) is then defined as: $\mathrm{\Gamma }(\mathbf{x},\mathbf{x}\prime )≔e{\upsilon }_{\mathbf{x}}e{\upsilon }_{\mathbf{x}\prime }^{\ast }$, where ℬ(𝒴) is the Banach space of bounded linear operators (i.e., Γ(x, x′), ∀x, x′ ∈ 𝒳) on 𝒴, for example, ℬ(𝒴) ⊆ IR^D×D, and $e{\upsilon }_{\mathbf{x}}^{\ast }$ is the adjoint of eυ_x. We have the following two properties about the RKHS and kernel.

Remark 1

The kernel Γ reproduces the value of a function f ∈ ℋ at a point x ∈ 𝒳. Indeed, ∀x ∈ 𝒳 and y ∈ 𝒴, we have $e{\upsilon }_{\mathbf{x}}^{\ast }\mathbf{y}=\mathrm{\Gamma }(·,\mathbf{x})\mathbf{y}$, so that 〈f(x), y〉 = 〈f, Γ(·, x)y〉_ℋ.

Remark 2

An RKHS defines a corresponding reproducing kernel. Conversely, a reproducing kernel defines a unique RKHS.

More specifically, for any $N\in \text{IN},{\{{\mathbf{x}}_n\}}_{n=1}^N\subseteq 𝒳$, and a reproducing kernel Γ, a unique RKHS can be defined by considering the completion of the space

$${\mathscr{H}}_N=\{\sum _{n=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_n){\mathbf{c}}_n:{\mathbf{c}}_n\in 𝒴\},$$ (16)

with respect to the norm induced by the inner product

$${\langle \mathbf{f},\mathbf{g}\rangle }_{\mathscr{H}}=\sum _{i,j=1}^N\langle \mathrm{\Gamma }({\mathbf{x}}_j,{\mathbf{x}}_i){\mathbf{c}}_i,{\mathbf{d}}_j\rangle \forall \mathbf{f},\mathbf{g}\in {\mathscr{H}}_N,$$ (17)

where $\mathbf{f}={\sum }_{i=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_i){\mathbf{c}}_i$ and $\mathbf{g}={\sum }_{j=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_j){\mathbf{d}}_j$.

Appendix B

Proof of Theorem 1

For any given reproducing kernel Γ, we can define a unique RKHS ℋ_N as in Eq. (16) in Appendix A. Let ${\mathscr{H}}_N^{\perp }$ be a subspace of ℋ,

$${\mathscr{H}}_N^{\perp }=\{\mathbf{v}\in \mathscr{H}:\mathbf{v}({\mathbf{x}}_n)=0,n\in {\text{IN}}_N\}.$$ (18)

From the reproducing property, i.e. Remark 1, $\forall \mathbf{v}\in {\mathscr{H}}_N^{\perp }$

$${\langle \mathbf{v},\sum _{n=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_n){\mathbf{c}}_n\rangle }_{\mathscr{H}}=\sum _{n=1}^N\langle \mathbf{v}({\mathbf{x}}_i),{\mathbf{c}}_n\rangle =0.$$ (19)

Thus ${\mathscr{H}}_N^{\perp }$ is the orthogonal complement of ℋ_N; then every v ∈ ℋ can be uniquely decomposed into components along and perpendicular to ${\mathscr{H}}_N:\mathbf{v}={\mathbf{v}}_N+{\mathbf{v}}_N^{\perp }$, where v_N ∈ ℋ_N and ${\mathbf{v}}_N^{\perp }\in {\mathscr{H}}_N^{\perp }$. Since by orthogonality ${\Vert {\mathbf{v}}_N+{\mathbf{v}}_N^{\perp }\Vert }_{\mathscr{H}}^2={\Vert {\mathbf{v}}_N\Vert }_{\mathscr{H}}^2+{\Vert {\mathbf{v}}_N^{\perp }\Vert }_{\mathscr{H}}^2$ and by the reproducing property v(x_n) = v_N(x_n), the regularized risk functional then satisfies

$$ℰ(\mathbf{v})=\frac{1}{2{\sigma }^2}\sum _{m=1}^M\sum _{n=1}^N{p}_{\mathit{\text{mn}}}{\Vert {\mathbf{y}}_m-{\mathbf{x}}_n-\mathbf{v}({\mathbf{x}}_n)\Vert }^2+\frac{\lambda }{2}{\Vert {\mathbf{v}}_N+{\mathbf{v}}_N^{\perp }\Vert }_{\mathscr{H}}^2\ge \frac{1}{2{\sigma }^2}\sum _{m=1}^M\sum _{n=1}^N{p}_{\mathit{\text{mn}}}{\Vert {\mathbf{y}}_m-{\mathbf{x}}_n-{\mathbf{v}}_N({\mathbf{x}}_n)\Vert }^2+\frac{\lambda }{2}{\Vert {\mathbf{v}}_N\Vert }_{\mathscr{H}}^2.$$ (20)

Therefore, the optimal solution of the regularized risk functional (9) comes from the space ℋ_N, and hence has the form (10). To solve the coefficients, we consider the definition of the smoothness functional ϕ(v) and the inner product (17)

$${\Vert \mathbf{v}\Vert }_{\mathscr{H}}^2={\langle \mathbf{v},\mathbf{v}\rangle }_{\mathscr{H}}={\langle \sum _{i=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_i){\mathbf{c}}_i,\sum _{j=1}^N\mathrm{\Gamma }(·,{\mathbf{x}}_j){\mathbf{c}}_j\rangle }_{\mathscr{H}}=\sum _{i,j=1}^N\langle \mathrm{\Gamma }({\mathbf{x}}_j,{\mathbf{x}}_i){\mathbf{c}}_i,{\mathbf{c}}_j\rangle =\mathbf{C}\mathbf{\Gamma }{\mathbf{C}}^{\mathrm{T}},$$ (21)

where C = (c₁, ⋯, c_N) ∈ IR^D×N is the coefficient set. The regularized risk functional then can be conveniently expressed in the following matrix form:

$$ℰ(\mathbf{C})=\frac{1}{2{\sigma }^2}\sum _{m=1}^M{\Vert ({\mathbf{Y}}_m^N-\mathbf{X}-\mathbf{C}\mathbf{\Gamma })d{({\mathbf{P}}_m)}^{1/2}\Vert }_F^2+\frac{\lambda }{2}\mathbf{C}\mathbf{\Gamma }{\mathbf{C}}^{\mathrm{T}},$$ (22)

where Γ ∈ IR^N×N is the so-called Gram matrix with the (i, j)-th element ${\mathbf{\Gamma }}_{\mathit{\text{ij}}}=\text{exp}\{-\frac{{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2}{2{\beta }^2}\},{\mathbf{Y}}_m^N=({\mathbf{y}}_m,\cdots ,{\mathbf{y}}_m)\in {\text{IR}}^{D\times N}$, P_m is the m-th row of P, ‖·‖_F denotes the Frobenius norm, and d(·) is the diagonal matrix. Taking the derivative of the last equation with respect to C and setting it to zero, we obtain the linear system in Eq. (11). Thus the coefficient set {c_n : n ∈ IN_N} of the optimal solution v is determined by the linear system (11).