Manifold Learning by Preserving Distance Orders

Abstract

Nonlinear dimensionality reduction is essential for the analysis and the interpretation of high dimensional data sets. In this manuscript, we propose a distance order preserving manifold learning algorithm that extends the basic mean-squared error cost function used mainly in multidimensional scaling (MDS)-based methods. We develop a constrained optimization problem by assuming explicit constraints on the order of distances in the low-dimensional space. In this optimization problem, as a generalization of MDS, instead of forcing a linear relationship between the distances in the high-dimensional original and low-dimensional projection space, we learn a non-decreasing relation approximated by radial basis functions. We compare the proposed method with existing manifold learning algorithms using synthetic datasets based on the commonly used residual variance and proposed percentage of violated distance orders metrics. We also perform experiments on a retinal image dataset used in Retinopathy of Prematurity (ROP) diagnosis.

1. Introduction

Due to the recent advances, acquisition of large volumes of high dimensional data has become more common in every aspect of daily life: stock market, social media, medical data, etc. Analysis and interpretation of such data requires finding meaningful low-dimensional structures in these huge data sets. Manifold learning attempts to accomplish such data explorations and dimensionality reductions.

Manifold learning can be regarded as identifying a nonlinear mapping from the original higher dimensional data space to a lower dimensional representation. Existing methods can be classified into three categories: global methods that tend to preserve global properties in the low-dimensional representation, local methods that aim to preserve the local geometry in the embedded space and techniques based on global alignment of multiple linear models [1]. Multidimensional scaling (MDS) is one of the global methods that finds a projection of the original data while preserving the pairwise Euclidean distances [2]. In the literature, various techniques are proposed to minimize MDS cost function [3, 4, 5, 6]. Similarly, in Isomap, one uses a geodesic distance estimation to use with MDS [7]. Different variations of Isomap have been proposed in the literature: landmark and conformal Isomap [8]. On the other hand, local methods [9, 10, 11, 12, 13, 14, 15] constructs the lower dimensional data using the local linear relations in the original space. Local tangent space alignment (LTSA) [9] represents the local geometry of the manifold with local tangent spaces that are learned through the neighborhood of each sample. Similarly, local linear embedding (LLE) [10] aims to preserve local neighborhood information, while Semidefinite Embedding (SDE) [11] involves preserving local isometries on a k-nn graph. Coifman et al. [14] presents a method that constructs local coordinates by learning a family of diffusion maps (DM). Another use of local geometry is by locally smooth manifold learning (LSML) [12, 13] which is based on learning a warping function, that takes any sample in the manifold and generates its neighbors. Stochastic neighborhood embedding (SNE) [15] and its variations [16, 17] are among probabilistic approaches that construct the neighborhood relations based on Gaussian kernels. Although local approaches have computational advantages, they might have limitations in preserving global geometry, especially if the data is sparse. Other methods that are based on global alignment of linear models aim to combine the local and global techniques by fitting a number of local linear models and merging them with a global alignment. Local linear coordination (LLC) [18] and manifold charting [19] methods fall into this category.

In this manuscript, we propose a nonlinear dimensionality reduction method that extends the basic idea used by the MDS and its variations. Although the ultimate goal is to preserve the distance orders, MDS algorithm only focuses on minimizing the mean-squared error between the input and output distances [2]. There is no explicit constraint on the distance orders during the solution of this optimization problem. Moreover, this minimization results in a linear relationship between distance spaces. Linear fit assumption between the distances in the original and low-dimensional projection spaces is very restricted and embedding is achieved in this restrictive family. To address these two important issues, we first generalize the mean squared error cost function to include more general relations between distance spaces. Instead of assuming a predefined relationship, we propose to also learn the relationship between distances while we project the data from the original space. Our only assumption on the relationship is to have a monotonic non-decreasing function in order to preserve the distance relationships observed in the original space in the projected space. Then, we develop a constrained optimization problem by incorporating the distance orders as inequality constraints. As a solution of this problem, we not only learn the data in the projected space but also learn the non-linear relation between distance spaces. The final form of the proposed optimization problem is a generalization of the existing global MDS-based manifold learning algorithms such that the existing methods are approximate solutions of the simplified version of this problem. In this manuscript, we focus on the formulation and theoretical aspects of the problem. Possible acceleration of the proposed method by convex relaxations and further approximations to analyze real data will be part of our future research.

Another commonly used manifold learning algorithm which has a nonlinear mapping between distance spaces is Sammons mapping [20]. Sammon’s map, a nonlinear extension of MDS, first maps the input data to a nonlinear predefined feature space and tries to preserve the distances in this feature space. Different than Sammon’s mapping, we assume an unknown nonlinear relationship between the input and output distances while preserving the distance orders from the original space.

The rest of the paper is organized as follows: We first define the notation used throughout the paper in section 2.1. Next, problem formulation is presented in section 2.2. The solution of the optimization problem and performance evaluation metrics are explained in section 2.3 and section 2.4 respectively. In section 3, we report the experiments and results and the paper is concluded in section 4.

2. Learning Algorithm

In this section, we describe the proposed method for manifold learning. We first define the data model and notations to be used throughout the manuscript. Then using this model we formulate the desired manifold learning problem and develop our algorithm. We derive an optimization problem that solves the manifold learning algorithm, starting with the commonly used cost function, mean-squared error minimization, and demonstrate that this cost function can be extended to include different distance relations between the original and projection space data points, and explicit constraints that preserve distance orders in the projected space.

2.1. Data Model and Notation

We represent the original and the projected data spaces as 𝒳 and 𝒴, respectively. Then, x_i ∈ 𝒳 and y_i ∈ 𝒴 with i = 1, …, N are the data points. In this representation, x and y are vectors and N is the number of the data points. We assume that dim(𝒳) = d ≥ dim(𝒴) = d̃. Moreover, we have $d_{i,j}^x$ and $d_{i,j}^y$ as the distances between the i^th and j^th data points in the original and the low-dimensional data spaces, ‖ · ‖ represents the L2-norm of a vector.

2.2. Problem Formulation

We formulate the manifold learning algorithm as a constrained optimization problem. Our approach restricts the minimum mean-squared error solutions used by some existing manifold learning algorithms [2, 7]. Specifically, these aim to minimize the difference between the distances of any two points in the original and projected spaces. That is, the difference between $d_{i,j}^y$ and $d_{i,j}^x$ for ∀i, j = 1, …, N is minimized, which on average results in a linear relationship between each $d_{i,j}^x$ and $d_{i,j}^y$ pair (as a result of the least-square solution).

$$\underset{{\mathit{y}}_{k}k=1,\dots ,N}{\text{min}}{\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-d_{i,j}^x\Vert }^2$$ (1)

where $d_{i,j}^y=\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert$ is the Euclidean distance and $d_{i,j}^x$ is the distance between the i^th and j^th points. Note here that the minimization is performed over the data points y_i in the projected space.

In the proposed algorithm, we compute $d_{i,j}^x$ as the estimated geodesic distance between the i^th and j^th data points. We follow the method described in [7] to compute the geodesic distances in the original space. First, the Euclidean distance between every pair of data points in the original space (data pairwise distance matrix) is computed. Then, a k-nearest neighbor (knn) graph or ε-ball graph is generated. That is, k-nearest neighbors of a data point or neighbors within ε distance for each datum is taken, and the edge lengths from points outside these areas to the reference datum are set to be infinite, and the pairwise distance matrix is updated accordingly. Finally, Floyd algorithm is applied over this matrix to find approximate geodesic distances between the data pairs [7]. Floyd’s algorithm, an example of dynamic programming, finds the shortest path between each pair of vertices in a weighted graph [21].

In our algorithm, we propose to generalize the minimum mean-squared error approach in (1) to include a broader relationship between the distances in the original and the low-dimensional projection space. For that purpose we rewrite (1) as

$$\underset{{\mathit{y}}_{k}k=1,\dots ,N}{\text{min}}\sum _{i=1}^{N-1}\sum _{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2$$ (2)

where h(·) is a monotonic nondecreasing function. We represent the derivative of h(·) as

$$h\prime (d_{i,j}^x)=\sum _{k=1}^{N-1}\sum _{l=k+1}^N{w}_{kl}k(d_{i,j}^x-d_{k,l}^x)$$ (3)

where k(η) is a translation-invariant kernel function and w_kl ’s are the multiplicative coefficients. We force w_kl ≥ 0 to have h(·) as a nondecreasing function. That is, we represent the derivative of h(·) as a nonnegative weighted sum of kernel functions. Monotonic nondecreasing functions h(·) will guarantee that we preserve the order of original distances in the projected space.

Then, we have

$$h(d_{i,j}^x)=\sum _{k=1}^{N-1}\sum _{l=k+1}^N{w}_{kl}K(d_{i,j}^x-d_{k,l}^x)$$ (4)

where

$$K(d_{i,j}^x-d_{k,l}^x)={\int }_{-\infty }^{d_{i,j}^x}k(\xi -d_{k,l}^x)d\xi$$ (5)

Kernel functions are positive semidefinite and hence are appropriate to represent the derivative of a monotonic nondecreasing function h(·) [22]. A translation-invariant kernel k(η) is chosen such that 0 ≤ k(η) < ∞, ${\int }_{-\infty }^{\infty }k(\eta )\mathrm{d}\eta =1$, and lim_η→0 k(η) = δ(η). We specifically use Gaussian radial basis functions (rbf), since they were shown to universally approximate any function with any desired accuracy [23]. Then, we rewrite (5) as

$$K(d_{i,j}^x-d_{k,l}^x)=\mathrm{\Phi }\left(\frac{d_{i,j}^x-d_{k,l}^x}{\sigma }\right)$$ (6)

where Φ(·) is the Gaussian cumulative distribution function, and σ is the kernel width (standard deviation), which is estimated using Silverman’s rule [24].

From (3), we have $M=\frac{N(N-1)}{2}$ different non-negative w_kl coefficients. We form the M × 1 size vector w as

$$\mathit{w}={[w_{12}{w}_{13}\cdots w_{(N-1)N}]}^T={[w_1{w}_2\cdots w_M]}^T$$ (7)

We also define ${\mathit{y}}_{\mathit{d}}={[{\mathit{y}}_1^T\cdots {\mathit{y}}_N^T]}^T$ as the N d̃ × 1 size data vector in the projected space. Using the definitions for w and y_d, and the specifications provided for the function h(·), we then formulate the manifold learning algorithm as the following optimization problem.

$$\begin{array}{c}\hfill \underset{{\mathit{y}}_{\mathit{d}},\mathit{w}}{\text{min}}{\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\mathit{w}⪰\mathbf{0}\hfill \end{array}$$ (8)

where ⪰ represents an element-wise inequality, such that we require each element of w is greater than or equal to zero. Note that in (8), we aim to avoid the global minimum solution where all y_i, i = 1, …, N are the same and w = 0, which is not a valid projection solution. Therefore, we update the problem as

$$\begin{array}{cc}\hfill \underset{{\mathit{y}}_{\mathit{d}},\mathit{w}}{\text{min}}\hfill & \hfill {\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\hfill & \hfill \mathit{w}⪰\mathbf{0}\text{and}{\sum }_{m=1}^M{w}_m>0\hfill \end{array}$$ (9)

The existing manifold learning algorithms do not explicitly enforce constraints to preserve the distance relationships observed in the original space. In our method, we propose to consider the following constraint: If T distance pairs in the original space are ordered as $d_{i_t,j_t}^x\le d_{i_t^{\prime },j_t^{\prime }}^x$, we require to have $d_{i_t,j_t}^y\le d_{i_t^{\prime },j_t^{\prime }}^y$. This is an approximation to preserving the order relation-ship between every distance pair in the original space. Selecting T achieves a reduction in complexity, but also may reflect preference about which distance orders are more important to preserve for the user. Then, we have the following optimization problem

$$\begin{array}{cc}\hfill \underset{{\mathit{y}}_{\mathit{d}},\mathit{w}}{\text{min}}\hfill & \hfill {\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\hfill & \hfill \mathit{w}⪰\mathbf{0},{\sum }_{m=1}^M{w}_m>0\text{and}{d}_{i_t,j_t}^y\le d_{i_t^{\prime },j_t^{\prime }}^{y}t=1,\dots ,T\hfill \end{array}$$ (10)

Depending on the data under consideration, feasibility in (10) could be difficult to achieve, it might not even be possible to satisfy all the inequality constraints; therefore, to obtain a solution to the proposed constrained optimization problem, we further update it by perturbing (relaxing) the distance inequality constraints to

$$\begin{array}{cc}\hfill \underset{{\mathit{y}}_{\mathit{d}},\mathit{w},\mathit{\xi }}{\text{min}}\hfill & \hfill {\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2+\gamma {\sum }_{t=1}^T{\alpha }_t{\xi }_t\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\hfill & \hfill \mathit{w}⪰\mathbf{0},\mathit{\xi }⪰\mathbf{0},{\sum }_{m=1}^M{w}_m>0\text{and}{d}_{i_t^{\prime },j_t^{\prime }}^y-d_{i_t,j_t}^y+{\xi }_t⪰0t=1,\dots ,T\hfill \end{array}$$ (11)

where ξ = [ξ₁ ⋯ ξ_T]^T is the vector of the perturbation slack variables which store the deviation from the nonlinear inequality constraints representing the order relationship between the distance pairs in the projection space, γ is the penalty factor forcing the problem to satisfy as many order relationships as possible, and α’s demonstrate the importance of different ξ’s such that preserving some inequalities could be more important than others.

In order to use in 2.3, we follow the above discussions, and define

• θ = [y_d^T, w^T, ξ^T]^T is an (N d̃ + M + T) × 1 vector,

• $f(\mathit{\theta })={\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\Vert d_{i,j}^y-h(d_{i,j}^x)\Vert }^2+\gamma {\sum }_{t=1}^T{\alpha }_t{\xi }_t$,

• A = [0 I] as (M + T) × (M + T + N d̃) size matrix,

• $\mathit{g}(\mathit{\theta })={[g_1(\mathit{\theta }),\dots ,g_T(\mathit{\theta })]}^T={[d_{i_1^{\prime },j_1^{\prime }}^y-d_{i_1,j_1}^y+{\xi }_1,\dots ,d_{i_T^{\prime },j_T^{\prime }}^y-d_{i_T,j_T}^y+{\xi }_T]}^T$,

• $\mathit{c}(\mathit{\theta })=\left[\begin{array}{c}\hfill \mathit{A}\mathit{\theta }\hfill \\ \hfill \mathit{g}(\mathit{\theta })\hfill \end{array}\right]={[c_1(\mathit{\theta }),\dots ,c_{M+2T}(\mathit{\theta })]}^T$.

Note that we improve the problem in (11) with the condition ${\sum }_{m=1}^M{w}_m\ge e_c$, and note also that due to the quadratic dependency of the cost function to w, we can always find a scaled w to obtain the minimum of the cost function at ${\sum }_{m=1}^M{w}_m=e_c$, and hence we define $c_{eq}={\sum }_{m=1}^M{w}_m-e_c=0$. Then, we rewrite the problem as

$$\begin{array}{cc}\hfill \underset{\mathit{\theta }}{\text{min}}\hfill & \hfill f(\mathit{\theta })\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\hfill & \hfill c_{eq}=0\text{and}\mathit{c}(\mathit{\theta })⪰\mathbf{0}\hfill \end{array}$$ (12)

In our implementation, we always used a chain of T = M − 1 inequalities $d_{i_1,j_1}^y\le d_{i_2,j_2}^y\le \dots \le d_{i_M,j_M}^y$ but one can choose arbitrary pairs of inequalities. In the following we also refer $d_{i_k,j_k}^y$ as $d_k^y$, then $g_k(\mathit{\theta })=d_{k+1}^y-d_k^y+{\xi }_k$.

2.3. Solution of the Optimization Problem

In this section, we explain the solution method that we employ to solve the constrained optimization problem proposed in (12). We first redefine the problem using extra slack variables as

$$\begin{array}{cc}\hfill \underset{\mathit{\theta },\mathit{s}}{\text{min}}\hfill & \hfill f(\mathit{\theta })\hfill \\ \hfill \mathrm{s}.\mathrm{t}.\hfill & \hfill c_{eq}=0,\mathit{c}(\mathit{\theta })-\mathit{s}=\mathbf{0},\text{and}\mathit{s}⪰\mathbf{0}\hfill \end{array}$$ (13)

Then the Lagrangian of the problem is computed as

$$L(\mathit{\theta },\mathit{s},\mathit{z})=f(\mathit{\theta })-c_{eq}q-{(\mathit{c}(\mathit{\theta })-\mathit{s})}^T\mathit{z}$$ (14)

where q and z = [z₁, …, z_3M−2]^T are the Lagrange multipliers.

Using the proposed problem in (13), and the Lagrangian in (14), we compute the Karush-Kuhn-Tucker (KKT) conditions for this problem [25]

$${\nabla }_{\mathit{\theta }}f(\mathit{\theta })-q{\nabla }_{\mathit{\theta }}{c}_{eq}-{\mathit{D}}^T(\mathit{\theta })\mathit{z}=\mathbf{0}$$ (15a)

$$\mathit{S}\mathit{z}=\mathbf{0}$$ (15b)

$$c_{eq}=0$$ (15c)

$$\mathit{c}(\mathit{\theta })-\mathit{s}=\mathbf{0}$$ (15d)

$$\mathit{s}⪰\mathbf{0}\text{and}\mathit{z}⪰\mathbf{0}$$ (15e)

where

• ${\nabla }_{\mathit{\theta }}f(\mathit{\theta })={[\frac{\partial f(\mathit{\theta })}{\partial {\theta }_1},\dots ,\frac{\partial f(\mathit{\theta })}{\partial {\theta }_{N\mathrm{d̃}+2M-1}}]}^T$ is the gradient of the cost function f (θ) with respect to θ,

• ∇_θc_eq = 1, a vector of ones,

• D(θ) is the Jacobian of the vector of functions c(θ), such that D^T (θ) = [∇_θc₁(θ) ⋯ ∇_θc_3M−2(θ)],

• S is a diagonal matrix with the s vector in the diagonal, such that S = diag(s).

Interior-point methods were shown to outperform active set [26] and augmented Lagrangian methods [27] in large scale problems [28, 29]. We therefore solve the problem in (13) using interior-point methods such that we define p_θ, p_s, p_q, and p_z as the step lengths in the variables θ, s, q, and z, respectively, and apply the Newton’s method to compute the primal-dual update matrix as

$$\left[\begin{array}{cccc}\hfill {\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}L(\mathit{\theta },\mathit{s},\mathit{z})\hfill & \hfill 0\hfill & \hfill -{\nabla }_{\mathit{\theta }}{c}_{eq}\hfill & \hfill -{\mathit{D}}^T(\mathit{\theta })\hfill \\ \hfill \mathbf{0}\hfill & \hfill \mathit{Z}\hfill & \hfill \mathbf{0}\hfill & \hfill \mathit{S}\hfill \\ \hfill {\nabla }_{\mathit{\theta }}{c}_{eq}^T\hfill & \hfill \mathbf{0}\hfill & \hfill 0\hfill & \hfill \mathbf{0}\hfill \\ \hfill \mathit{D}(\mathit{\theta })\hfill & \hfill -\mathit{I}\hfill & \hfill \mathbf{0}\hfill & \hfill \mathbf{0}\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathit{p}}_{\mathit{\theta }}\hfill \\ \hfill {\mathit{p}}_{\mathit{s}}\hfill \\ \hfill p_q\hfill \\ \hfill {\mathit{p}}_{\mathit{z}}\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\nabla }_{\mathit{\theta }}f(\mathit{\theta })-q{\nabla }_{\mathit{\theta }}{c}_{eq}-{\mathit{D}}^T(\mathit{\theta })\mathit{z}\hfill \\ \hfill \mathit{S}\mathit{z}\hfill \\ \hfill c_{eq}\hfill \\ \hfill \mathit{c}(\mathit{\theta })-\mathit{s}\hfill \end{array}\right]$$ (16)

where

• ${\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}L(\mathit{\theta },\mathit{s},\mathit{z})={\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}f(\mathit{\theta })-{\sum }_{m=1}^{3M-2}{\nabla }_{\mathit{\theta }\mathit{\theta }}^2{c}_m(\mathit{\theta })z_i$ is the Hessian of the Lagrangian, and

• Z = diag(z).

We solve the system in (16) using the algorithm described in [30, 31, 32]. We approximate the Hessian of the Lagrangian using limited-memory Broyden-Fletcher-Goldfarb-Shanno (lBFGS) method [33]. lBFGS is a limited memory approximation of the BFGS method to be used in large scale optimization problems [25]. We provide the derivations of gradient ∇_θf(θ), Jacobian D(θ), and Hessian ${\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}L(\mathit{\theta },\mathit{s},\mathit{z})$ in the appendix. The dimension of unknowns, n, in our problem is O(N²) and, we know that lBFGS takes O(n²) time per iteration, hence the run time for the optimization algorithm per iteration is O(N⁴). The optimization method converges to a local minima due to the non-convexity of the proposed problem. Convex relaxations to this problem and convergence analysis is a topic of our future work.

2.4. Performance Analysis

In this section, we describe a metric that we define to demonstrate the performance of the proposed method. The metric we consider is the percentage of the pairwise inequalities violated in the low-dimensional space:

$$\tau =\frac{T_{\upsilon }}{T_t}*100$$ (17)

where T_t = M (M − 1)/2 is the total number of distance inequalities since we always choose to employ a chain of M − 1 inequalities and T_υ is the number of distance relationships violated in the projected space. In the proposed method, we compute T_t considering the estimated geodesic distance relationships observed in the original space. Recall that in our algorithm, for example if the geodesic distance $d_{i,j}^x$ is larger than $d_{k,l}^x$, we propose to preserve such relationships approximately for every data points indexed with i, j, k, l ∈ [1, …, N] in the projected space. We then accordingly compute T_υ by counting violated order relationships. According to the definition in (17), the lower the τ is, the more accurate the subspace projection is. We refer to this metric as constraint violation percentage (CVP).

Another commonly used performance metric is the residual variance between the estimated geodesic distances in the original space and the Euclidean distances in the projected space [7]. Defining ${\mathit{d}}_{\mathit{y}}={[d_{1,2}^y,\dots ,d_{N-1,N}^y]}^T$ and ${\mathit{d}}_{\mathit{x}}={[d_{1,2}^x,\dots ,d_{N-1,N}^x]}^T$ as two M × 1 vectors, the residual variance is computed using

$$R=1-\frac{\mathit{\text{cor}}({\mathit{d}}_{\mathit{x}},{\mathit{d}}_{\mathit{y}})}{\sqrt{{\sigma }_{d_y}^2{\sigma }_{d_x}^2}}$$ (18)

where

• ${\sigma }_{d_y}^2={\sum }_{i=1}^{N-1}{\sum }_{j=i}^N{(d_{i,j}^y-{\mathrm{d̅}}_y)}^2$, with d̅_y as the mean,

• ${\sigma }_{d_x}^2={\sum }_{i=1}^{N-1}{\sum }_{j=i}^N{(d_{i,j}^x-{\mathrm{d̅}}_x)}^2$, with d̅_x as the mean,

• $\mathit{\text{cor}}({\mathit{d}}_{\mathit{x}},{\mathit{d}}_{\mathit{y}})={\sum }_{i=1}^{N-1}{\sum }_{j=i}^N(d_{i,j}^x-{\mathrm{d̅}}_x)(d_{i,j}^y-{\mathrm{d̅}}_y)$.

3. Numerical Examples

We demonstrate the performance of the proposed method on synthetic datasets and compare with the following existing methods: Isomap [7], LTSA [9], LLE [10], SDE [11], LSML [12, 13], DM [14]. Penalty factor γ in (11) is taken as 1 and the scalar e_c that bounds the sum of w’s in (12) is selected as 10 in our experiments. α coefficient that indicate the importance of preserving an order constraints is taken as 1 for all order constraints. The variables that are being optimized are initialized randomly. The geodesic distance estimation is done with a knn-graph where k = 5. Similarly, neighborhood information is represented through knn-graphs with k = 5 in Isomap, LTSA, SDE and LSML methods. The number of neighbors for LLE is set as k = 5. We use the default values for all other parameters in the methods, that we compare against the proposed technique.

In the first set of experiments, we perform noise analysis on the synthetically generated growing band (GB) dataset with N = 50 samples, which contains a single dimensional manifold (see Figure 2(a)). For a given noise variance σ², the two dimensional dataset is generated by the following model

$$\mathbf{x}={[t\beta (t)]}^T,t\in [0,\rho ]$$ (19)

where ρ is selected as 10 and β(t) = 𝒩 (0, σ²t) is a Gaussian random variable with mean 0 and variance σ²t. For this data set we use different noise levels, σ² changing from 0.001 to 0.1. In this experiment, our goal is to explore possible non-linear relationship between the distances in the original and the projection spaces. The performance metrics we use are constraint violation percentage (CVP) and residual variance as described in Section 2.4. In Figure 2, and Table 1, we report noise analysis results. Figure 2 displays the CVP and residual variance for different values of the noise variance using the proposed technique and all other techniques. Figure 2(b) is a zommed in version of Figure 2(a) displaying the results for LSML, DM, Isomap and the proposed methods. The residual variance for LLE is omitted in Figure 2(c) for better visualization, since it produces very high values compared to the other methods. As can be seen the proposed method shows a better performance than the other methods with respect to both evaluation criteria. The method that perfoms similar to the proposed method is Isomap. All the other methods are local and they perform poorly due to the sparsity of the data. The performance difference between Isomap and the proposed method increases as the noise variance increases, because for low noise variances, the data resembles to a line more and the distance relations are close to being linear. Notice that the residual variance shows the same trend with CVP, since both metrics aim to quantify the consistency between distances, see also Table 1 for the summary of performance analysis results. Figure 1(b) shows the h(·) function as a result of the proposed optimization scheme for σ² = 0.1. The resulting fit does not show a linear relation, instead it maps the small distances to even smaller distances and the large distances to even larger ones. Our analysis shows that this kind of relationship outperforms the linear fit assumed by MDS.

Figure 2.

Noise analysis on GB dataset: (a) CVP on datasets with different noise variances using LTSA [9], LLE [10], SDE [11], LSML [12, 13], DM [14], Isomap [7] and the proposed methods, (b) Zoomed in version of (a) with LSML [12, 13], DM [14], Isomap [7] and the proposed methods for better visualization, (c) Comparative residual variance results. Residual variance for LLE is omitted since it gives very high values.

Table 1.

Performance analysis on GB dataset with different noise variances (σ²) (top), on spiral dataset (middle) and on Swiss roll dataset (bottom).

	σ2	LTSA	LLE	SDE	LSML	DM	Isomap	Proposed
# Violated Constraints	0.001	45856	10085	10500	7999	8204	4325	4738
	0.012	50324	17520	21111	11305	14250	8369	8140
	0.023	49974	29876	27048	18646	18306	13077	13154
	0.034	50137	18397	23712	22296	18881	12074	12126
	0.045	56904	42447	40638	32656	21800	16168	15380
	0.056	55594	29117	33005	22574	25582	19745	18766
	0.067	61210	255814	49481	29075	30130	24515	23671
	0.078	64284	34232	38870	33809	34714	27706	26834
	0.089	59952	32911	43207	29071	33610	24856	23735
	0.1	63429	41436	46387	36335	36975	32244	29907
Residual Variance	0.001	0.0320	0.9640	0.0002	0.0007	0.0001	0.00003	0.00004
	0.012	0.0392	0.9503	0.0066	0.0029	0.0035	0.0019	0.0019
	0.023	0.0376	0.9863	0.0103	0.0052	0.0045	0.0030	0.0029
	0.034	0.0411	0.9597	0.0082	0.0103	0.0049	0.00297	0.00299
	0.045	0.0479	0.9826	0.0242	0.0303	0.0082	0.0066	0.0063
	0.056	0.0460	0.9768	0.0165	0.0095	0.0110	0.0086	0.0080
	0.067	0.0548	1.0	0.0372	0.0161	0.0153	0.0129	0.0120
	0.078	0.0619	0.9611	0.0255	0.0201	0.0207	0.0166	0.0151
	0.089	0.0559	0.9626	0.0317	0.0169	0.0197	0.0158	0.0132
	0.1	0.0591	0.9838	0.0338	0.0230	0.0238	0.0210	0.0179

	LTSA	LLE	SDE	LSML	DM	Isomap	Proposed
#Violated Constraints	114430	210611	17138	16814	231680	8729	8519
CVP	0.1526	0.2809	0.0229	0.0224	0.3090	0.0116	0.0114
Residual Variance	0.2151	0.9063	0.0040	0.0049	0.78	0.0013	0.0010

	LTSA	LLE	SDE	LSML	DM	Isomap	Proposed
#Violated Constraints	1151299	440909	262700	183962	1281232	97043	68838
CVP	0.1807	0.0692	0.0412	0.0289	0.2011	0.0152	0.0108
Residual Variance	0.3646	0.0386	0.0136	0.0094	0.4917	0.0019	0.0011

Figure 1.

(a) Growing band dataset for noise variance σ² = 0.045. The 2D data samples are indicated with blue squares. X coordinates of red dots illustrate the original 1D manifold. (b) Resulting h(·) function on GB dataset with σ² = 0.1.

Secondly, we carry out experiments on a noisy spiral data with N = 50 samples (see Figure 3(a)). Resulting h(·) function is shown in Figure 3(b). The comparative results can be seen in Table 1. We outperform all other methods, but the performance of the proposed method is close to Isomap, and this can also be seen in h(·) function in Figure 3(b) which demonstrates an almost linear relationship. Even though we achieve an almost linear fitting, in the proposed method, we do not assume any predefined model in the algorithm, instead we learn this linear relationship as part of the solution. Other methods that give a comparable performance with the proposed method on this dataset are SDE and LSML.

Figure 3.

Spiral dataset: (a) Noisy data samples are shown with blue squares, red dots indicate the original samples before adding noise, (b) resulting h(·) function.

Thirdly, we perform experiments on the Swiss roll dataset with N = 85 samples (Figure 4(a)). 3D samples from the original space is projected to a 2D space in this experiment. Our goal is to demonstrate the performance of our method on a dataset having different curvature levels. Table 1 and Figure 5 report the comparative results. Also, resulting h(·) function that represents the relationship between d^x and d^ys is reported in Figure 4(b). Since data is synthetically generated, original 2D coordinates are known and are displayed in Figure 5(a). Figures 5(b) and 5(c) show the 2D projection results with Isomap and the proposed methods, respectively. Each sample is identified with an index number and the numbers are displayed on the nodes overlayed with Delaunay triangulation edges for better visualization. In the figure, we only display the projection result for Isomap, since it has the closest performance to our method on this dataset. Although the 2D projection with the proposed method seems rotated with respect to the original data, it preserves the distance relations better than the Isomap method, and the results in Table 1 supports this observation. Note that, knn-graph does not contain any shortcuts between different sides of the Swiss roll due to the small k value. However, the motivation of the method is not to overcome the limitations of the geodesic distance estimation. Instead, we would like to solve the general optimization problem given an estimated distance matrix. With this purpose, same parameters are used for local neighborhood definitions in all of the methods.

Figure 4.

Swiss roll dataset: (a) Data samples are shown with black dots overlayed with the knn-graph edges (red) where k = 5, (b) resulting h(·) function.

Figure 5.

2D representation of the Swiss roll dataset: (a) Original data in 2D, (b) Result with Isomap, and (c) Result with the proposed method.

Lastly, we carry out experiments on a retinal image dataset. Retinal images are widely used by doctors to follow, treat and diagnose various diseases. Retinopathy of prematurity (ROP) is among the diseases that can be diagnosed through the use of retinal images. It is a disease affecting low-birth weight infants, in which blood vessels in the retina of the eye develop abnormally and cause potential blindness. ROP is diagnosed from dilated retinal examination by an ophthalmologist, and may be successfully treated if diagnosed appropriately [34]. According to the international classification system, clinical ROP diagnosis has three classes, plus disease, pre-plus and neither [35]. Plus disease is a critical parameter which identifies severe ROP and is characterized by tortuosity of the arteries and dilation of the veins in the retina. Pre-plus represents vascular abnormalities insufficient for plus disease but with more arterial tortuosity and venous dilation than normal. Infants with plus disease require treatment to prevent blindness, whereas those without plus disease may be monitored by serial ophthalmic examination without treatment. Studies have found that clinical plus disease diagnosis is often subjective and qualitative, and that there is significant inconsistency even among experts [36, 37]. Hence, diagnosis of ROP is a vital and challenging task. In this experiment, we use a retinal image dataset, that consists of 34 images that are diagnosed by 22 experts [38]. Vessels are manually segmented in the images. Based on manual segmentation, we compute cumulative tortuosity for each center line point of vessels and employ mean and second central moment (CM2) of these values as features [39]. For a curve, cumulative tortuosity is defined as the ratio of the curve length to the distance between the two endpoints. As can be seen in the scatter plot in Figure 6(a), the features represent a one dimensional manifold with varying noise levels as in GB dataset. Figure 6(b) shows result of our manifold learning algorithm on the dataset along with some example images. Expert diagnostic decisions are also displayed on top of each image where the numbers represent the number of experts decided plus disease (+), pre-plus (±) and neither (−) respectively. The most important observation that we make from this figure is that the amount of tortuos vessels increases as we go through the manifold. Tortuosity plays an important role for ROP diagnosis. We also observe this role by analyzing the expert diagnostic decisions on the manifold such that the number of plus-disease decisions increases, while the number of neither decisions decreases as we trace the projected points from left to right. Moreover, the number of pre-plus decisions is high in the images in the middle of the projected curve compared to the images at the left and rightmost ends. In this example, preserving distance orders during dimensionality reduction is crucial, since tortuosity is a critical factor for ROP diagnosis.

Figure 6.

Experiment on retinal images. (a) scatter plot of the features: mean tortuosity and second central moment (CM2) of tortuosity, (b) result of the proposed method along with some example images indicated with green dots. Horizontal axis represents the projected points. The numbers above each image show the number of expert decisions plus disease (+), pre-plus (±) and neither (−) respectively. Notice that, amount of curvy vessels in images increases as we go from left to right. This correlates with the expert diagnostic decisions such that the number of plus disease decisions increases from left to right, while the number of neither decisions decreases. Moreover, the number of pre-plus decisions is high in the images in the middle of the manifold and lower for images at the left and rightmost ends.

4. Conclusion

We developed a nonlinear dimensionality reduction method that is a generalization of multidimensional scaling technique. Rather than using the common mean-squared error as an unconstrained cost function, we formulated a constrained optimization problem. In this problem, we assumed an unknown monotonic nondecreasing relationship modeled by radial basis function interpolation between the distances in the original and the projected spaces to be learned as a part of the manifold learning problem. Moreover, we incorporated explicit constraints on the distance orders in the projected space. Using interior-point methods, we solved this optimization problem, and in addition to obtaining the low-dimensional representation, we also learned the nonlinear relationship between the distances. We also proposed a new performance evaluation metric based on the number of the violated distance orders. Using this metric and residual variance as the performance measures, we compared our algorithm with other popular methods on synthetic datasets. The experiments demonstrated that the proposed method outperforms the other algorithms. The local algorithms performed poorly due to the sparsity of the data. The algorithm that performed close to the proposed method was Isomap, which is global and is a special case of our method. We applied the proposed algorithm on a real dataset where preserving distance orders is crucial for correct disease diagnosis. Future work includes extending the formulation of the optimization problem with convex relaxation to obtain faster solutions for larger datasets.

highlights.

• -A novel manifold learning method which preserves pairwise distance order relations in the projected space
• -Theoretical formulation of the constrained optimization problem extending classical MDS-based mean-squared error minimization
• -A new performance metric that involves number of preserved distance orders
• -Proposed method also provides the relation between distances in original and lower dimensional spaces

Appendix

In this appendix, we compute the gradient of the cost function, the Jacobian of the inequality constraints, and the Hessian of the Lagrangian that we define in (15) and (16). We first start with the derivation of the gradient ∇_θ f (θ). Following the definition of θ

$${\nabla }_{\mathit{\theta }}f(\mathit{\theta })=\left[\begin{array}{c}\hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}}f(\mathit{\theta })\hfill \\ \hfill {\nabla }_{\mathit{w}}f(\mathit{\theta })\hfill \\ \hfill {\nabla }_{\mathit{\xi }}f(\mathit{\theta })\hfill \end{array}\right].$$ (20)

Then rewriting $h(d_{i,j}^x)={\mathit{w}}^T\varphi (i,j)$ and $\gamma {\sum }_{m=1}^M{\alpha }_m{\xi }_m=\gamma {\mathit{\xi }}^T\mathit{\alpha }$, where w is defined in (7), ${\varphi }_{ij}={\left[\mathrm{\Phi }\right(\frac{d_{i,j}^x-d_{1,2}^x}{\sigma }),\dots ,\mathrm{\Phi }(\frac{d_{i,j}^x-d_{N-1,N}^x}{\sigma }\left)\right]}^T$, and α = [α₁, …, α_M−1], we compute

$${\nabla }_{\mathit{w}}f(\mathit{\theta })=\sum _{i=1}^{N-1}\sum _{j=i+1}^N-2[d_{i,j}^y-h(d_{i,j}^x)]{\varphi }_{ij}$$ (21a)

$${\nabla }_{\mathit{\xi }}f(\mathit{\theta })=\gamma \mathit{\alpha }$$ (21b)

Note here that ${\nabla }_{{\mathit{y}}_{\mathit{d}}}f(\mathit{\theta })={[\frac{\partial f(\mathit{\theta })}{\partial {\mathit{y}}_1^T},\dots ,\frac{\partial f(\mathit{\theta })}{\partial {\mathit{y}}_N^T}]}^T$, then

$$\frac{\partial f(\mathit{\theta })}{\partial {\mathit{y}}_k}=\sum _{i=1\text{and}i\ne k}^{N}2[\Vert {\mathit{y}}_i-{\mathit{y}}_k\Vert -h(d_{i,k}^x)]{\Vert {\mathit{y}}_i-{\mathit{y}}_k\Vert }^{-1}({\mathit{y}}_k-{\mathit{y}}_i)$$ (22)

Recall that D^T (θ) = [∇_θc₁(θ) ⋯ ∇_θc_3M−2(θ)], then using the definition of c(θ), we have D^T (θ) = [A^T ∇_θ g₁(θ) ⋯ ∇_θ g_M−1 (θ)]. Here ${\nabla }_{\mathit{\theta }}{g}_n(\mathit{\theta })={[{\nabla }_{{\mathit{y}}_{\mathit{d}}}^T{g}_n(\mathit{\theta }){\nabla }_{\mathit{w}}^T{g}_n(\mathit{\theta }){\nabla }_{\mathit{\xi }}^T{g}_n(\mathit{\theta })]}^T$. If $d_{n+1}=d_{k,l}^y$ and $d_n=d_{i,j}^y$, from Section 2.2, g_n (θ) = ‖ y_l − y_k ‖ − ‖ y_i − y_j ‖ + ξ_n, then ∇_wg_n (θ) = 0, ∇_ξ g_n (θ) = [0, …, 0, 1, 0, …, 0], a vector of zeros with 1 at the n^th location, and ${\nabla }_{{\mathit{y}}_{\mathit{d}}}{g}_n(\mathit{\theta })={[{\nabla }_{{\mathit{y}}_1}^T{g}_n(\mathit{\theta }),\dots ,{\nabla }_{{\mathit{y}}_N}^T{g}_n(\mathit{\theta })]}^T$. From the definition of g_n (θ), we have ∇_ym g_n(θ) = 0 for m = 1, …, N, and m ≠ i, j, k, l, and

$${\nabla }_{{\mathit{y}}_i}{g}_n(\mathit{\theta })=\frac{({\mathit{y}}_j-{\mathit{y}}_i)}{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert },{\nabla }_{{\mathit{y}}_j}{g}_n(\mathit{\theta })=\frac{({\mathit{y}}_i-{\mathit{y}}_j)}{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert },{\nabla }_{{\mathit{y}}_k}{g}_n(\mathit{\theta })=\frac{({\mathit{y}}_k-{\mathit{y}}_l)}{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert },{\nabla }_{{\mathit{y}}_l}{g}_n(\mathit{\theta })=\frac{({\mathit{y}}_l-{\mathit{y}}_k)}{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }$$ (23)

Recall the definition of Hessian ${\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}L(\mathit{\theta },\mathit{s},\mathit{z})={\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}f(\mathit{\theta })-{\sum }_{m=1}^{3M-2}{\nabla }_{\mathit{\theta }\mathit{\theta }}^2{c}_m(\mathit{\theta })z_i$, using this definition, we first demonstrate the computation of the Hessian of the cost function

$${\nabla }_{\mathit{\theta }\mathit{\theta }}^{2}f(\mathit{\theta })=\left[\begin{array}{ccc}\hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}{\mathit{y}}_{\mathit{d}}}^{2}f(\mathit{\theta })\hfill & \hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^{2}f(\mathit{\theta })\hfill & \hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^{2}f(\mathit{\theta })\hfill \\ \hfill {({\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^{2}f(\mathit{\theta }))}^T\hfill & \hfill {\nabla }_{\mathit{w}\mathit{w}}^{2}f(\mathit{\theta })\hfill & \hfill {\nabla }_{\mathit{w}\mathit{\xi }}^{2}f(\mathit{\theta })\hfill \\ \hfill {({\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^{2}f(\mathit{\theta }))}^T\hfill & \hfill {({\nabla }_{\mathit{w}\mathit{\xi }}^{2}f(\mathit{\theta }))}^T\hfill & \hfill {\nabla }_{\mathit{\xi }\mathit{\xi }}^{2}f(\mathit{\theta })\hfill \end{array}\right].$$ (24)

Then using (21), it is easy to show that ${\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^{2}f(\mathit{\theta })={\nabla }_{\mathit{w}\mathit{\xi }}^{2}f(\mathit{\theta })={\nabla }_{\mathit{\xi }\mathit{\xi }}^{2}f(\mathit{\theta })=\mathbf{0}$. Again from (21), we compute

$${\nabla }_{\mathit{w}\mathit{w}}^{2}f(\mathit{\theta })=\sum _{i=1}^{N-1}\sum _{j=i+1}^{N}2{\varphi }_{ij}^T{\varphi }_{ij}.$$ (25)

Noticing that ${\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^{2}f(\mathit{\theta })=\left[\begin{array}{c}\hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_1\partial \mathit{w}}\hfill \\ \hfill \cdots \hfill \\ \hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_N\partial \mathit{w}}\hfill \end{array}\right]$, we have

$$\frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_k\partial \mathit{w}}=\sum _{i=1\text{and}i\ne k}^N-2{\varphi }_{ik}^T{\Vert {\mathit{y}}_i-{\mathit{y}}_k\Vert }^{-1}({\mathit{y}}_k-{\mathit{y}}_i).$$ (26)

We can write ${\nabla }_{{\mathit{y}}_{\mathit{d}}{\mathit{y}}_{\mathit{d}}}^{2}f(\mathit{\theta })=\left[\begin{array}{ccc}\hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_1\partial {\mathit{y}}_1}\hfill & \hfill \cdots \hfill & \hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_1\partial {\mathit{y}}_N}\hfill \\ \hfill ·\hfill & \hfill ·\hfill & \hfill ·\hfill \\ \hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_N\partial {\mathit{y}}_1}\hfill & \hfill \cdots \hfill & \hfill \frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_N\partial {\mathit{y}}_N}\hfill \end{array}\right]$, then

$$\frac{{\partial }^{2}f(\mathit{\theta })}{\partial {\mathit{y}}_k\partial {\mathit{y}}_j}=({\Vert {\mathit{y}}_j-{\mathit{y}}_k\Vert }^{-1}-2)\mathit{I}-\frac{{({\mathit{y}}_j-{\mathit{y}}_k)}^T({\mathit{y}}_j-{\mathit{y}}_k)}{{\Vert {\mathit{y}}_j-{\mathit{y}}_k\Vert }^3}$$ (27)

To complete the derivation of the Hessian of the Lagrangian, we continue with the computation of ${\nabla }_{\mathit{\theta }\mathit{\theta }}^2{c}_m(\mathit{\theta })$ for m = 1, …, 3M − 2. Recalling the formula for the Jacobian of the inequality constraints we derive above, it is straightforward to show that ${\nabla }_{\mathit{\theta }\mathit{\theta }}^2{c}_m(\mathit{\theta })=\mathbf{0}$ for m = 1, …, 2M − 1, and ${\nabla }_{\mathit{\theta }\mathit{\theta }}^2{c}_m(\mathit{\theta })={\nabla }_{\mathit{\theta }\mathit{\theta }}^2{g}_m(\mathit{\theta })$ for m = 2M, …, 3M − 2 such that

$${\nabla }_{\mathit{\theta }\mathit{\theta }}^2{g}_m(\mathit{\theta })=\left[\begin{array}{ccc}\hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}{\mathit{y}}_{\mathit{d}}}^2{g}_m(\mathit{\theta })\hfill & \hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^2{g}_m(\mathit{\theta })\hfill & \hfill {\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^2{g}_m(\mathit{\theta })\hfill \\ \hfill {({\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^2{g}_m(\mathit{\theta }))}^T\hfill & \hfill {\nabla }_{\mathit{w}\mathit{w}}^2{g}_m(\mathit{\theta })\hfill & \hfill {\nabla }_{\mathit{w}\mathit{\xi }}^2{g}_m(\mathit{\theta })\hfill \\ \hfill {({\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^2{g}_m(\mathit{\theta }))}^T\hfill & \hfill {({\nabla }_{\mathit{w}\mathit{\xi }}^2{g}_m(\mathit{\theta }))}^T\hfill & \hfill {\nabla }_{\mathit{\xi }\mathit{\xi }}^2{g}_m(\mathit{\theta })\hfill \end{array}\right].$$ (28)

Then using (23) and the computation of the Jacobian matrix, we can show that ${\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{\xi }}^2{g}_m(\mathit{\theta })={\nabla }_{\mathit{w}\mathit{\xi }}^2{g}_m(\mathit{\theta })={\nabla }_{\mathit{\xi }\mathit{\xi }}^2{g}_m(\mathit{\theta })={\nabla }_{\mathit{w}\mathit{w}}^2{g}_m(\mathit{\theta })={\nabla }_{{\mathit{y}}_{\mathit{d}}\mathit{w}}^2{g}_m(\mathit{\theta })=\mathbf{0}$, and

$${\nabla }_{{\mathit{y}}_{\mathit{d}}{\mathit{y}}_{\mathit{d}}}^2{g}_m(\mathit{\theta })=\left[\begin{array}{ccc}\hfill {\nabla }_{{\mathit{y}}_1{\mathit{y}}_1}^2{g}_m(\mathit{\theta })\hfill & \hfill \cdots \hfill & \hfill {\nabla }_{{\mathit{y}}_1{\mathit{y}}_N}^2{g}_m(\mathit{\theta })\hfill \\ \hfill ·\hfill & \hfill ·\hfill & \hfill ·\hfill \\ \hfill {\nabla }_{{\mathit{y}}_N{\mathit{y}}_1}^2{g}_m(\mathit{\theta })\hfill & \hfill \cdots \hfill & \hfill {\nabla }_{{\mathit{y}}_N{\mathit{y}}_N}^2{g}_m(\mathit{\theta })\hfill \end{array}\right].$$ (29)

From (23) and the definition of g_n (θ) recall that ∇_ym g_n (θ) = 0 for m = 1, …, N, and m ≠ i, j, k, l, then ${\nabla }_{{\mathit{y}}_m{\mathit{y}}_{m\prime }}^2{g}_n(\mathit{\theta })=\mathbf{0}$, for m′ = 1, …, N, also ${\nabla }_{{\mathit{y}}_i{\mathit{y}}_k}^2{g}_n(\mathit{\theta })={\nabla }_{{\mathit{y}}_i{\mathit{y}}_l}^2{g}_n(\mathit{\theta })={\nabla }_{{\mathit{y}}_j{\mathit{y}}_k}^2{g}_n(\mathit{\theta })={\nabla }_{{\mathit{y}}_j{\mathit{y}}_l}^2{g}_n(\mathit{\theta })=\mathbf{0}$. We compute

$${\nabla }_{{\mathit{y}}_i{\mathit{y}}_i}^2{g}_n(\mathit{\theta })=\frac{-\mathit{I}{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^2+({\mathit{y}}_i-{\mathit{y}}_j){({\mathit{y}}_i-{\mathit{y}}_j)}^T}{{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^3}{\nabla }_{{\mathit{y}}_i{\mathit{y}}_j}^2{g}_n(\mathit{\theta })=\frac{\mathit{I}{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^2-({\mathit{y}}_j-{\mathit{y}}_i){({\mathit{y}}_j-{\mathit{y}}_i)}^T}{{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^3}{\nabla }_{{\mathit{y}}_j{\mathit{y}}_j}^2{g}_n(\mathit{\theta })=\frac{-\mathit{I}{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^2+({\mathit{y}}_j-{\mathit{y}}_i){({\mathit{y}}_j-{\mathit{y}}_i)}^T}{{\Vert {\mathit{y}}_i-{\mathit{y}}_j\Vert }^3}{\nabla }_{{\mathit{y}}_k{\mathit{y}}_k}^2{g}_n(\mathit{\theta })=\frac{\mathit{I}{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^2-({\mathit{y}}_k-{\mathit{y}}_l){({\mathit{y}}_k-{\mathit{y}}_l)}^T}{{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^3}{\nabla }_{{\mathit{y}}_k{\mathit{y}}_l}^2{g}_n(\mathit{\theta })=\frac{-\mathit{I}{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^2+({\mathit{y}}_k-{\mathit{y}}_l){({\mathit{y}}_k-{\mathit{y}}_l)}^T}{{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^3}{\nabla }_{{\mathit{y}}_l{\mathit{y}}_l}^2{g}_n(\mathit{\theta })=\frac{\mathit{I}{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^2-({\mathit{y}}_l-{\mathit{y}}_k){({\mathit{y}}_l-{\mathit{y}}_k)}^T}{{\Vert {\mathit{y}}_k-{\mathit{y}}_l\Vert }^3}.$$ (30)