Total Bregman Divergence and Its Applications to DTI Analysis

Abstract

Divergence measures provide a means to measure the pairwise dissimilarity between “objects,” e.g., vectors and probability density functions (pdfs). Kullback–Leibler (KL) divergence and the square loss (SL) function are two examples of commonly used dissimilarity measures which along with others belong to the family of Bregman divergences (BD). In this paper, we present a novel divergence dubbed the Total Bregman divergence (TBD), which is intrinsically robust to outliers, a very desirable property in many applications. Further, we derive the TBD center, called the t-center (using the ℓ₁-norm), for a population of positive definite matrices in closed form and show that it is invariant to transformation from the special linear group. This t-center, which is also robust to outliers, is then used in tensor interpolation as well as in an active contour based piecewise constant segmentation of a diffusion tensor magnetic resonance image (DT-MRI). Additionally, we derive the piecewise smooth active contour model for segmentation of DT-MRI using the TBD and present several comparative results on real data.

I. INTRODUCTION

In applications that involve measuring the dissimilarity between two objects (vectors, matrices, functions, images, and so on) the definition of a divergence/distance becomes necessary. A good divergence measure should be precise and statistically robust. The state of the art has many widely used divergences. The square loss function (SL) has been used widely for regression analysis; KL divergence [1], has been applied to compare the difference between two pdfs; and the Mahalanobis distance [2], is used to measure the dissimilarity between two random vectors of the same distribution. All the aforementioned divergences are special cases of the Bregman divergence which of late has been widely researched by many both from a theoretical and practical viewpoint, see [3]–[5] and references therein. At this juncture, it would be worth inquiring, why does one need yet another divergence? The answer would be that none of the existing divergences are statistically robust and one would need to use M-estimators from robust statistics literature to achieve robustness. This robustness however comes at a price, which is, computational cost and accuracy. Moreover, some of the divergences lack invariance to transformations such as similarity, affine, etc. Such invariance becomes important when dealing with for example, segmentation, it is desirable to achieve invariance to similarity or affine transformations that two different scans of the same patient might be related by.

In this work, we propose a new class of divergences which measure the orthogonal distance (in contrast to the ordinal distance used in Bregman divergence) between the value of a differentiable convex function at the first argument and its tangent at the second argument. We dub this divergence the total Bregman divergence (TBD). A geometrical illustration of the difference between TBD and BD is given in Fig. 1. d_f(x, y) is the Bregman divergence between x and y based on a differentiable convex function f and δ_f(x, y) is the TBD between x and y based on the same differentiable convex function f. We can observe that d_f(x, y) will change if we apply a rotation to the graph (x, f(x)), while δ_f(x, y) does not.

Fig. 1.

In each figure, d_f(x, y) (dotted line) is BD, δ_f(x, y) (bold line) is TBD. Left figure shows d_f(x, y) and δ_f(x, y) before rotating the coordinate system. Right figure shows d_f(x, y) and δ_f(x, y) after rotating the coordinate system. d_f(x, y) changes with rotation while δ_f(x, y) is invariant with rotation.

Bregman divergence has been widely used in clustering, a ubiquitous task in machine learning applications. In clustering, cluster centers are defined using either a distance or a divergence. In this paper, we will define a cluster center using the TBD in conjunction with the ℓ₁-norm that we dub the t-center. The t-center can be viewed as the cluster representative that minimizes the ℓ₁-norm TBD between itself and the members of a given population. We derive an analytic expression for the t-center which gives it a major advantage over its rivals (for example, the Riemannian median for SPD tensors). Since, the TBD can be viewed as a weighted BD with the weight being inversely proportional to the magnitude gradient of the convex function, the t-center can thus be viewed as a weighted median of sorts. This weighting scheme makes t-center robust to noise and outliers, since it’s inversely dependent on the gradient of the convex function. We prove the robustness property theoretically in Section III and experimentally demonstrate it in the experiments section. Another salient feature of the t-center is that it can be computed very efficiently due to its closed form expression leading to efficient clustering algorithms, a topic of our continuing research.

To illustrate the robustness and computational efficiency properties of TBD, we applied our theory to the space of order-2 symmetric positive definite (SPD) tensors which are commonly encountered in applications such as diffusion tensor magnetic resonance imaging (DT-MRI) [6], [7], computer vision (structure tensors), elastography, etc. We prove that TBD between SPD tensors is invariant to transformations from the special linear group (SL(n)). Recall that, if A ∈ ℝ^n×n, then A ∈ SL(n) implies det A = 1. Given random variables X and Y, where X ~ 𝒩(x; P) and Y ~ 𝒩(y; Q), 𝒩(․; P) and 𝒩(․; Q) are Gaussian distributions with zero mean and covariances P and Q, respectively, we can define BD(X, Y) and generalize it to define TBD(X, Y). We can show that TBD(X, Y) is only related to P and Q, thus, TBD(P, Q) = TBD(X, Y). If x ↦ Ax, then P ↦ A′PA, TBD has the property of TBD(A′PA, A′QA) = TBD(P, Q), ∀A ∈ SL(n).We prove that the ℓ₁ t-center, P*, for a population of SPD tensors {Q_i} has an analytic form; furthermore, t-centers P* and P̂* corresponding to the original and transformed tensors is related by A and P̂* = A′P* A. This property makes t-center suitable for atlas construction applications in tensor field processing, which is very useful in DT-MRI [8]–[11]. Atlases are usually defined as representatives (most often as an average of sorts) of a population of images, shapes, etc. Atlas construction using t-center is efficient and robust to outliers, a property that is derived theoretically and demonstrated experimentally in Section IV. Further, we present quantitative comparisons between the proposed and other existing divergences, such as the Frobenius norm [12], Riemannian metric [9], [10], [13]–[15], symmetrized KL divergence [14], [16], [17] and the log-Euclidean distance [18]. Although some of these divergences are invariant to transformations in SL(n), none of them are statistically robust to outliers encountered in tensor field processing. When using the Riemannian metric to compute population statistics, neither the mean nor the median can be computed in closed form, which makes this metric computationally very expensive. Finally, we perform segmentation on synthetic tensor fields and real DTI images using TBD and compare the results with other divergences demonstrating the efficiency and robustness of TBD to outliers over its rivals.

The rest of this paper is organized as follows. In Section II we review the conventional Bregman divergence, followed by the definition of TBD and derivation of its properties. Section III introduces the t-center, which is derived from TBD, and delves into its better accuracy as a representative than centers obtained from other divergence measures or metrics. The detailed description of the experimental design and results with quantitative comparison with other divergence are presented in Section IV. Finally, we draw conclusions in Section VI.

II. TOTAL BREGMAN DIVERGENCE

In this section, we first recall the definition of conventional Bregman divergence [4] and then define the TBD. Both divergences are dependent on the corresponding convex and differentiable function that f : X ↦ ℝ that induces the divergence.

A. Definition of TBD and Examples

Definition II.1: The Bregman divergence d associated with a real valued strictly convex and differentiable function f defined on a convex set X between points x, y ∈ X is given by

$$d_f(x,y)=f(x)-f(y)-\langle x-y,\nabla f(y)\rangle$$ (1)

where ∇f(y) is the gradient of f at y and 〈·, ·〉 is the inner product determined by the space on which the inner product is being taken.

For example, if f : ℝⁿ ↦ ℝ, then 〈·, ·〉 is just the inner product of vectors in ℝⁿ, and d_f(·, y) can be seen as the distance between the first order Taylor approximation to f at y and the function evaluated at x. Bregman divergence d_f is non-negative definite and does not satisfy the triangular inequality thus making it a divergence. As shown in Fig. 1, Bregman divergence measures the ordinate distance.

Definition II.2: The TBD δ_f associated with a real valued strictly convex and differentiable function f defined on a convex set X between points x, y ∈ X is defined as

$${\delta }_f(x,y)=\frac{f(x)-f(y)-\langle x-y,\nabla f(y)\rangle }{\sqrt{1+\Vert \nabla f(y){\Vert }^2}}$$ (2)

〈·, ·〉 is inner product as in definition II.1, and ‖∇f(y)‖² = 〈∇f(y), ∇f(y)〉 generally.

Consider the Bregman divergence d_f(η, ζ) = f(η) − f(ζ) − 〈η − ζ, ∇f(ζ)〉 obtained for strictly convex and differentiable generator f. Let x̂ denote the point (x, z = f(x)) of X × ℝ lying on the graph ℱ = {x̂ = (x, z = f(x))|x ∈ X}. We visualize the Bregman divergence as the z-vertical difference between the hyperplane H_ζ tangent at ζ̂ and the hyperplane parallel to H_ζ and passing through η̂

$$d_f(\eta ,\zeta )=f(\eta )-H_{\zeta }(\eta )$$ (3)

with H_ζ(x) = f(ζ) + 〈x − ζ, ∇f(ζ)〉. If instead of taking the vertical distance, we choose the minimum distance between those two hyperplanes H_η and H_ζ, we obtain the total Bregman divergence (by analogy to total least square fitting where the projection is orthogonal).

Since the distance between two parallel hyperplanes 〈x, a〉 + b₁ = 0 and 〈x, a〉 + b₂ = 0 is |b₁ − b₂|/‖a‖, letting a = (∇f(ζ), −1) and b₁ = f(η) − 〈η, ∇f(ζ)〉, and b₂ = f(ζ) − 〈ζ, ∇f(ζ)〉, we deduce that the total Bregman divergence is

$${\delta }_f(\eta ,\zeta )=\frac{d_f(\eta ,\zeta )}{\sqrt{1+\Vert \nabla f(\zeta ){\Vert }^2}}.$$ (4)

As shown in Fig. 1, TBD measures the orthogonal distance. Compared to the BD, TBD contains a weight factor (the denominator) which complicates the computations. However, this structure brings up many new and interesting properties and makes TBD an “adaptive” divergence measure in many applications. Note that, in practice, X can be an interval, the Euclidean space, a d-simplex, the space of non-singular matrices or the space of functions [19]. For instance, in the application to SPD tensor interpolation, we let p and q be two pdfs, and f(p) ≔ ∫ p log p, then δ_f(p, q) becomes what we call the Total Kullback–Leibler divergence (tKL). Note that if q is a pdf or a probability mass function (pmf), then ‖∇f(q)‖² = ∫(1+log q)²q. More on tKL later in Section IV. Table I lists some TBDs with various associated convex functions f.

TABLE I.

TBD δ_f Corresponding to f and the t-Center for a Set of Objects {x_i}. ẋ = 1 − x, ẏ = 1 − y, x′ is the Transpose of x. Δ^d is d-Simplex. c is the Normalization Constant to Make it a pdf, $w_i=1/\sqrt{1+{\Vert \nabla f(x_i)\Vert }^2}/({\displaystyle {\sum }_j}1/\sqrt{1+{\Vert \nabla f(x_j)\Vert }^2})$

X	f (x)	δf (x, y)	t-center	Remark
ℝ	x2	(x−y)2 (1+4y2)−1/2	∑i wixi	total square loss (tSL)
ℝ − ℝ−	x log x	$(x\text{ log }\frac{x}{y}-(x-y))/\sqrt{1+{(1+\text{log }y)}^2}$	∏i(xi)wi
[0, 1]	x log x + ẋ log ẋ	$(x\text{ log }\frac{x}{y}+\dot{x}\text{ log }\frac{\dot{x}}{\dot{y}})/\sqrt{1+y{(1+\text{log }y)}^2+\dot{y}{(1+\text{log }\dot{y})}^2}$	$\frac{{\displaystyle {\sum }_i}{({\mathbf{x}}_{\mathbf{i}}/(1-{\mathbf{x}}_{\mathbf{i}}))}^{w_i}}{1+{\displaystyle {\sum }_i}{({\mathbf{x}}_{\mathbf{i}}/(1-{\mathbf{x}}_{\mathbf{i}}))}^{w_i}}$	total logistic loss
ℝ+	− log x	$(\frac{x}{y}-\text{log}\frac{x}{y}-1)/\sqrt{1+y^{-2}}$	1/(∑i wi/xi)	total Itakura-Saito distance
ℝ	ex	$(e^x-e^y-(x-y)e^y)/\sqrt{1+e^{2y}}$	∑i wixi
ℝd	‖x‖2	${\Vert x-y\Vert }^2/\sqrt{1+4{\Vert y\Vert }^2}$	∑i wixi	total squared Euclidean distance
ℝd	x′ Ax	$((x-y)\prime A(x-y))/\sqrt{1+4{\Vert \mathit{\text{Ay}}\Vert }^2}$	∑i wixi	total Mahalanobis distance
Δd	${\displaystyle {\sum }_{j=1}^d}{x}_j\text{ log }{x}_j$	$\left({\displaystyle {\sum }_{j=1}^d}{x}_j\text{ log }\frac{x_j}{y_j}\right)/\sqrt{1+{\displaystyle {\sum }_{j=1}^d}{y}_j{(1+\text{log }{y}_j)}^2}$	c∏i(xi)wi	total KL divergence (tKL)
ℂm × n	${\Vert \mathbf{x}\Vert }_F^2$	${\Vert x-y\Vert }_F^2/\sqrt{1+4{\Vert y\Vert }_F^2}$	∑i wixi	total squared Frobenius

III. t-CENTER

In many applications of computer vision and machine learning such as image and shape retrieval, clustering, classification, etc., it is common to seek a representative for a set of objects having similar features. This representative normally is a cluster center of sorts, thus, it is desirable to seek a center that is robust to outliers and is efficient to compute. In this section, we use the TBD to derive an ℓ₁-norm cluster center that we call the t-center, and explore its properties.

A. Definition

Let f : X ↦ ℝ be a convex and differentiable function and E = {x₁, x₂, ⋯, x_n} be a set of n points in X, then, the TBD ${\delta }_f^p$ between a point x ∈ X and E associated with f and ℓ_p norm is defined as

$${\delta }_f^p(x,E)={\left({\displaystyle \sum _{i=1}^n}{({\delta }_f(x,x_i))}^p\right)}^{1/p}.$$ (5)

The ℓ_p-norm center $x_p^{*}$ is defined as $x_p^{*}={\text{arg min}}_x{\delta }_f^p(x,E)$. These centers bear the name of median (p = 1), mean (p = 2) and circumcenter (p → ∞). In this paper, we call the median (p = 1) the t-center and will derive an analytic form for the t-center and focus on its applications. Before delving into the t-center, we define the circumcenter and the centroid (mean).

B. ℓ_∞-Norm Circumcenter x_c

In the limiting case, the circumcenter amounts to solving for

$$x_c=\text{arg}\underset{x}{\text{ min }}\underset{i}{\text{max }}{\delta }_f(x,x_i).$$ (6)

x_c does not have a closed form, even though it can be solved to an arbitrarily fine approximation in Riemannian cases [5].

C. ℓ₂-Norm Centroid x_m

$$x_m=\text{arg }\underset{x}{\text{min}}\sqrt{{\displaystyle \sum _{i=1}^N}{({\delta }_f(x,x_i))}^2}.$$ (7)

Like the ℓ_∞-norm circumcenter, ℓ₂-norm mean does not have a closed form in general and is hence expensive to compute.

D. The ℓ₁-Norm t-Center

Given a set E, we can obtain the ℓ₁-norm t-center x* of E by solving the minimization problem

$$x^{*}=\text{arg}\underset{x}{\text{ min }}{\delta }_f(x,E)=\text{arg }\underset{x}{\text{min}}{\displaystyle \sum _{i=1}^n}{\delta }_f(x,x_i)$$ (8)

Theorem III.1: The t-center of a population of objects (densities, vectors, etc.) with respect to a given divergence exists, and is unique.

Proof: To find x*, we take the derivative of $F(x)={\displaystyle {\sum }_{i=1}^n}{\delta }_f(x,x_i)$ with respect to x, and set it to 0, as in

$$\nabla F(x)={\displaystyle \sum _{i=1}^n}\frac{\nabla f(x)-\nabla f(x_i)}{\sqrt{1+\Vert \nabla f(x_i){\Vert }^2}}=0.$$ (9)

Solving (9) yields

$$\nabla f(x^{*})=({\displaystyle \sum _{i=1}^n}{w}_i\nabla f(x_i))/\left({\displaystyle \sum _{i=1}^n}{w}_i\right)$$ (10)

where w_i = (1 + ‖∇f(x_i)‖²)^−1/2 is the weight that depends on ∇f(x_i). Recall that TBD δ_f(·, y) is convex for any fixed y ∈ X, we know that $F(x)={\delta }_f^1(x,E)$ is also convex as it is the sum of n convex functions. Hence, the solution to (9) exists and is indeed a minimizer of F(x). Furthermore, since f is convex, ∇f is monotonic, so x* is unique.

To obtain an explicit formula for x*, we will make use of the dual space. Suppose g is the legendre dual function of f in the space X, and the dual space is denoted as Y. Then ∀x ∈ X, ∃y ∈ Y

$$g(y)=\underset{x}{\text{sup }}\{\mathit{\text{yx}}-f(x)\},\text{and }f(x)=\underset{y}{\text{sup }}\{\mathit{\text{xy}}-g(y)\}$$ (11)

where x and y satisfy y = ∇f(x) and x = ∇g(y). Thus far, we know that the t-center x* satisfies (10), meaning that its legendre dual y* should satisfy $y^{*}=({\displaystyle {\sum }_{i=1}^n}{w}_i\nabla f(x_i))/({\displaystyle {\sum }_{i=1}^n}{w}_i)$, and x* = ∇g(y*).

Note that, in most cases, if f is given, then we can get the explicit form for x*, but x* does not have the same expression for all convex functions, instead, x* is dependent on the convex generator function for the chosen divergence. For a better illustration of theorem III.1, we provide three concrete examples of TBD with their t-centers in analytic form. The first is for the square loss function that we call total square loss (tSL), the second is for the exponential function and the third is the total Kullback–Leibler (tKL) as a measure of divergence between probability densities.

• tSL: f(x) = x², the t-center x* = (∑_i w_ix_i)/(∑_i w_i), where $w_i={(1+4x_i^2)}^{-1/2}$;

• Exponentials: f(x) = e^x, the t-center x* = (∑_i w_ix_i)/(∑_i w_i), where w_i = (1 + e^2x_i)^−1/2;

• tKL: f(q) = ∫ q log qdx, which is the negative entropy [1], and E = {q₁, q₂, ⋯, q_n} is a set of pdfs, the t-center

  $$q^{*}=\text{arg }\underset{q}{\text{min}}{\displaystyle \sum _{i=1}^n}\frac{{\displaystyle {\int }_X}q\text{ log }\frac{q}{q_i}\mathit{\text{dx}}}{\sqrt{1+{\displaystyle {\int }_X}{(1+\text{log }{q}_i)}^2{q}_i\mathit{\text{dx}}}}.$$ (12)

  q* is obtained by solving the Euler–Lagrange equation of the minimization in (12), to obtain

  $$q^{*}=c{\displaystyle \prod _i^n}{q}_i^{w_i},w_i=\frac{{\mu }_i}{{\displaystyle {\sum }_j}{\mu }_j},{\mu }_i=\frac{1}{1+{\displaystyle \int }{(1+\text{log}{q}_i)}^2{q}_i\mathit{\text{dx}}}$$

  where c is a normalization constant such that ∫_X q*(x)dx = 1. Furthermore, the t-centers for the most commonly used divergences can be derived in analytic form as shown in the Table I.

Theorem III.2: The t-center is statistically robust to outliers.

Proof: The robustness of t-center is analyzed by the influence function of an outlier y. Let x* be the t-center of E = {x₁, ⋯, x_n}. When ε% (ε small) of outlier y is mixed with E, x is influenced by the outliers, and the new center becomes to x̃* = x* + εz(y). We call z(y) the influence function. The center is robust when z(y) does not grow even when y is very large. The influence curve is given explicitly in the case of ℓ₁-norm. x̃* is the minimizer of

$$(1-\epsilon )\frac{1}{n}{\displaystyle \sum _{i=1}^n}\delta (x,x_i)+\delta (x,y).$$ (13)

Hence, by differentiating the above function, setting it equal to zero at x̃* = x* + εz and using the Taylor expansion, we have

$$z(y)=\frac{G^{-1}}{w(y)}(\nabla f(y)-\nabla f(x^{*}))$$ (14)

where $G=(1/n){\displaystyle {\sum }_{i=1}^n}\nabla \nabla f(x^{*})w(x_i),\text{ and }w(y)=\sqrt{1+{\Vert \nabla f(y)\Vert }^2}$. Hence, the t-center is robust when z is bounded, i.e., ∀y, |z| < c, c is a constant

$$|z(y)|<\frac{G^{-1}}{w(y)}(\Vert \nabla f(y)\Vert +\Vert \nabla f(x^{*})\Vert ).$$ (15)

Since w(y) ≥ ‖∇f(y)‖ and w(y) ≥ 1, therefore

$$|z(y)|<G^{-1}(1|+\Vert \nabla f(x^{*})\Vert )=c.$$ (16)

To make this proof more understandable, we give a simple example of the Euclidean case when f(x) = (1/2) ‖x‖², x ∈ ℝ^d. Then

$$G=\frac{1}{n}{\displaystyle \sum _{i=1}^n}\frac{1}{\sqrt{1+\Vert x_i{\Vert }^2}}I$$ (17)

and $z(y)=G^{-1}(y/\sqrt{1+{\Vert y\Vert }^2})$, when ‖y‖ is large, this is approximated by z(y) = G⁻¹(y/‖y‖), which implies z is bounded, and the t-center is robust for large ‖y‖.

Note that the influence function for the ordinary Bregman divergence is z = y, and hence is not robust.

Using the ℓ₁-norm t-center x* has advantages over other centers resulting from the norms with p > 1 in the sense that x* has an analytic form and is robust to outliers. These advantages are explicitly evident in the experiments presented subsequently.

IV. TBD APPLICATIONS

We now develop the applications of the t-center in interpolating diffusion tensors in DT-MRI data and segmentation of tensor fields specifically DT-MRI.

A. SPD Tensor Interpolation Applications

We now define tKL, the total Kullback–Leibler divergence between symmetric positive definite (SPD) rank-2 tensors (SPD matrices), and show that it is invariant to transformations belonging to the special linear group SL(n). Further, we compute the t-center using tKL for SPD matrices, which has a closed form expression, as the weighted harmonic mean of the population of the tensors.

tKL between order two SPD tensors/matrices P and Q is derived using the negative entropy of the zero mean Gaussian density functions they correspond to. Note that order two SPD tensors can be seen as covariance matrices of zero mean Gaussian densities. Suppose

$$p(x;P)=\frac{1}{\sqrt{{(2\pi )}^n\text{det }P}}\text{exp }(-\frac{1}{2}x\prime P^{-1}x)$$ (18)

$$q(x;Q)=\frac{1}{\sqrt{{(2\pi )}^n\text{det }Q}}\text{exp }(-\frac{1}{2}x\prime Q^{-1}x)$$ (19)

then

$$\mathit{\text{tKL}}(P,Q)=\frac{{\displaystyle \int }p\text{ log }\frac{p}{q}\mathit{\text{dx}}}{\sqrt{1+{\displaystyle \int }{(1+\text{ log }q)}^{2}q\mathit{\text{dx}}}}=\frac{\text{log }(\text{det}(P^{-1}Q))+\mathit{\text{tr}}(Q^{-1}P)-n}{2\sqrt{c+\frac{{(\text{log}(\text{det }Q))}^2}{4}-\frac{n(1+\text{log }2\pi )}{2}\text{log}(\text{det }Q)}}$$ (20)

where c = (3n/4) + (n² log 2π/2) + ((n log 2π)²/4). When an SL(n) transformation is applied on x, i.e., x ↦ Ax, then P ↦ A′PA and Q ↦ A′QA. It is easy to see that

$$\mathit{\text{tKL}}(P,Q)=\mathit{\text{tKL}}(A\prime \mathit{\text{PA}},A\prime \mathit{\text{QA}}),\forall A\in \mathit{\text{SL}}(n)$$ (21)

which means that tKL between SPD tensors is invariant under the group action, when the group member belongs to SL(n). Given an SPD tensor set ${\{Q_i\}}_{i=1}^m$, its t-center P* can be obtained from (12) and

$$P^{*}={\left({\displaystyle \sum _i}{w}_i{Q}_i^{-1}\right)}^{-1},w_i=\frac{{\mu }_i}{{{\displaystyle \sum }}_j{\mu }_j}$$ (22)

where ${\mu }_i={(2\sqrt{c+({(\text{log}(\text{det}{Q}_i))}^2/4)-(n(1+\text{log}2\pi )/2)\text{log}(\text{det}{Q}_i)})}^{-1}$. It can be seen that w_i(Q_i) = w_i(A′Q_iA), ∀A ∈ SL(n). If a transformation A ∈ SL(n) is applied, the new t-center will be P̂* = (∑ w_i(A′Q_iA)⁻¹)⁻¹ = A′P* A, which means that if ${\{Q_i\}}_{i=1}^m$ are transformed by some member of SL(n), then the t-center will undergo the same transformation.

Also we can compute the tSL between P and Q from the density functions p, q and f(p) = ∫ p²dx, and the result is

$$\mathit{\text{tSL}}(P,Q)=\frac{{\displaystyle \int }{(p-q)}^2\mathit{\text{dx}}}{\sqrt{1+{\displaystyle \int }{(2q)}^2\mathit{\text{qdx}}}}=\frac{1/\sqrt{\text{det}(2P)}+1/\sqrt{\text{det}(2Q)}-2/\sqrt{\text{det}(P+Q)}}{{(2\pi )}^n+4\sqrt{{(2\pi )}^n}/\sqrt{\text{det}(3Q)}}$$

also tSL(P, Q) = tSL(A′PA, A′QA), ∀A ∈ SL(n), which means that tSL is also invariant under SL(n) transformations. Similarly, we can also prove that the total Itakura-Saito distance and total squared Euclidean distance between SPD matrices are invariant under SL(n) transformations. For the rest of this paper, we will focus on tKL.

There are several ways to define the distance between SPD matrices, e.g. using the Frobenius norm [12], Riemannian metric [10], [13]–[15], symmetrized KL divergence [14], [16], [17] and the log-Euclidean distance [18], respectively, defined as

$$d_F(P,Q)={\Vert P-Q\Vert }_F=\sqrt{\mathit{\text{tr}}((P-Q)\prime (P-Q))}$$ (23)

$$d_R(P,Q)=\sqrt{{\displaystyle \sum _{i=1}^n}{\text{log}}^2{\lambda }_i}$$ (24)

where λ_i, i = 1, ⋯, n, are eigenvalues of P⁻¹Q

$${\mathit{\text{KL}}}_s(P,Q)=\frac{1}{4}\mathit{\text{tr}}(Q^{-1}P+P^{-1}Q-2I)$$ (25)

$$\mathit{\text{LE}}(P,Q)={\Vert \text{log}(P)-\text{log}(Q)\Vert }_F.$$ (26)

d_F (․, ․) is not invariant to transformations in SL(n), d_R(․, ․), KL_s(․, ․) and LE(․, ․) are invariant to transformations in GL(n), but none of them are robust to outliers encountered in the data for e.g., in tensor interpolation. For d_R(․, ․), neither its mean nor its median is in closed form, which makes it computationally very expensive as the population size and the dimensionality of the space increases. Here, the Karcher mean denoted by $P_M^{*}$ is defined as the minimizer of the sum of squared Riemannian distances and the median $P_R^{*}$ is minimizer of the sum of Riemannian distances [9]

$$P_M^{*}=\text{arg}\underset{P}{\text{min}}{\displaystyle \sum _{i=1}^m}{d}_R^2(P,Q_i)\text{ and }{P}_R^{*}=\text{arg}\underset{P}{\text{min}}{\displaystyle \sum _{i=1}^m}{d}_R(P,Q_i).$$

For simplicity in notation, we denote $d_R^2(P,Q_i)$ by d_M(P, Q_i), and $P_M^{*}={\text{arg min}}_P{\displaystyle {\sum }_{i=1}^m}{d}_M(P,Q_i)$. Even though there are many algorithms (for example [8]–[10], [15]) to solve the geodesic mean and median, most of them adopt an iterative (gradient descent) method. Performing gradient descent on matrix manifolds can be tricky and rather complicated in the context of convergence issues (see [20]) and hence is not preferred over a closed form computation.

KL_s in (25) has a closed form mean [14], [16], which is 𝒢(𝒜(Q₁, ⋯, Q_m), ℋ(Q₁, ⋯, Q_m), where 𝒜 is the arithmetic mean of ${\{Q_i\}}_{i=1}^m$, ℋ is the harmonic mean of ${\{Q_i\}}_{i=1}^m$, and 𝒢 is the geometric mean of 𝒜 and ℋ. However, it can be shown that the mean computed using sum of squared KL_s divergences is not statistically robust since all tensors are treated equally. This is because, neither KL_s nor the sum of squared KL_ss are robust functions. This is demonstrated in the experimental results to follow. First we will observe the visual difference between the aforementioned divergences/distances. Fig. 2 shows the isosurfaces centered at the identity matrix with radii r = 0.1, 0.5 and 1 respectively. From left to right are d_F(P, I) = r, d_R(P, I) = r, KL_s(P, I) = r and tKL(P, I) = r. These figures indicate the degree of anisotropy in the divergences/distances.

Fig. 2.

The isosurfaces of d_F (P, I) = r, d_R (P, I) = r, KL_s (P, I) = r and tKL (P, I) = r shown from left to right. The three axes are eigenvalues of P.

B. Piecewise Constant DTI Segmentation

Given a noisy diffusion tensor image (DTI) T₀(x)—a field of positive definite matrices, our model for DTI segmentation is based on the Mumford–Shah functional [21]

$$E(C,T)=\alpha {\displaystyle \underset{\mathrm{\Omega }}{\int }}\delta (T_0(x),T(x))\mathit{\text{dx}}+{\displaystyle \underset{\mathrm{\Omega }-C}{\int }}|\nabla T(x)|\mathit{\text{dx}}+\beta |C|$$ (27)

where α and β are control parameters, δ is the tKL divergence same as (20). Ω is the region of the tensor field, T(x) is an approximation to T₀(x), which can be discontinuous only along C. However, in a simplified segmentation model, a field T₀(x) can be represented by piecewise constant regions [21], [22]. Therefore, we consider the following binary segmentation model for DTI:

$$E(C,T_1,T_2)={\displaystyle \underset{R}{\int }}\delta (T_0(x),T_1)\mathit{\text{dx}}+{\displaystyle \underset{R^c}{\int }}\delta (T_0(x),T_2)\mathit{\text{dx}}+\beta |C|$$ (28)

T₁ is the t-center of DTI for the region R inside the curve C and T₂ is the t-center of the DTI for the region R^c outside C

$$T_1={\left({\displaystyle \sum _{Q_i\in R}}{w}_i{Q}_i^{-1}\right)}^{-1},w_i=\frac{{\mu }_i}{{\displaystyle {\sum }_{Q_j\in R}}{\mu }_j}.$$ (29)

$$T_2={\left({\displaystyle \sum _{Q_i\in R^c}}{w}_i{Q}_i^{-1}\right)}^{-1},w_i=\frac{{\mu }_i}{{\displaystyle {\sum }_{Q_j\in R^c}}{\mu }_j}.$$ (30)

The Euler Lagrange equation of (28) is

$$(\delta (T_0(x),T_1)-\delta (T_0(x),T_2)+\beta \kappa )\mathbf{N}=0$$ (31)

where κ is the curvature, κ = ∇ · (∇C/|∇C|), and N is the normal of C, N = ∇C. C can be updated iteratively according to the following equation:

$$\frac{\partial C}{\partial t}=-(\delta (T_0(x),T_1)-\delta (T_0(x),T_2)+\beta \kappa )\mathbf{N}.$$ (32)

At each iteration, we will fix C, update T₁ and T₂ according to (29) and (30), and then freeze T₁ and T₂ to update C.

In the level set formulation of the active contour [23], let ϕ be the signed distance function of C and choose it to be negative inside and positive outside. Then the curve evolution (32) can be reformulated using the level set framework

$$\frac{\partial \varphi }{\partial t}=(\delta (T_0(x),T_1)-\delta (T_0(x),T_2)+\beta \nabla \cdot \frac{\nabla \varphi }{|\nabla \varphi |}|\nabla \varphi |.$$ (33)

C. Piecewise Smooth DTI Segmentation

For complicated DTI images, the piecewise constant assumption does not hold. Therefore, we have to turn to the more general model, the piecewise smooth segmentation. In this paper, we follow Wang et al.’s [16] model but replace their divergence (KL_s) with tKL, resulting in the following functional:

$$E(C,T)=\alpha {\displaystyle \underset{R}{\int }}\delta (T_0(x),T_R(x))\mathit{\text{dx}}+{\displaystyle \underset{R^c}{\int }}\delta (T_0(x),T_{R^c}(x))\mathit{\text{dx}}+{\displaystyle \underset{\mathrm{\Omega }-C}{\int }}p(T)(x)\mathit{\text{dx}}+\beta |C|$$

the third term measures the lack of smoothness of the field using the Dirichlet integral [24], giving us the following curve evolution equation:

$$\frac{\partial C}{\partial t}=\left(\alpha \delta (T_0(x),T_{R^c})+{\displaystyle \sum _{y\in N_{R}c(x)}}\delta (T_{R^c},T_{R^c}(y)\right)\mathbf{N}-(\alpha \delta (T_0(x),T_R)+{\displaystyle \sum _{y\in N_R(x)}}\delta (T_R,T_R(y))-\beta \kappa )\mathbf{N}$$

where N_R and N_R^c are the sets of x’s neighboring pixels inside and outside of the region R respectively. In the discrete case, one can use appropriate neighborhoods for 2-D and 3-D. We apply the two stage piecewise smooth segmentation algorithm in [16] to numerically solve this evolution.

V. EXPERIMENTAL RESULTS

We performed two sets of experiments here: 1) tensor interpolation and 2) tensor field segmentation. The experimental results are compared with those obtained by using other divergence measures discussed above.

A. Tensor Interpolation

SPD tensor interpolation is a crucial component of DT-MRI analysis involving segmentation, registration and atlas construction [25], where a robust distance/divergence measure is very desirable. First, we perform tensor interpolation on a set of tensors with noise and outliers. We fix an SPD tensor Ḡ as the ground truth tensor and generate noisy tensors G from it by using a the Monte Carlo simulation method as was done in [26], [27]. This entails, given a b-value (we used 1200 s/mm²) a zero gradient baseline image S̄₀, and six vectors ${\{q_i\}}_{i=1}^6$, the magnitude of the noise free complex-valued diffusion weighted signal is given by, ${\overline{S}}_i={\overline{S}}_0\text{exp}(-{\mathit{\text{bq}}}_i^{\prime }\overline{G}{q}_i)$. We add Gaussian distributed (mean zero and variance σ² = 0.1) noise to the real and imaginary channels of this complex valued diffusion weighted signal and take its magnitude to get ${\overline{S}}_i={\overline{S}}_0\text{exp}(-{\mathit{\text{bq}}}_i^{\prime }{\mathit{\text{Gq}}}_i)$. S_i then has a Rician distribution with parameters (σ, S̄_i). G is obtained by fitting the ${\{S_i\}}_{i=1}^6$ using Log-Euclidean tensor fitting. To generate outliers, we first generate a 3 × 3 matrix Z, with each entry drawn from the normal distribution, and the outlier tensor G is given by G = exp(ZZ′). We compute the eigen-vectors U and eigen-values V of G, i.e. G = U diag (V)U′, and then rotate U by a rotation matrix r(α, β, γ) in 3-D, to get U_o, where α, β and γ are uniformly distributed in [0, π/2]. Then the outlier is given by $G_o=U_o\mathit{\text{diag}}(\mathit{\text{randperm}}(V))U_o^{\prime }$. To generate an image, the same process is repeated at every randomly chosen voxel.

The tKL t-center, the geometric median, the geometric mean, the KL_s mean and the log-Euclidean mean M for 21 SPD tensors along with 0, 5, 10, 15, and 20 outliers are then computed. The difference between the various means/medians, and the ground truth Ḡ are measured in four ways, using the Frobenius norm ‖Ḡ − M‖_F, the ℓ_∞ distance ‖Ḡ − M‖_∞, angle between the principal eigenvectors of M and Ḡ, and the difference of fractional anisotropy index (FA) ([28], [29]) for M and Ḡ, i.e., |FA(M) − FA(Ḡ)|. The results are shown in Table II from left to right, top to bottom.

TABLE II.

Frobenius Norm Distance d_F, ℓ_∞ Distance, Angle Between the Principle Eigenvectors of the Ground Truth and the Eigenvector of the Mean/Median Found by the Divergences, and the Difference Between the FA of the Ground Truth and the Computed Mean/Median are Shown From Left to Right, Top to Bottom. There are 21 Original Tensors and the Number of Outliers is Shown in the Top Row of Each Table

Errors in computed mean measured by Frobenius norm dF							Errors in computed mean measured by ℓ∞
# of outliers	0	5	10	15	20		# of outliers	0	5	10	15	20
tKL	0.8857	1.0969	1.1962	1.3112	1.4455		tKL	0.6669	0.8687	1.0751	1.1994	1.5997
dR	1.3657	1.4387	1.5348	1.6271	1.7277		dR	0.8816	1.3498	1.5647	1.8826	2.1904
dM	1.4291	1.5671	1.8131	2.1560	2.5402		dM	1.3056	1.4160	1.6755	1.9390	2.5153
KLs	3.4518	3.5744	3.7122	4.1040	4.3235		KLs	4.1738	4.3352	4.3463	4.3984	4.4897
LE	1.0638	1.5721	1.6249	1.6701	1.8227		LE	0.8768	1.2843	1.6947	1.9418	2.2574
Errors in computed angle							Errors in computed FA
# of outliers	0	5	10	15	20		# of outliers	0	5	10	15	20
tKL	0.1270	0.3875	0.6324	1.8575	2.9572		tKL	0.0635	0.0738	0.0753	0.0792	0.0857
dR	0.4218	0.9058	8.9649	17.4854	44.9157		dR	0.0983	0.1529	0.1857	0.1978	0.2147
dM	0.8147	1.1985	10.1576	21.8003	43.7922		dM	0.1059	0.1976	0.2008	0.2193	0.2201
KLs	0.8069	0.9134	14.9706	26.1419	44.9595		KLs	0.0855	0.1675	0.2005	0.2313	0.2434
LE	0.6456	0.9373	8.9937	17.0636	44.7635		LE	0.0972	0.1513	0.1855	0.1968	0.2117

From the tables, we can see that the t-center yields the best approximation to the ground truth, and faithfully represents the directionality and anisotropy. The robustness of t-center over others such as the Riemannian mean and median, KL_s mean, and log-Euclidean mean is quite evident from this table. Even though geometric median seems to be competitive to the t-center obtained using tKL in the case of lower percentage of outliers, the geometric median computation however is much slower than that of t-center computation. This is because the t-center for tKL has a closed form while the geometric median does not. Table III shows the CPU time to find the tensor mean/median using different divergences. We use 1000 SPD tensors, along with 0, 10, 100, 500, and 800 outliers respectively. All tensors are generated in the same way as described in the first experiment. The time is averaged by repeating the experiment 10 times on a PC, with Intel Core 2 Duo CPU P7370, 2 GHz, 4 GB RAM, on 32-bit Windows Vista OS. Table II and III depict the superior robustness to outliers and the computational efficiency in estimating the tKL t-center for SPD tensor interpolation in comparison to its rivals.

TABLE III.

Time (Seconds) Spent in Finding the Mean/Median Using Different Divergences. There are 1000 Original Tensors and the Number of Outliers is Shown in the Top Row

time
Divergences\# of outliers	0	10	100	500	800
tKL	0.02	0.02	0.02	0.03	0.03
KLs	0.03	0.03	0.03	0.04	0.04
dR	0.58	1.67	4.56	93.11	132.21
dM	0.46	1.42	3.13	72.74	118.05
LE	0.02	0.02	0.03	0.03	0.03

B. Tensor Field Segmentation

We now describe experiments on segmentation of synthetic and real DTI images.

1. Segmentation of Synthetic Tensor Fields: The first synthetic tensor field is composed of two types of homogeneous tensors. Fig. 3 depicts the synthetic data and the segmentation results. We added different levels of noise to the tensor field using the method described in Section V-A, segmented it using the aforementioned divergences and compared the results using the dice coefficient [30]. We also added different percentages of outliers to the tensor fields and segmented the resulting tensor fields. Fig. 4 depicts the comparison of segmentation results from different methods using the dice coefficient, for varying noise levels with σ varying from 0, 0.02, 0.04, ⋯, 0.2). Fig. 5 displays the comparison of dice coefficient with different percentage (0, 5, 10, ⋯, 50) of outliers. The results show that even in the presence of large amounts of noise and outliers, tKL yields very good segmentation results in comparison to rivals. However, in our experiments, we observed that the segmentation accuracy is inversely proportional to the variance of the outlier distribution.

2. Segmentation of DTI Images: In this section, we present segmentation results on real DTI images from a rat spinal cord, an isolated rat hippocampus and a rat brain. The data were acquired using a PGSE with TR = 1.5 s, TE = 28.3 ms, bandwidth = 35 Khz, 21 diffusion weighted images with a b-value of 1250 s/mm² were collected. A 3 × 3 diffusion tensor in each DTI image is illustrated as an ellipsoid [16], whose axes’ directions, and lengths correspond to its eigen-vectors, and eigen-values, respectively. The same initialization is used for each divergence based segmentation. We let all the divergence based methods run until convergence, record their time and compare the results.

Fig. 3.

From left to right are initialization, intermediate step, and final segmentation.

Fig. 4.

Dice coefficient comparison for tKL, KL_s, d_R, d_M, and LE segmentation of synthetic tensor field with increasing level (x-axis) of noise.

Fig. 5.

Dice coefficient comparison for tKL, KL_s, d_R, d_M, and LE segmentation of synthetic tensor field with increasing percentage (x-axis) of outliers.

We apply the piecewise constant segmentation model on a single slice (108 × 108) of the rat spinal cord, and apply the piecewise smooth segmentation model on the molecular layer from single slices of size (114 × 108) for rat corpus callosum (CC) and (90 × 90) for the rat hippocampus, respectively. Fig. 6(a)–(c) shows the initialization, Fig. 6(d) shows the segmentation results, and Table IV records their execution time. The results confirm that when compared to other divergences, tKL yields a more accurate segmentation in a significantly shorter amount of CPU time.

Fig. 6.

(a)–(c) Initialization. (d) Segmentation results, from left to right, using tKL, KL_s, d_R, d_M, and LE.

TABLE IV.

Time (Seconds) Comparison for Segmenting the Rat Spinal Cord, Corpus Callosum and Hippocampus Using Different Divergences

Divergences	tKL	KLs	dR	dM	LE
Time for Cord	33	68	72	81	67
Time for CC	87	183	218	252	190
Time for hippocampus	159	358	545	563	324

Apart from segmentation in 2-D slices, we also demonstrate 3-D DTI image segmentation using the proposed divergence. Fig. 7 depicts the process of segmenting rat corpus callosum (114 × 108 × 11) using the piecewise constant segmentation model. The result demonstrates that tKL can segment this white matter bundle quite well.

Fig. 7.

tKL segmentation of a 3-D rat corpus callosum. (a)–(d) A 2-D slice of the corresponding evolving surface, from left to right are initialization, intermediate steps and final segmentation. (e) A 3-D view of the segmentation result.

VI. CONCLUSION

In this paper, we developed a novel divergence dubbed the TBD, that is intrinsically robust to noise and outliers. The goal here was not simply to develop yet another divergence measure but to develop an intrinsically robust divergence. This was achieved via alteration of the basic notion of divergence which measures the ordinate distance between its convex generator and its tangent approximation to the orthogonal distance between the same. This basic idea parallels the relationship between least-squares and total least squares but the implications are far more significant in the field of information geometry and its applications since the entire class of the well known Bregman divergences have been redefined and some of their theoretical properties studied. Specifically, we derived an explicit formula for the t-center which is the TBD-based median, that is robust to outliers. In the case of SPD tensors, the t-center was shown to be SL(n) invariant. However, the story is not yet complete and further investigations are currently underway.

The robustness (to outliers) property of TBD was demonstrated here via applications to SPD tensor field interpolation and segmentation. The results favorably demonstrate the competitiveness of our newly defined divergence in comparison to existing methods not only in terms of robustness, but also in terms of computational efficiency and accuracy as well. Our future research will focus on applications of TBD to atlas construction, registration and other such tasks applied to DWMRI acquired from human brains.