Continuous Maximal Flows and Wulff Shapes: Application to MRFs

Abstract

Convex and continuous energy formulations for low level vision problems enable efficient search procedures for the corresponding globally optimal solutions. In this work we extend the well-established continuous, isotropic capacity-based maximal flow framework to the anisotropic setting. By using powerful results from convex analysis, a very simple and efficient minimization procedure is derived. Further, we show that many important properties carry over to the new anisotropic framework, e.g. globally optimal binary results can be achieved simply by thresholding the continuous solution. In addition, we unify the anisotropic continuous maximal flow approach with a recently proposed convex and continuous formulation for Markov random fields, thereby allowing more general smoothness priors to be incorporated. Dense stereo results are included to illustrate the capabilities of the proposed approach.

1. Introduction

Convex and continuous approaches to low level vision tasks are appealing for two reasons: (i) the convexity of the formulation ensures global optimality of the obtained solutions; and (ii) setting the problem in a continuous (i.e. non-combinatorial) domain often results in intrinsically data-parallel algorithms, that can be significantly accelerated e.g. by modern graphics processing units. Typically, the main challenge is to find a convex (and optionally continuous) formulation of an apparently non-convex problem. For instance, solving a pairwise Markov random field with multiple labels and linear or convex discontinuity costs requires an embedding of the original formulation into a higher dimensional space (see [6, 13, 11, 12] for combinatorial approaches and [20] for a continuous formulation).

Finding global optimizers for Markov random fields with pairwise and non-convex priors is generally NP-hard, and only approximation algorithms are known. Well-established approaches for MRF optimization in computer vision include belief propagation [26], graph cut methods [7, 15], message passing approaches [14, 16], and sequential fusion of labeling proposals [17, 25].

One major inspiration for this work is the continuous maximal flow framework proposed in [2], which provides globally optimal and efficient solutions to minimal surface problems for image segmentation (e.g. [9]) and stereo [21]. The original framework for continuous maximal flows uses isotropic, i.e. non-direction dependent, capacity constraints. The flow between neighboring nodes (as induced by the ℝ^N topology) is uniformly limited in all directions. The utilization of anisotropic capacities allows the flow to prefer certain, spatially varying directions.

It turns out, that the flow field with capacity constraints directly corresponds to the dual vector field employed in efficient minimization of total variation energies [10]. The spatially varying isotropic capacity constraint in [2] recurs as weighted total variation [8]. Anisotropic total variation for image denoising was theoretically analyzed in [19], and joint image smoothing and estimation of local anisotropy orientation was proposed in [3].

This work is outlined as follows: Section 2 introduces anisotropic capacity constraints and analyzes the relationship with continuous minimal surface and maximal flow formulations. An iterative optimization procedure maintaining primal and dual variables is presented as well. Section 3 uses strong duality results from non-smooth convex optimization to deduce the corresponding dual energy, that provides further theoretical insights, and is a useful indicator to stop the iterations. In Section 4 the results obtained in the previous sections are used to derive novel procedures for certain classes of Markov random fields, thus providing an extension and unification of [2] and [20]. Section 5 concludes this work.

2. Anisotropic Continuous Maximal Flows

This section presents an extension of continuous maximal flows with isotropic capacity constraints as proposed in [2] to anisotropic node capacities. Fortunately, the approach used in [2] to prove the correctness and stability of the underlying partial differential equations for isotropic capacity constraints can be easily generalized to the anisotropic setting as shown in the next sections.

2.1. Wulff Shapes

The aim of this section is to introduce a technique allowing to rewrite non-differentiable terms like ‖ · ‖ as better manageable expressions: unconstrained optimization of certain non-differentiable objective functions can be formulated as nested optimization of a scalar product term subject to a feasibility constraint induced by the respective Wulff shape (e.g. [19]):

Definition 1

Let ϕ : ℝ^N → ℝ be a convex, positively 1-homogeneous function (i.e. ϕ(λx) = λϕ(x) for λ > 0). The Wulff shape W_ϕ is the set

$$W_{\varphi }≔\{y\in {ℝ}^N:〈y,x〉\le \varphi (x)\forall x\in {ℝ}^N\},$$ (1)

where we denote the inner product in ℝ^N by 〈·, ·〉.

The Wulff shape W_ϕ is a convex, bounded and closed set. Further 0 ∈ W_ϕ. Given a Wulff shape W_ϕ the generating function ϕ(·) can be recovered by

$$\varphi (x)=\underset{y\in W_{\varphi }}{max}〈y,x〉.$$ (2)

For consistency with [2] and with the dual energy formulation (Section 3), we will use Eq. 2 in a slightly modified form with negated dual variables, i.e.

$$\varphi (x)=\underset{-y\in W_{\varphi }}{max}〈-y,x〉=\underset{y\in -W_{\varphi }}{max}〈-y,x〉,$$ (3)

where we introduce the negated Wulff shape,

$$-W_{\varphi }=\{-y:y\in W_{\varphi }\}.$$

An important set of functions ϕ satisfying the convexity and positive 1-homogeneity constraint are norms. The Wulff shape for the Euclidean norm, ‖ · ‖₂ is the unit ball with respect to ‖ · ‖₂. In general, the Wulff shape for the l^p norm is the unit ball of the dual norm, ‖ · ‖_q, with 1/p + 1/q = 1.

One useful observation about Wulff shapes in the context of this work is the geometry of the Wulff shape for linear combinations of convex and positively 1-homogeneous functions. It is easy to see that the following holds:

Observation 1

Let ϕ and ψ be two convex, positively 1-homogeneous functions, and k ∈ ℝ > 0. Then, the following relations hold:

$$W_{\varphi +\psi }=W_{\varphi }\oplus W_{\psi },$$ (4)

where A ⊕ B denotes the Minkowski sum of two sets A and B. Further,

$$W_{k\varphi }=\frac{1}{k}{W}_{\varphi }=\{\frac{y}{k}:y\in W_{\varphi }\}$$ (5)

This observation allows us to derive the geometry of Wulff shapes for positive linear combinations of given functions from their respective individual Wulff shapes in a straightforward manner.

In [19] is it shown, that we have ϕ (x) = 〈y, x〉 for non-zero x and y ∈ W_ϕ if and only if

$$y\in \partial W_{\varphi }\text{and}x\in N_{W_{\varphi }}(y),$$ (6)

where ∂W_ϕ denotes the boundary of the Wulff shape and N_Wϕ(y) is the set of outward pointing normals at y ∈ ∂W_ϕ. Note that this observation is equivalent to the Fenchel-Young equation (see Section 3), but expressed directly in geometric terms.

2.2. Minimal Surfaces and Continuous Maximal Flows

Appleton and Talbot [2] formulated the task of computing a globally minimal surface separating a known source location from the sink location as a continuous maximal flow problem. Instead of directly optimizing the shape of the separating surface ∂A, the indicator function of the respective region A enclosing the source is determined.

In this section we derive the main result of [2] for anisotropic capacity constraints. Let Ω ⊂ ℝ^N be the domain of interest, and S ⊂ Ω and T ⊂ Ω the source and sink regions, respectively. Further, let ϕ_x be a family of convex and positively 1-homogeneous functions for every x ∈ Ω. Then the task is to compute a binary valued function u : Ω → {0, 1}, that is the minimizer of

$$E(u)={\mathit{\int }}_{\mathrm{\Omega }}{\varphi }_{\mathbf{x}}(\nabla u)d\mathbf{x},$$ (7)

such that u(x) = 1 for all x ∈ S and u(x) = 0 for all x ∈ T. Note that we have replaced the usually employed weighted Euclidean norm by the general, spatially varying weighting function ϕ_x. In the following we will drop the explicit dependence of ϕ_x on x and will only use ϕ. Since ϕ is convex and positively 1-homogeneous, we can substitute ϕ(∇u) by max_y∈Wϕ 〈−y, ∇u〉 (recall Eq. 3) and obtain:

$$E(u)={\mathit{\int }}_{\mathrm{\Omega }}\underset{-y\in W_{\varphi }}{max}〈-y,\nabla u〉d\mathbf{x}.$$ (8)

We will collect all dual variables y ∈ −W_ϕ into a vector field p : Ω → −W_ϕ. Hence, we arrive at the following optimization problem:

$$\underset{u}{min}\underset{\mathbf{p}}{max}E(u,\mathbf{p})=\underset{u}{min}\underset{\mathbf{p}}{max}{\mathit{\int }}_{\mathrm{\Omega }}〈-\mathbf{p},\nabla u〉d\mathbf{x}$$ (9)

together with the source/sink conditions on u and the generalized capacity constraints on p,

$$-\mathbf{p}(\mathbf{x})\in W_{{\varphi }_{\mathbf{x}}}\forall \mathbf{x}\in \mathrm{\Omega }.$$ (10)

In many applications the Wulff shape W_ϕ is centrally symmetric, and the constraint −p(x) ∈ W_ϕx is then equivalent to p(x) ∈ W_ϕx.

The functional derivatives of Eq. 9 with respect to u and p are

$$\frac{\partial E}{\partial u}=\text{div}\mathbf{p}\frac{\partial E}{\partial \mathbf{p}}=-\nabla u$$ (11)

subject to Eq. 10 and the source and sink constraints on u. Since we minimize with respect to u, but maximize with respect to p, the gradient descent/ascent updates are

$$\frac{\partial u}{\partial \tau }=-\text{div}\mathbf{p}\frac{\partial \mathbf{p}}{\partial \tau }=-\nabla u$$ (12)

subject to −p ∈ W_ϕ. These equations are exactly the ones presented in [2], the only difference lies in the generalized capacity constraint on p. The basic setup is illustrated in Fig. 1.

Figure 1.

Basic illustration of the anisotropic continuous maximal flow approach. The minimum cut separating the source S and the sink T, the flow field p, and one anisotropic capacity constraint W_ϕ (convex shape) are depicted.

Correctness at Convergence

The first order optimality conditions are (recall Eq. 11):

$$\text{div}\mathbf{p}=0$$ (13)

$$\nabla u=0\text{if}-\mathbf{p}\in W_{\varphi }\backslash \partial W_{\varphi }$$ (14)

$$\nabla u\in N_{W_{\varphi }}(-\mathbf{p})\text{if}-\mathbf{p}\in \partial W_{\varphi }.$$ (15)

The first equations implies that there are no additional sources or sinks in the vector field p and flow lines connect the source and the sink. If the generalized capacity constraint Eq. 10 is active (i.e. p ∈ −W_ϕx), we have ∇u ∈ N_Wϕ(−P) by Eq. 6. Further, we have 〈 −p, ∇u 〉 = ϕ(∇u) ≥ 0 whenever the capacity is saturated, since ϕ is a non-negative function. Consequently, u is a non-decreasing function along the flow lines of − p (or equivalently, u is non-increasing along the flow lines of p), which is the key observation to prove the essentially binary nature of u. The proof for the optimality after thresholding u by an arbitrary value θ ∈ (0,1) is an extended version of the one given in [2]. Since the proof relies on duality results derived later in Section 3, we postpone the proof to the appendix.

2.3. Numerical Considerations

Discretization

For finite lattices the gradient operator ∇ and the divergence operator div need to be dual. We do not employ the staggered grid approach proposed in [2], but use the same underlying grids for u and p in our implementation. ∇u is evaluated using forward differences, whereas div p is computed by backward differences. In order to ensure that the negated gradient and the divergence are adjoint linear operators, in the finite setting, suitable boundary conditions are required [10, 8].

The gradient descent equations in Eq. 12 are applied using an explicit Euler scheme with a uniform time step τ. Since these equations are the same discrete wave equations as in [2], the same upper bound on the time step holds, i.e. τ < 1/√N. The update of p is followed by a reprojection step to enforce p ∈ −W_ϕ.

Complex Wulff Shapes

In many cases this reprojection step is just a clamping operation (e.g. if ϕ is the L₁ norm) or a renormalization step (if ϕ is the Euclidean norm). Determining the closest point in the respective Wulff shape can be more complicated for other choices of ϕ. In particular, if the penalty function is of the form (ϕ + ψ), then it is easy to show that maintaining separate dual variables for ϕ and ψ yields to the same optimal solution – with the drawback of increased memory consumption.

Terminating the Iterations

A common issue with iterative optimization methods is a suitable criterion when to stop the iterations. An often employed, but questionable stopping criterion is based on the length of the gradients or update vector. In [2] a stopping criterion was proposed, that tests whether u is sufficiently binary. This criterion is problematic in several cases, e.g. if a reasonable binary initialization is already provided. A common technique in optimization to obtain true quality estimates for the current solutions is to utilize strong duality. The current primal energy (which is minimized) yields a upper bound on the true minimum, whereas the corresponding dual energy provides a lower bound. If the gap is sufficiently small, the iterations can be terminated with guaranteed optimality bounds. Since the formulation and discussion of duality in the context of anisotropic continuous maximal flows provides interesting connections, we devote the next section to this topic.

3. Dual Energy

Many results on the strong duality of convex optimization problems are well-known, e.g. primal and dual formulations for linear programs and Lagrange duality for constrained optimization problems. The primal energy Eq. 7 is generally non-smooth and has constraints for the source and sink regions. Thus, we employ the main result on strong duality from convex optimization to obtain the corresponding dual energy (e.g. [5]).

3.1. Fenchel Conjugates and Duality

The notion of dual programs in convex optimization is heavily based on subgradients and conjugates:

Definition 2

Let f : ℝ^N → ℝ be a convex and continuous function. y is a subgradient at x₀ if

$$\forall x:f(x_0)+〈y,x-x_0〉\le f(x)$$

The subdifferential ∂f(x) is the set of subgradients of f at x, i.e. ∂f(x) = {y : y is a subgradient of f at x}.

Intuitively, for functions f : ℝ → ℝ, subgradients of f at x₀ are slopes of lines containing (x₀, f(x₀)) such that the graph f is not below the line.

Definition 3

Let f : ℝ^N → ℝ be a convex and semi-continuous function. The Fenchel conjugate of f, denoted by f* : ℝ^N → ℝ is defined by

$$f^{\ast }(z)=\underset{y}{sup}[〈z,y〉-f(y)].$$ (16)

f** = f if and only if f is convex and semi-continuous. From the definition of f*, we obtain

$$f(y)\ge 〈z,y〉-f^{\ast }(z)\forall y,z\in {ℝ}^N,$$ (17)

i.e. the graph of f is above the hyperplane [y, 〈z, y〉 − f*(z)] (Fenchel-Young inequality). Equality holds for given y, z if and only if z is a subgradient of f (or equivalently, y is a subgradient of f*, Fenchel-Young equation).

Let A ∈ ℝ^{M × N} be a matrix (or a linear operator in general). Consider the following convex optimization problem for convex and continuous f and g:

$$p=\underset{y\in {ℝ}^N}{inf}[f(y)+g(Ay)].$$ (18)

The corresponding dual program is [5]:

$$d=\underset{z\in {ℝ}^M}{sup}[-f^{\ast }(A^{T}z)-g^{\ast }(-z)].$$ (19)

Fenchel's duality theorem states, that strong duality holds (under some technical condition), i.e. p = d. Further, since the primal program is a minimization problem and the dual program is maximized,

$$f(y)+g(Ay)\ge -f^{\ast }(A^{T}z)-g^{\ast }(-z)$$ (20)

is always true for any y ∈ ℝ^N and z ∈ ℝ^M. This inequality allows to provide an upper bound on the optimality gap (i.e. the distance to the true optimal energy) in iterative algorithms maintaining primal and dual variables.

3.2. Dual Energy for Continuous Maximal Flows

Now we consider again the basic energy to minimize (recall Eq. 7)

$$E(u){\int }_{\mathrm{\Omega }}{\varphi }_{\mathbf{x}}(\nabla u(\mathbf{x}))d\mathbf{x}.$$

Without loss of generality, we assume Ω to be 3-dimensional, since our applications are in such a setting. Let A denote the gradient operator, i.e. A(u) = ∇u. In the finite setting, A will be represented by a sparse 3|Ω| × |Ω| matrix with ±1 values at the appropriate positions.

The function g represents the primal energy,

$$g(A\circ u):={\int }_{\mathrm{\Omega }}{\varphi }_{\mathbf{x}}((A\circ u)(\mathbf{x}))d\mathbf{x}.$$ (21)

(A ∘ u)(x) denotes ∇u(x) Since A is a linear operator and ϕ_x is convex, g is Convex with respect to u.

The second function in the primal program, f, encodes the restrictions on u. The value attained by u(x) should be 1 for source locations x ∈ S, and 0 for sink regions, u(x) = 0 for x ∈ T. W.l.o.g. we can assume u(x) ∈ [0,1] for all x ∈Ω. Without this assumption it turns out that the dual energy is unbounded (−∞) if div p(x) ≠ 0 for some x ∈ Ω not in the source or the sink region (see below). We choose f as

$$f(u)=\{\begin{array}{ll}0& \text{if}u\text{complies with the constraints above}\\ \infty & \text{otherwise}\end{array}$$ (22)

We denote the space of functions satisfying the constraints (equivalently, with finite f(u)) by .

The dual variable is a vectorial function (vector field) p : Ω → ℝ³. Since g(∇u) is a sum (integral) of ϕ_x acting on independent components of ∇u, we have for the conjugate function g*:

$$g^{\ast }(-\mathbf{p})={\int }_{\mathrm{\Omega }}{\varphi }_{\mathbf{x}}^{\ast }(-\mathbf{p}(\mathbf{x}))d\mathbf{x}.$$ (23)

Since all ϕ_x share the same properties, we drop the index and derive the Fenchel conjugate for any convex, positively 1-homogeneous function ϕ(y). First, note that the Wulff shape W_ϕ is nothing else than the subdifferential of ϕ at 0, i.e. W_ϕ = ∂ϕ(0):

$$\begin{array}{ll}z\in \partial \varphi (0)& \iff \forall y:\varphi (0)+〈z,y-0〉\le \varphi (y)\\ & \iff \forall y:〈z,y〉\le \varphi (y)\iff z\in W_{\varphi },\end{array}$$

where we made use of ϕ(0 · y) = 0 · ϕ(y) = 0. Hence, we obtain for every z ∈W_ϕ (by the Fenchel-Young equation)

$${\varphi }^{\ast }(z)=〈z,0〉-\varphi (0)=0.$$ (24)

If z ∉ W_ϕ, then there exists a y such that 〈z, y〉 > ϕ(y). Further, 〈z, λy〉 > ϕ(λy) = λϕ(y) for any λ > 0, and therefore ϕ*(z) = sup_y [〈z,y〉 − ϕ(y)] is unbounded. In summary, we obtain

$${\varphi }^{\ast }(z)=\{\begin{array}{ll}0& \text{if}z\in W_{\varphi }\\ \infty & \text{otherwise},\end{array}$$ (25)

i.e. the conjugate of ϕ is the indicator function of the respective Wulff shape.

The conjugate of f, f*, can be computed directly. The argument of f* is a mapping from Ω to negated divergences, q : Ω → ℝ with

$$q(\mathbf{x})=(A^{\ast }\circ \mathbf{p})(\mathbf{x})=-\text{div}\mathbf{p}(\mathbf{x}),$$

where A* is the adjoint operator to A (recall that ∇* = − div). Further, let Ω̊ denote the domain without the source and sink sets, i.e. Ω̊ = Ω \ (S ∪ T), then

$$f^{\ast }(q)=\underset{u}{sup}[〈q,u〉-f(u)]$$ (26)

$$=\underset{u\in \mathcal{U}}{max}[〈q,u〉]$$ (27)

$$=\underset{u\in \mathcal{U}}{max}[{\int }_{\stackrel{\circ }{\mathrm{\Omega }}}qud\mathbf{x}+{\int }_{S}qud\mathbf{x}+{\int }_{T}qud\mathbf{x}]$$ (28)

$$={\int }_{\stackrel{\circ }{\mathrm{\Omega }}}max(0,q)d\mathbf{x}+{\int }_{S}qd\mathbf{x},$$ (29)

since u is fixed to 1 x ∈ S, u (x) = 0 for x ∈ T, and between 0 and 1 for locations neither at the source nor the sink.

In summary, the primal energy to evaluate is given in Eq. 7, and the dual energy (recall Eq. 19) is given by

$$E^{\ast }(\mathbf{p})={\int }_S\text{div}\mathbf{p}d\mathbf{x}+{\int }_{\stackrel{\circ }{\mathrm{\Omega }}}-max(0,-\text{div}\mathbf{p})d\mathbf{x}$$ (30)

$$={\int }_S\text{div}\mathbf{p}d\mathbf{x}+{\int }_{\stackrel{\circ }{\mathrm{\Omega }}}min(0,\text{div}\mathbf{p})d\mathbf{x}$$ (31)

subject to p(x) ∈ − W_ϕx. The two terms in the dual energy have direct interpretations: the first term,

$${\int }_S\text{div}\mathbf{p}d\mathbf{x}$$ (32)

measures the total outgoing flow from the source, while the second term, ∫_Ω̊ min(0, div p) dx, penalizes additional sinks in the interior of Ω. Thus, it corresponds directly to the well-known min-cut/max-flow theorem.

If we do not add a bounds constraint on u, here u(x) ∈ [0,1], then it is easy see show that a strict penalizer on the divergence outside S and T is obtained. The dual energy contains just the source term,

$$E^{\ast }(\mathbf{p})={\int }_S\text{div}\mathbf{p}d\mathbf{x}$$ (33)

subject to p ∈ −W_ϕx and div p = 0 in Ω̊. For p not satisfying these constraints the dual energy is −∞. Algorithms as the gradient descent updates in Eq. 12 achieve div(p) = 0 only in the limit, hence the strict penalty on div(p) ≠ 0 is not beneficial. Nevertheless, Eq. 33 is useful for analysis after convergence.

The bounds constraint u(x) ∈ [0,1] is redundant for the primal energy, since u(x) ∈ [0,1] is always true for minima of Eq. 7 subject to the source and sink constraints. In the dual setting this range constraints has the consequence, that any capacity-constrained flow p is feasible. Any additional flow generated by p at the source S, but vanishing in the interior is subtracted from the total flow in Eq. 31, hence the overall dual energy remains the same, and bounded and unbounded formulations are equivalent. In practice, we observed substantially faster convergence if u is clamped to [0,1] after each update step, and a meaningful value for the duality gap is thus obtained.

3.3. Equivalence of the Dual Variables

It remains to show, that the dual vector field p introduced in Section 2 (Eq. 9) in facts corresponds to the dual vectorial function p introduced in the dual energy formulation (Eq. 31). This is easy to see: after convergence of the primal-dual method, the primal-dual energy is

$$\begin{array}{ll}E(u,\mathbf{p})& ={\int }_{\mathrm{\Omega }}〈-\mathbf{p},\nabla u〉d\mathbf{x}={\int }_{\mathrm{\Omega }}\text{div}\mathbf{p}·ud\mathbf{x}\\ & ={\int }_{\stackrel{\circ }{\mathrm{\Omega }}}\text{div}\mathbf{p}·ud\mathbf{x}+{\int }_S\text{div}\mathbf{p}·ud\mathbf{x}+{\int }_T\text{div}\mathbf{p}·ud\mathbf{x}\\ & ={\int }_S\text{div}\mathbf{p}d\mathbf{x},\end{array}$$

since u is one in S and zero in T, and div p = 0 in Ω̊. Consequently, the primal-dual energy in terms of p after convergence is identical to the total flow emerging from the source, i.e. equal to the dual energy. Thus, p as maintained in the primal-dual formulation is equivalent to the argument of the pure dual one.

4. Application to Markov Random Fields

In this section we address the task of globally optimizing a Markov random field energy, where every pixel in the image domain can attain a label from a discrete and totally ordered set of labels ℒ. Without loss of generality, we assume that the labels are represented by non-negative integers, i.e. ℒ = {0,…, L − 1}. Similar to [22, 21, 2, 20], we formulate the label assignment problem with linearly ordered label sets as a segmentation task in higher dimensions.

4.1. From Markov Random Fields to Anisotropic Continuous Maximal Flows

We denote the (usually rectangular) image domain by ℐ and particular pixels in bold font, e.g. x ∈ ℐ. We will restrict ourselves to the case of two-dimensional image domains. A labeling function Λ : ℐ → ℒ, x ↦ Λ(x) maps pixels to labels. The task is to find a labeling function that minimizes an energy functional comprised of a label cost and a regularization term, i.e. to find the minimizer of

$$E(\mathrm{\Lambda })={\int }_{ℐ}(c(\mathbf{x},\mathrm{\Lambda }(\mathbf{x}))+V(\nabla \mathrm{\Lambda },{\nabla }^2\mathrm{\Lambda },\dots ))d\mathbf{x},$$ (34)

where c(x, Λ(x)) denotes the cost of selecting label Λ(x) at pixel x and V(·)is the regularization term. Even for convex regularizers V(·)is optimization problem is generally difficult to solve exactly, since the data costs are typically highly non-convex. By embedding the labeling assignment into higher dimensions, and by appropriately restating the energy in Eq. 34, a convex formulation can be obtained. Let Ω be the product space of ℐ and ℒ i.e. Ω = ℐ × ℒ.

In the following, we introduce the level function u,

$$u(\mathbf{x},l)=\{\begin{array}{ll}1& \text{if}\mathrm{\Lambda }(\mathbf{x})<l\\ 0& \text{otherwise}.\end{array}$$ (35)

Since one label must be assigned to every pixel, we require that u(x, L) = 1 for all x ∈ ℐ. Further, u(x, 0) = 0 by construction. In particular, the gradients of these function in spatial direction (∇_x) and in the label direction (∇_l) play an important role, i.e.

$${\nabla }_{\mathbf{x}}u=\left(\begin{array}{c}\partial u/\partial x\\ \partial u/\partial y\end{array}\right){\nabla }_{l}u=\partial u/\partial l.$$

Finally, ∇u specifies the full gradient of u in spatial and label directions, ∇u = (∂u/∂x, ∂u/∂y, ∂u/∂l)^T.

We assume, that the regularization energy in Eq. 34 can be written in terms of the gradient of the level function, i.e. V(·) can be formulated as

$${\int }_{\mathrm{\Omega }}{\psi }_{\mathbf{x},l}(\nabla u)d\mathbf{x}dl,$$ (36)

where ψ_x,l is a family of convex and 1-positively homogeneous functions, that shapes the regularization term. The same technique as proposed in [20] can be applied to rewrite the data term:

$${\int }_{ℐ}c(\mathbf{x},\mathrm{\Lambda }(\mathbf{x}))d\mathbf{x}={\int }_{ℒ}{\int }_{ℐ}c(\mathbf{x},l)|{\nabla }_{l}u(\mathbf{x},l)|d\mathbf{x}dl$$ (37)

$$={\int }_{\mathrm{\Omega }}c|{\nabla }_{l}u|d\mathbf{x}dl,$$ (38)

where |∇_lu| ensures that costs stay positive even for general choices of u, that attain any real value and are not monotonically increasing in their label argument.

By combining the data fidelity and regularization terms, we can rewrite the original energy Eq. 34 purely in terms of u (where we omit the explicit dependence of x and l),

$$E(u)={\int }_{\mathrm{\Omega }}[\psi (\nabla u)+c|{\nabla }_{l}u|]d\mathbf{x}dl.$$ (39)

Since ψ_x,l and c| · | are convex and 1-positively homogeneous, their sum ϕ_x,l(·) : = ψ_x,l(·) + c_x,l |·| shares this property. Finally, we can rewrite Eq. 39 as

$$E(u)={\int }_{\mathrm{\Omega }}\varphi (\nabla u)d\mathbf{x}dl$$ (40)

Since ϕ_x,l is again a family of positively 1-homogeneous functions, the anisotropic continuous maximal flow approach described in Section 2 can be applied. Specifically, the solution u after convergence is essentially binary and monotone in the label direction, thus the optimal labeling function Λ can be easily recovered from u.

4.2. Choices for ψ

In [20] a different approach is taken to analyze the particular choice ψ_x(∇_u) = ‖∇_xu‖₂ (resulting in a special case of the updates in Eq. 12). By the co-area formula it can be shown, that this choice for ψ_x corresponds to the total variation regularization of the underlying labeling function Λ,

$${\int }_{ℐ}\Vert \nabla \mathrm{\Lambda }\Vert d\mathbf{x}={\int }_{\mathrm{\Omega }}{\Vert {\nabla }_{\mathbf{x}}u\Vert }_{2}d\mathbf{x}dl.$$ (41)

As such, using ψ_x for the smoothness term results in the preference of piecewise constant assigned labels. The corresponding Wulff shape W_ϕx,l of ϕ_x,l = ‖ ∇_xu‖₂ + αc|∇_lu| is a cylinder with its height proportional to the label cost c(x, l). Since the regularization term is isotropic in the image domain, i.e. the smoothness term does not depend on the actual image content, we denote this particular regularization as homogeneous total variation. Note that it is easy to “squeeze” this cylinder depending on the strength and orientation of edges in the reference image in the stereo pair (Figure 2). In order to use only relevant image edges, the structure tensor of the denoised reference image can be utilized to shape the resulting Wulff shape. We use the phase field approach [1] for the Mumford-Shah functional [18] to obtain piecewise smooth images. The positive impact on the resulting depth maps (with the sampling insensitive Birchfield-Tomasi matching cost [4]) is illustrated in Figure 3.

Figure 2.

The Wulff shape for homogeneous (left) and edge driven (right) total variation regularizers.

Figure 3.

Depth maps with and without edge driven regularization. (a) is based on homogeneous total variation regularization, and (b) utilizes edge driven regularization. Observe that in (a) the low data fidelity weight yields to substantial loss of detail in the depth map, which is clearly better preserved in (b).

The total variation regularization on the depth map is rather inappropriate if the labels directly correspond to metric depth values in 3D space instead of pure image disparities. A more suitable choice is ψ = ‖∇u‖, if we assume that labels represent equidistantly sampled depth values. This particular choice penalizes the 3D surface area and yields smoother 3D models largely suppressing staircasing artifacts. This specific choice for regularization complies with smoothing the depth map based on an induced surface metric [24]. It can be readily verified that the respective Wulff shape is a capsule-like shape (i.e. a cylinder with half-spheres attached to its base and top face). Figure 4(a–b) visually compares the 3D meshes for the “Dino ring” dataset [23] obtained using total variation regularization ‖∇_xu‖ (Fig. 4(a)) and 3D regularization ‖∇u‖ (Fig. 4(b)). Figure 4(c) and (d) display the 3D models obtained for a brick wall, for which a laser scanned reference range image is available. The RMS for (c) is 5.1cm, and 3.2cm for (d) with respect to the known range image.

Figure 4.

3D meshed obtained for ψ = ‖∇_xu‖ (a, c) and ψ = ‖∇u‖ (b, d). Clearly, ‖∇_xu‖ favors piece-wise constant results. In (d) the strong matching term causes a visible step structure especially in the foreground. Using a lower value of λ in (c) removes the floor (bottom part of the mesh).

The underlying update equations Eq. 12 are very suitable to be accelerated by a modern GPU. Our current CUDA-based implementation executed on a Geforce 8800 Ultra is able to achieve two frames per second for 320 × 240 images and 32 disparity levels (aiming for a 2% duality gap at maximum).

5. Conclusion

This work analyzes the relationship between continuous maximal flows, Wulff shapes, convex analysis and Markov random fields with convex and homogeneous pairwise priors. It is shown that strong results from isotropic continuous maximal flow carry over to flows with anisotropic capacity constraints. The underlying theory yields extensions to a recently proposed continuous and convex formulation for Markov random fields with total variation-based smoothness priors. The numerical simplicity and the data-parallel nature of the minimization method allows an efficient implementation on highly parallel devices like modern graphics processing units.

Future work will address the extension of this work to more general classes of MRFs by investigating other linear operators than ∇, e.g. using ‖∇_x ∇_lu‖ in the regularization term corresponds to the LP-relaxation of MRFs with a Potts discontinuity prior.

A. The Potential u is Essentially Binary

This section shows, that thresholding of a not necessarily binary primal solution u* of the anisotropic geodesic energy Eq. 7 yields to a equally globally optimal solution. Let u* and p* be a pair of primal and dual globally optimal solutions for the continuous maximal flow energy Eq. 9. Recall that the optimal dual energy is the total flow leaving the source S (Eq. 33):

$$v^{\ast }:=E^{\ast }(p^{\ast })={\int }_S\text{div}\mathbf{p}d\mathbf{x},$$ (42)

and that v* is equal to the primal energy E(u*) = ∫ ϕ(∇u*). The thresholded result of u*, u_θ, is given by

$$u_{\theta }(\mathbf{x})=\{\begin{array}{ll}1& \text{if}u(\mathbf{x})\ge \theta \\ 0& \text{otherwise}.\end{array}$$

Further, define A_θ := {x ∈ Ω : u*(x) ≥ θ}. Note that for x ∈ Ω with ∇u_θ(x) ≠ 0 (or equivalently x ∈ ∂A_θ) we have ∇u*(x) ≠ 0 by construction. Even a stronger relation holds: ∇u_θ(x) = k(x) ∇u*(x) for a positive scalar k(x), since both gradients point in the same direction. Further, the capacity constraint is active at x ∈ A_θ: −p*(x) ∈ ∂W_ϕx. Thus, for all x ∈ A_θ (omitting the explicit dependence on x)

$$\begin{array}{ll}〈-{\mathbf{p}}^{\ast },\nabla u_{\theta }〉& =〈-{\mathbf{p}}^{\ast },k\nabla u^{\ast }〉\\ & =k〈-{\mathbf{p}}^{\ast },\nabla u^{\ast }〉=k\varphi (\nabla u^{\ast })\\ & =\varphi (k\nabla u^{\ast })=\varphi (\nabla u_{\theta }).\end{array}$$

Overall we have:

$$\begin{array}{ll}{\int }_{\mathrm{\Omega }}\varphi (\nabla u_{\theta })d\mathbf{x}& ={\int }_{\mathrm{\Omega }}〈-p^{\ast },\nabla u_{\theta }〉d\mathbf{x}\\ & ={\int }_{\mathrm{\Omega }}\text{div}{p}^{\ast }·u_{\theta }d\mathbf{x}={\int }_S\text{div}{p}^{\ast }=v^{\ast },\end{array}$$

since div p* = 0 in Ω̊ and u_θ is 1 in S and 0 in T. This means, that the binary function u_θ has the same energy as u* and is likewise optimal.