Simultaneous optical flow and source estimation: Space–time discretization and preconditioning

Abstract

We consider the simultaneous estimation of an optical flow field and an illumination source term in a movie sequence. The particular optical flow equation is obtained by assuming that the image intensity is a conserved quantity up to possible sources and sinks which represent varying illumination. We formulate this problem as an energy minimization problem and propose a space–time simultaneous discretization for the optimality system in saddle-point form. We investigate a preconditioning strategy that renders the discrete system well-conditioned uniformly in the discretization resolution. Numerical experiments complement the theory.

1. Introduction

Optical flow is the apparent motion of objects, surfaces, and edges in a visual scene. In Computer Vision the optical flow is calculated as the flow field registering pixels of a movie, which we understand as a sequence of ordered images. The classical optical flow equation is based on the following assumptions:

• 1.Brightness constancy along each characteristic of the flow.
• 2.A slowly varying image sequence.

The first assumption implies that changes of intensities caused by varying illumination (for instance shades) influence and distort the recovered flow. In this paper we admit violations of this assumption, and aim at estimating global changes of illumination and the effect of them onto the flow field. Indeed, the problem we consider here consists in the simultaneous determination of the optical flow, the brightness, and its possible sources and sinks in space and time. While standard optical flow algorithms process two successive frames of a movie sequence at a time [18–20,3,13,6,1], we work with a continuous space–time simultaneous formulation similarly to [23]. A brief overview of gradient based optical flow models and algorithms may be found in [14].

Related to our work is [24]. Therein, the standard optical flow constraint was replaced by the continuity equation ${\partial }_t\rho +\mathrm{\nabla }\cdot (\rho v)=0$, motivated by mass conservation in fluid flow. We relax the continuity equation by allowing distributed sources g, and replace the momentum ρv by one unknown j, called the flux; we thus obtain the constraint ${\partial }_t\rho +\mathrm{\nabla }\cdot j=g$.

The work [17], see also references therein, suggested incorporating physical models to account for illumination changes, such as the diffusion of the heat observed by infrared cameras. The parameters governing the physical model are then estimated in parallel with the optical flow. Our ansatz is more naive, in the sense that we treat the possible change in illumination as an additional unknown, but a physical description can in principle be used instead.

As a further difference to the bulk of optical flow literature we treat the density ρ as an unknown. We thus estimate $(\rho ,j,g)$ in parallel by minimizing a functional $\mathcal{J}$ which consists of a discrepancy term for ρ and regularization terms for j and g. This is similar to [7]; however, using the flux j instead of the velocity v results in linear optimality conditions characterizing the minimizer of the functional $\mathcal{J}$. While this is convenient for computation, it leads to somewhat unexpected flow patterns (see Sections 5–6).

It is important to note that these optimality conditions form a set of equations that are coupled in space–time. This is typical for optimal control problems with spatio-temporal constraints due to the coupling of the original problem forward in time and the adjoint problem backward in time. The main contribution of this work is to propose and test a preconditioner for the discretized space–time system in saddle-point form that is robust in the discretization parameters.

The outline of the paper is as follows. In Section 2 we introduce our optical flow model, which consists in the minimization of a quadratic cost functional over a space of space–time dependent functions subject to an optical flow constraint. In Section 3 the functional analytic framework is established. In Section 4 we rewrite the optimality conditions in a saddle-point form and discuss its discretization and preconditioning. In Section 5 we report on our numerical experiments. Section 6 concludes the paper.

In the following, $\langle \cdot ,\cdot \rangle$ denotes the duality pairing, while ${\langle \cdot ,\cdot \rangle }_X$ and ${\Vert \cdot \Vert }_X$ denote the scalar product and the norm of a Hilbert space X. The usual Lebesgue and Sobolev spaces on a domain D are written as $L_p(D)$ and $H^s(D)$. We abbreviate ${\Vert \cdot \Vert }_{L_2(D)}$ as ${\Vert \cdot \Vert }_D$, and similarly for other $L_2$ spaces.

2. Optical flow model

We consider a movie sequence of time length $T>0$, with each frame defined on a rectangular domain $D\subset {\mathbb{R}}^2$. We write $J:=(0,T)$ for the temporal interval. As explained in the introduction above, our optical flow model is the scalar transport equation

(1)

Here, ρ denotes the density which is the intensity of the image considered as a piecewise constant function; j denotes the flux which is the optical flow weighted by the density; and g is a function that models varying spatial illumination. The divergence ∇⋅ acts on the spatial variable only. The functions ρ and g are scalar-valued, while j is vector-valued on $\overline{J}\times D$. Integrating (1) over a subdomain $D^{\prime }\subset D$ and using the divergence theorem for the j term, one sees that the density ρ is either transported in or out of $D^{\prime }$ over its boundary, or is created/annihilated by means of the source term g.

In this paper we consider the problem of identifying the flux j and the source g from a finite number of frames of a movie, indexed by $\mathcal{T}\subseteq \overline{J}$:

$${\rho }_{\tau }\in L_2(D),\tau \in \mathcal{T}.$$ (2)

We aim at minimizing the data fidelity functional

(3)

subject to the transport equation (1) and further constraints on the flux j and the source g as discussed below. Here, the symbol denotes the averaged sum $\frac{1}{\mathrm{\#}\mathcal{T}}{\sum }_{\tau \in \mathcal{T}}$.

For comparison purposes we recall the standard optical flow equation, which reads as follows:

$${\partial }_t\rho +\mathrm{\Phi }\cdot \mathrm{\nabla }\rho =0\text{in}J\times D.$$ (4)

Eq. (4) can be formally derived from (1) by identifying j and ρΦ and neglecting small terms: Indeed, from (1) we get

$${\partial }_t\rho +\mathrm{\Phi }\cdot \mathrm{\nabla }\rho +\rho \mathrm{\nabla }\cdot \mathrm{\Phi }=0\text{in}J\times D.$$ (5)

Hence, if the term $\rho \mathrm{\nabla }\cdot \mathrm{\Phi }$ is small (meaning that there is a slowly varying velocity), and if there are no changes in illumination, the equations are identical.

3. Functional analytic framework

We now introduce function spaces for which the transport equation (1) is well-defined. We shall work with image intensity $\rho \in H^{1,0}:=H^1(J;L_2(D))$, flux $j\in H^{0,\mathrm{div}}:=L_2(J;H^{\mathrm{div}}(D))$, and source $g\in H^{0,0}:=L_2(J;L_2(D))$. The superscripts indicate the Sobolev smoothness in time and space, respectively. Here, $H^{\mathrm{div}}(D)$ is the space of vector-valued functions in ${[L_2(D)]}^2$ with distributional divergence in $L_2(D)$. As a consequence of the Fubini–Tonelli theorem [25, Section 0.3], the space $H^{0,0}$ is isometrically isomorphic to $L_2(J\times D)$. The norms on $H^1(J)$ and $H^{\mathrm{div}}(D)$ are defined by

$${\Vert f\Vert }_{H^1(J)}^2:=T^{-2}{\Vert f\Vert }_J^2+{\Vert f^{\prime }\Vert }_J^2,f\in H^1(J),$$

$${\Vert v\Vert }_{H^{\mathrm{div}}(D)}^2:=\mathrm{diam}{(D)}^{-2}{\Vert v\Vert }_D^2+{\Vert \mathrm{\nabla }\cdot v\Vert }_D^2,v\in H^{\mathrm{div}}(D),$$

where the scaling $T^{-2}$ and $\mathrm{diam}{(D)}^{-2}$ is to match the units. We introduce the product space

$$X:=H^{1,0}\times H^{0,\mathrm{div}}\times H^{0,0}$$

and associate it with the norm

$${\Vert (\rho ,j,g)\Vert }_X^2:={\Vert \rho \Vert }_{H^{1,0}}^2+{\Vert j\Vert }_{H^{0,\mathrm{div}}}^2+{\Vert g\Vert }_{H^{0,0}}^2.$$ (6)

The transport equation constraint (1) is reformulated via the linear operator

$$G:X\to H^{0,0},G(\rho ,j,g):={\partial }_t\rho +\mathrm{\nabla }\cdot j-g.$$ (7)

Since G is continuous, the preimages of closed subsets are closed; in particular, its kernel

$$\mathrm{\Gamma }:=G^{-1}(0)\subset X$$ (8)

is a closed linear subspace. It contains precisely the triples $(\rho ,j,g)\in X$ that satisfy the transport equation (1).

Given two regularization parameters ${\alpha }_j>0$ and ${\alpha }_g>0$, we define the penalization functional

(9)

and, recalling $F_{\mathcal{T}}$ from (3), also the cost functional

(10)

The parameters ${\alpha }_j$ and ${\alpha }_g$ are dimensionless. The following lemma is straightforward.

Lemma 1

The functional $\mathcal{J}$ is Gâteaux differentiable on X. With ${\rho }_{\tau }$ from (2), its Gâteaux derivative at $(\rho ,j,g)\in X$ is the continuous linear functional ${\mathcal{J}}^{\prime }(\rho ,j,g)=A(\rho ,j,g)-\ell$, where $A:X\to X^{\prime }$ and $\ell \in X^{\prime }$ are given by

(11)

(12)

Moreover, A is continuous, self-adjoint, and Γ-elliptic,

$$\exists \alpha >0:\langle Av,v\rangle \ge \alpha {\Vert v\Vert }_X^2\forall v\in \mathrm{\Gamma }.$$ (13)

Further, A induces a seminorm $⦀\cdot ⦀$ on X given by

(14)

This seminorm is in fact a norm on $\mathrm{\Gamma }=G^{-1}(0)$, where it is equivalent to ${\Vert \cdot \Vert }_X$.

With the help of this lemma and standard arguments from variational calculus, one can show strict convexity and coercivity properties of the functional $\mathcal{J}$:

Lemma 2

The functional $\mathcal{J}$ defined in (10) is continuous and convex on X. Moreover, it is strictly convex on Γ,

$$\mathcal{J}(\lambda v_{+}+(1-\lambda )v_{-})<\lambda \mathcal{J}(v_{+})+(1-\lambda )\mathcal{J}(v_{-})\forall v_{\pm }\in \mathrm{\Gamma },\lambda \in (0,1),$$ (15)

$v_{+}\ne v_{-}$, and coercive on Γ,

$$\exists \alpha >0,\beta \ge 0:\mathcal{J}(v)\ge \alpha {\Vert v\Vert }_X^2-\beta \forall v\in \mathrm{\Gamma }.$$ (16)

Our optical flow problem now reads as follows:

Definition 1

For given regularization parameters $\alpha =({\alpha }_j,{\alpha }_g)$, the optical flow $u_{\alpha }$ is the unique minimizer of the functional $\mathcal{J}$ over Γ.

Existence of the minimizer is due to standard arguments of the calculus of variations [12], because $\mathcal{J}$ is non-negative and proper ($\mathcal{J}\ne \infty$) on Γ. Uniqueness follows from the strict convexity (15) of $\mathcal{J}$ on Γ.

The minimizer $u_{\alpha }$ of $\mathcal{J}$ is equivalently characterized in terms of first order optimality conditions. By [12, Theorems 1.3–1.4 in §3.1.3], we have ${\mathcal{J}}^{\prime }(u_{\alpha })=0$, where ${\mathcal{J}}^{\prime }(u)\in {\mathrm{\Gamma }}^{\prime }$ is the Gateaux derivative of $\mathcal{J}$ at $u\in \mathrm{\Gamma }$. To simplify the notation we will omit the dependence on α. Using Lemma 1, the requirement ${\mathcal{J}}^{\prime }(u)=0\in {\mathrm{\Gamma }}^{\prime }$ is equivalent to the variational problem

(17)

The fact that this variational problem is posed on the implicitly defined subspace $\mathrm{\Gamma }=G^{-1}(0)$ leads to the saddle-point problem introduced in the next section.

4. Numerical solution

In order to minimize $\mathcal{J}$ we solve the equivalent variational problem (17). The constraint $u\in \mathrm{\Gamma }$ is implemented using a Lagrange multiplier λ to complement (17) to a saddle-point problem: Find $(u,\lambda )\in X\times H^{0,0}$ such that

$$\langle Au,v\rangle +\langle Gu,\mu \rangle +\langle Gv,\lambda \rangle =\langle \ell ,v\rangle \forall (v,\mu )\in X\times H^{0,0}.$$ (18)

4.1. Discretization

In order to discretize (18) we introduce finite dimensional subspaces $X_h\subset X$ and $M_h\subset H^{0,0}$, as specified below, and consider the discrete saddle-point system: Find $(u_h,{\lambda }_h)\in X_h\times M_h$ such that

$$\langle A{u}_h,v\rangle +\langle G{u}_h,\mu \rangle +\langle Gv,{\lambda }_h\rangle =\langle \ell ,v\rangle \forall (v,\mu )\in X_h\times M_h.$$ (19)

With $A_h:X_h\to X_h^{\prime }$ and $G_h:X_h\to M_h^{\prime }$, defined by $A_{h}v:=(Av){|}_{X_h}$ and $G_{h}v:=(Gv){|}_{M_h}$, $v\in X_h$, as well as ${\ell }_h:=\ell {|}_{X_h}$, the system (19) is equivalent to

(20)

where $G_h^{\prime }:M_h\to X_h^{\prime }$ is the $H^{0,0}$-adjoint.

Analogously to the continuous case, we set

$${\mathrm{\Gamma }}_h:=\{u_h\in X_h:\langle G{u}_h,\mu \rangle =0\forall \mu \in M_h\}.$$ (21)

The Brezzi equivalence theorem, see [8, Satz III.4.3], states that the left-hand-side of (19) defines an isomorphism $X_h\times M_h\to X_h^{\prime }\times M_h^{\prime }$ if and only if the following two conditions are fulfilled:

• 1.
  A is ${\mathrm{\Gamma }}_h$-elliptic,

  $$\exists {\alpha }_h>0:\langle Av,v\rangle \ge {\alpha }_h{\Vert v\Vert }_X^2\forall v\in {\mathrm{\Gamma }}_h.$$ (22)
• 2.
  The discrete inf-sup constant is positive,

  $${\beta }_h:=\underset{\lambda \in M_h\setminus \{0\}}{\inf }\underset{v\in X_h\setminus \{0\}}{\sup }\frac{\langle Gv,\lambda \rangle }{{\Vert v\Vert }_X{\Vert \lambda \Vert }_{H^{0,0}}}>0.$$ (23)

We shall employ a discretization $X_h\times M_h\subset X\times H^{0,0}$ for which the two rather restrictive conformity conditions

$${\mathrm{\Gamma }}_h\subset \mathrm{\Gamma }\text{and}{M}_h\subset G{X}_h$$ (24)

hold. The first, together with Γ-ellipticity (13) of A, implies ${\mathrm{\Gamma }}_h$-ellipticity (22) of A, and the second immediately implies the discrete inf-sup condition (23) with ${\beta }_h\ge 1$.

The square domain D is partitioned into non-overlapping open rectangles and/or triangles, collected in $D_h$. Geometric compatibility conditions are imposed following [11, p. 51]: each edge of any geometric element $K\in D_h$ is either part of the boundary ∂D or is also an edge of some other geometric element $K^{\prime }\in D_h$. By $D_h^{\partial }$ we mean the set of edges E of all geometric elements in $D_h$ (shared edges occur only once). The temporal interval $J=(0,T)$ is partitioned into open subintervals collected in $J_h$, such that $\overline{J}={\bigcup }_{I\in J_h}\overline{I}$. Quantities in $H^{0,0}$, such as ${\partial }_t\rho$, $\mathrm{\nabla }\cdot j$, g, and λ, are discretized as piecewise constants on each space–time geometric element $I\times K$, $(I,K)\in J_h\times D_h$. The conserved quantity $\rho \in H^{1,0}$ is discretized by continuous functions $J\to L_2(D)$ that are affine on each $I\in J_h$ and have values in the space of piecewise constant functions on $D_h$. The flux $j\in H^{0,\mathrm{div}}$ is discretized by assigning a flux density $j_{IE}$ to each pair $(I,E)\in J_h\times D_h^{\partial }$, such that $j_{IE}=\frac{1}{|I\Vert E|}{\int }_I{\int }_{E}j(t,x)\cdot n(x)d\sigma (x)dt$, and interpolated linearly into the inner of each geometric element $I\times K\in J_h\times D_h$. This is trivially possible on each rectangle, and corresponds to Raviart–Thomas interpolation on triangles [8, p. 141]. It is then clear that the conformity conditions (24), and therefore (22) and (23), are satisfied.

4.2. Preconditioning

Several classes of preconditioners have been developed for saddle-point systems of the form (20), see the survey [4]. Preconditioners based on computationally efficient approximations of the inverses of $A_h$ and of the Schur complement $S_h=G_h{A}_h^{-1}{G}_h^{\prime }$ are widely used. However, our $A_h$ is not, in general, invertible on the whole finite-dimensional space $X_h$. Indeed, recall from (11) that A has the block structure

(25)

where $v=(\rho ,j,g)$, so that the first block is not positive definite on $H^{1,0}$.

In [5] a similar saddle-point system for a related problem from image registration (which corresponds to $\mathrm{\#}\mathcal{T}=2$ in our case) was obtained, for which block-triangular preconditioners were proposed. We replace the triangular preconditioner by the symmetric and indefinite preconditioner ${\mathcal{P}}_h:X_h\times M_h\to X_h^{\prime }\times M_h^{\prime }$, given by

$${\mathcal{P}}_h:=\left(\begin{array}{cc}\hfill {\hat{A}}_h\hfill & \hfill 0\hfill \\ \hfill G_h\hfill & \hfill I\hfill \end{array}\right)\left(\begin{array}{cc}\hfill {\hat{A}}_h^{-1}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill -{\hat{S}}_h\hfill \end{array}\right)\left(\begin{array}{cc}\hfill {\hat{A}}_h\hfill & \hfill G_h^{\prime }\hfill \\ \hfill 0\hfill & \hfill I\hfill \end{array}\right),$$ (26)

see [2]. It was shown in [21] that this allows to transform the indefinite saddle-point system into a symmetric positive definite one for a new inner product (see (27) below), so that the conjugate gradient (CG) method can be applied. This is not possible for the preconditioned system of [5] where the spectrum is nonreal. Another well-known method to transform the indefinite saddle-point system into a symmetric positive definite one was proposed in [10,9]. There it is also required that $A_h$ be positive definite on $X_h$, which is not true in our case.

It was shown in [21, Theorem 2.1] that if

$${\mathcal{N}}_h:=\left(\begin{array}{cc}\hfill {\hat{A}}_h-A_h\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill G_h{\hat{A}}_h^{-1}{G}_h^{\prime }-{\hat{S}}_h\hfill \end{array}\right)$$ (27)

is positive definite on $X_h\times M_h$ then ${\mathcal{P}}_h^{-1}{\mathcal{A}}_h$ is symmetric positive definite with respect to the scalar product defined by ${\mathcal{N}}_h$. To that end, assume:

• 1.${\hat{A}}_h>A_h$ and $G_h{\hat{A}}_h^{-1}{G}_h^{\prime }>{\hat{S}}_h$ to assert that ${\mathcal{N}}_h>0$, as well as
• 2.${\hat{A}}_h\le M_0{A}_h$ on $\ker G_h$, for some real constant $M_0\ge 1$,
• 3.$G_h{\hat{A}}_h^{-1}{G}_h^{\prime }\le M_1{\hat{S}}_h$ for some real constant $M_1\ge 1$.

Under those assumptions, the spectrum of ${\mathcal{P}}_h^{-1}{\mathcal{A}}_h$ is positive and is contained in an interval determined by the constants $M_0$ and $M_1$ [21, Theorem 2.2], specifically, with $M_{01}:=1+\frac{1}{2M_0}-\frac{1}{2M_1}$,

$$\frac{{\lambda }_{\max }({\mathcal{P}}_h^{-1}{\mathcal{A}}_h)}{{\lambda }_{\min }({\mathcal{P}}_h^{-1}{\mathcal{A}}_h)}\le \frac{M_1+\sqrt{M_1^2-M_1}}{M_{01}-\sqrt{M_{01}^2-\frac{1}{M_0}}}\le \frac{1}{2}{(1+\sqrt{5})}^2{M}_0{M}_1.$$ (28)

Importantly, ${\hat{A}}_h\le M_0{A}_h$ in the second condition is only required to hold on $\ker G_h={\mathrm{\Gamma }}_h$, where $A_h$ is positive definite by (22), and not on all of $X_h$.

We now specify our choice of ${\hat{A}}_h$ and ${\hat{S}}_h$. For ${\hat{A}}_h$ we take the block operator (25) with the first block replaced by

$$2C_{\rho }({(\mathrm{\#}\mathcal{T})}^{-2}{\Vert {\partial }_t\rho \Vert }_{H^{0,0}}^2+T^{-2}{\Vert \rho \Vert }_{H^{0,0}}^2),$$ (29)

while the second and the third nonzero blocks are multiplied by two. Here, $C_{\rho }>0$ is computed to satisfy

(30)

for all $f\in H^1(J)$. This entails ${\hat{A}}_h\ge 2A_h$, so that ${\hat{A}}_h>A_h$ is fulfilled in the first assumption. Up to normalization, (29) is the $H^{1,0}$ norm of ρ, and this being the space where ρ is sought, it is a natural positive definite regularization of the indefinite block of A. The normalization factors have been chosen with the motivation that the constant $C_{\rho }$ can be fixed independently of T and $\mathcal{T}$, as long as the subintervals defined by $\mathcal{T}$ are of comparable length. Boundedness of ${\hat{A}}_h$ and ${\mathrm{\Gamma }}_h$-ellipticity (22) of $A_h$ yield $\langle {\hat{A}}_{h}v,v\rangle \le \Vert {\hat{A}}_h\Vert {\Vert v\Vert }_X^2\le \Vert {\hat{A}}_h\Vert {\alpha }_h^{-1}\langle Av,v\rangle$ for any $v\in \ker G_h$, so that the second assumption is fulfilled with $M_0\le \Vert {\hat{A}}_h\Vert {\alpha }_h^{-1}$. Alternatively, after discretization, if $\mathcal{T}$ is dense enough compared to the temporal discretization, the right hand side of (30) may be estimated by a multiple of . This multiple yields a possible value for $M_0\ge 2$, taking care of the first block in the relation ${\hat{A}}_h\le M_0{A}_h$; the inequality holds trivially for the other two blocks.

To obtain ${\hat{S}}_h$ we exploit the fact that the composition $G_h{\hat{A}}_h^{-1}{G}_h^{\prime }$ is an isomorphism on $H^{0,0}$. Indeed, boundedness is clear, while positivity is obtained from using the inf-sup condition (23): $\langle G_h{\hat{A}}_h^{-1}{G}_h^{\prime }\lambda ,\lambda \rangle \ge {\Vert {\hat{A}}_h\Vert }^{-1}{\Vert G_h^{\prime }\lambda \Vert }_{X^{\prime }}^2\ge {\beta }_h^2{\Vert {\hat{A}}_h\Vert }^{-1}{\Vert \lambda \Vert }_{H^{0,0}}^2$. Therefore we take for ${\hat{S}}_h$ the $C_{\lambda }$-fold of the Riesz isomorphism on $H^{0,0}$, with, say, $0<C_{\lambda }<{\beta }_h^2{\Vert {\hat{A}}_h\Vert }^{-1}$. With this choice, the first assumption holds, and the constant $M_1$ in the third assumption may be taken as $\Vert {\hat{A}}_h^{-1}\Vert {\Vert G_h\Vert }^2$. In the numerical experiments below we use $C_{\lambda }:=\frac{1}{2}$.

It can be seen from the factorization (26) that the application of the inverse of the preconditioner only requires the application of ${\hat{A}}_h^{-1}$ and ${\hat{S}}_h^{-1}$ (as well as $G_h$ and $G_h^{\prime }$). The application of ${\hat{S}}_h^{-1}$ is trivial since we work with piecewise constant basis functions. In order to apply the block-diagonal operator ${\hat{A}}_h^{-1}$, it is useful to observe that due to the choice of space–time tensor product basis functions each block of ${\hat{A}}_h$ has the Kronecker product form $T\otimes X$. The application of $T\otimes X$ and ${(T\otimes X)}^{-1}=T^{-1}\otimes X^{-1}$ is done by means of the matrix identity ${(\mathbf{T}\otimes \mathbf{X})}^{\pm 1}\mathrm{Vec}(\mathbf{u})=\mathrm{Vec}({\mathbf{X}}^{\pm 1}\mathbf{u}{\mathbf{T}}^{\pm \mathsf{T}})$, where Vec stacks the columns of a matrix one after another into one long vector. In our Matlab implementation, we use this identity extensively and apply the direct solver to compute ${\mathbf{X}}^{-1}\mathbf{u}$ column-wise, $\mathbf{u}{\mathbf{T}}^{-\mathsf{T}}$ row-wise, etc. This considerably speeds up the computation compared to the computation in the Vec form. Although we do not pursue this possibility here, we note that this tensor product structure allows to compute the unknown in a low rank tensor format (for instance the truncated singular value decomposition) [15,16] to reduce the memory requirements and the overall computational complexity. In the context of space–time discretization with PDE-constrained optimization this has been explored in [22].

5. Numerical examples

5.1. Robustness of the preconditioner

We investigate the robustness of the preconditioner with respect to the spatial discretization. The image domain is $D:=(0,3)\times (0,4)$ and the temporal interval is $J:=(0,2)$. A sequence of partitions $D_{2^{-k}}$ for $k=0,1,2,\dots ,7$, of D is obtained by subdividing D into $(3\times 2^k)\times (4\times 2^k)$ equal rectangles. In each case, the partition $J_h$ of J consists of 100 subintervals of equal length; this choice is arbitrary but does not significantly affect the following results. The input data (2) consists of two frames ${\rho }_0$ and ${\rho }_2$, hence $\mathcal{T}=\{0,2\}$, as shown in Fig. 1. The parameter choices in the penalization functional (9) are ${\alpha }_j={10}^{-4}$ and ${\alpha }_g=1$, which promotes the flux and puts a heavy penalty on the source term. We performed 20 CG iterations with the preconditioner (26) and the norm (27) as described in Section 4.2. The results are depicted in Fig. 1. In Fig. 2 (left) we report on the value of the cost functional in the course of the CG iteration, for different spatial resolutions levels k. We observe that the iteration is robust in k. In Fig. 2 (right), the convergence of the time-averaged flux over the horizontal midline (for the final CG iterate) as a function of the spatial resolution is shown. First order convergence in terms of the total number of degrees of freedom is observed.

Fig. 1.

Top: Input data ${\rho }_{\tau }$ for τ = 0 (left) and τ = 2 (right). Middle, left to right: The divergence of the computed flux, ∇ ⋅ j, at t ≈ 0, t ≈ 1 and t ≈ 2. For the purpose of visualization, the flux is interpolated from the edges to the midpoint of each geometric element. Bottom: The computed source term g at t = 0,1,2. Note the small scale of ${10}^{-3}$ in accordance with the choice of ${\alpha }_g/{\alpha }_j={10}^4$. See Section 5.1.

Fig. 2.

Left: The value of the cost functional $\mathcal{J}$ in the course of the conjugate gradient iteration. The curves for different spatial resolutions overlap. Right: First order convergence of the time-averaged flux over the horizontal midline as the spatial resolution is increased. See Section 5.1.

To further test robustness of the preconditioner, we consider the energy error ${\Vert x^{\star }-x_i\Vert }_{{\mathcal{N}}_h{\mathcal{P}}_h^{-1}{\mathcal{A}}_h}^2$, where $x_i$ is the i-th CG iterate and $x^{\star }$ is the exact solution (approximated by $x^{\star }\approx x_{30}$). We compute the number of CG iterations that are necessary to reduce the initial energy error by a factor of 100. The maximum over the spatial discretization levels k, for different values of the regularization parameters ${\alpha }_j$ and ${\alpha }_g$, is shown in Table 1. At discretization level $k=7$, the system has close to 100M degrees of freedom. We see that this number does exhibit a dependence on the regularization parameters, but stays moderate and approximately constant across the considered range.

Table 1.

Number of CG iterations to reduce the initial energy error by a factor of 100 for various choices of the regularization parameters ${\alpha }_j$ (horizontal) and ${\alpha }_g$ (vertical). The number shown is the maximum over the spatial discretization levels k = 0,…,7. See Section 5.1.

${\alpha }_g$	${\alpha }_j$
${10}^{-4}$	${10}^{-3}$	${10}^{-2}$	${10}^{-1}$	100
102	13	11	10	9	5
101	13	11	10	8	5
100	13	11	10	8	5
${10}^{-1}$	13	11	9	8	7
${10}^{-2}$	12	10	8	6	5

5.2. Vienna Prater movie

In this example we apply the method to a movie showing the Ferris wheel at the Vienna Prater park turning clockwise. A vertical dark strip moving from left to right across the image has been superimposed artificially, simulating shading, see Fig. 3. The movie consists of 200 frames of $256\times 216$ pixels each. This data is mapped to the domain $D=(0,4)\times (0,3)$ and the temporal interval $J=(0,2)$. We look at frames 91 and 92 and zoom in on two regions highlighted in Fig. 3. The computed discrete flux between these two frames is shown in Fig. 4 for the first region, and in Fig. 5 for the second region, for different values of the regularization parameters. In both cases, there is a strong flux westward (←) across the superimposed dark strip, which moves eastwards (→). There is a less distinct flux directed north–east (↗) across the bright spokes of the Ferris wheel (in the second region). It is interesting to note that the model generates a strong, almost uniform flux j across the superimposed dark strip in the opposite direction of the movement of the strip in order to transport the bright intensity from its bow to its stern. One observes finer flow pattern within the strip away from its boundary caused by the rotation of the Ferris wheel, as the graycolor displaying $\mathrm{\nabla }\cdot j$ in Fig. 5 suggests. On the other hand, the moving bright spokes of the Ferris wheel cause the intensities to flow across them in the same direction (as one might expect from the optical flow model). Concerning the role of the regularization parameters ${\alpha }_j$ and ${\alpha }_g$, we observe in Fig. 5 meaningful results for $({\alpha }_j,{\alpha }_g)=({10}^{-4},1)$ and $({\alpha }_j,{\alpha }_g)=(1,{10}^{-4})$, which are, up to scaling, visually very similar. This is so because, roughly speaking, ${\partial }_t\rho$ is decomposed into two parts that relate to each other as $1:{10}^{-4}$, and each part is captured by either $\mathrm{\nabla }\cdot j$ or g. For ${\alpha }_j={\alpha }_g={10}^{-2}$ we observe qualitatively different and less meaningful results, presumably because of overpenalization in the functional (10).

Fig. 3.

Two consecutive frames ${\rho }_{\tau }$ for τ ≈ 0.91 in the example in Section 5.2. Highlighted in white are the regions of [43,64]×[1,26] pixels (left) and [75,96]×[1,26] pixels (right). The Ferris wheel rotates clockwise and the superimposed dark vertical strip moves eastwards (→).

Fig. 4.

Computed discrete flux and source for the example in Section 5.2 in the region of [43,64]×[1,26] pixels. Top to bottom, the values of $({\alpha }_j,{\alpha }_g)$ are $({10}^{-4},1)$, $({10}^{-2},{10}^{-2})$, and $(1,{10}^{-4})$. Left: The computed discrete flux j with the background color showing the divergence ∇ ⋅ j. Right: The computed source g.

Fig. 5.

Computed discrete flux and source for the example in Section 5.2 in the region of [75,96]×[1,26] pixels. Top to bottom, the values of $({\alpha }_j,{\alpha }_g)$ are $({10}^{-4},1)$, $({10}^{-2},{10}^{-2})$, and $(1,{10}^{-4})$. Left: The computed discrete flux j with the background color showing the divergence ∇ ⋅ j. Right: The computed source g.

6. Conclusions

We have considered a version of the optical flow equations in which the image brightness evolves as a conserved quantity up to possible sources (or sinks). We have formulated the problem of estimating the optical flow and the source as an energy minimization problem. We have investigated a space–time discretization and preconditioning strategy for the resulting saddle-point equations. The discretization was shown to be stable in the Galerkin sense, and the preconditioner to be robust in the discretization resolution albeit with a mild dependence on the regularization parameters. The conjugate gradient method (with a suitable scalar product for which the preconditioned system matrix is symmetric and positive definite) allows to solve the complete space–time problem within a few iterations. The transport equation model with the chosen penalization functional produces rather global flows; we do not expect this to happen if the “kinetic energy” ${\Vert j\Vert }^2/\rho$ is penalized instead of the divergence of the flux j and wish to investigate this type of penalization next. We note, however, that our model is typical for control problems with a quadratic cost functional subject to a linear time-dependent PDE constraint, and we therefore expect the results to be applicable elsewhere.