Local Minima Free Parameterized Appearance Models

Abstract

Parameterized Appearance Models (PAMs) (e.g. Eigentracking, Active Appearance Models, Morphable Models) are commonly used to model the appearance and shape variation of objects in images. While PAMs have numerous advantages relative to alternate approaches, they have at least two drawbacks. First, they are especially prone to local minima in the fitting process. Second, often few if any of the local minima of the cost function correspond to acceptable solutions. To solve these problems, this paper proposes a method to learn a cost function by explicitly optimizing that the local minima occur at and only at the places corresponding to the correct fitting parameters. To the best of our knowledge, this is the first paper to address the problem of learning a cost function to explicitly model local properties of the error surface to fit PAMs. Synthetic and real examples show improvement in alignment performance in comparison with traditional approaches.

1. Introduction

Since the early work of Sirovich and Kirby [21] parameterizing the human face using Principal Component Analysis (PCA) and the successful eigenfaces of Turk and Pentland [23], many computer vision researchers have used PCA techniques to construct linear models of optical flow, shape or graylevel [3, 10, 4, 6, 19, 14, 5, 11]. In particular, Parameterized Appearance Models (PAMs) (e.g. eigentracking [4], active appearance models [6, 10, 17, 9, 11], morphable models [5, 14]) have proven to be an appropriate statistical tool for modeling shape and appearance variation of objects in images. In PAMs, the appearance/shape models of objects are built by performing PCA on training data. Once the models have been constructed, finding the location/configuration of an object of interest in a testing image is achieved by minimizing a cost function w.r.t. some transformation (motion) parameters; this is referred to as the fitting process.

Although widely used, PAMs suffer from two problems in the fitting process. First, they are especially prone to local minima. Second, often few, if any, of the local minima of the cost function correspond to acceptable solutions. Figures 1a,d,f illustrate these problems. Fig. 1d plots the error surface constructed by translating the testing image (Fig. 1c) around the ground truth landmarks (Fig. 1c) and computing the values of the cost function. The cost function is based on a PCA model constructed from labeled training data (Fig. 1a). Fig. 1f shows the contour plot of this error surface. As can be observed, any gradient-based optimization method is likely to get stuck at local minima, and will not converge to the global minimum. Moreover, the global minimum of this cost function is not at the desired position, the black dot of Fig. 1d, which corresponds to the correct landmarks' locations. These problems occur because the PCA model is constructed without considering the neighborhoods of the correct motion parameters (parameters that correspond to ground truth landmarks of training data). The neighborhoods determine the local minima properties of the error surface, and should be taken into account while constructing the models.

Figure 1.

Learning a better model for image alignment. (d,f): surface and contour plot of the PCA model. It has many local minima; (e, g): Local Minima Free PAM (LMF-PAM) method learns a better error surface to fit PAMs. This figure is best seen in color.

In this paper, we propose to learn the cost function (i.e. appearance model) that has a local minimum at the “expected” location and no other local minima in its neighborhood. This is done by enforcing constraints on the gradients of the cost function at the desired location and its neighborhood. Fig. 1e,g plot the error surface and contours of the learned cost function. This cost function has a local minimum in the expected place (black dot of Fig. 1e), and no other local minima near by.

2. Previous work

Over the last decade, appearance models have become increasingly important in computer vision and graphics. In particular, PAMs have been proven useful for alignment, detection, tracking, and face synthesis [5, 4, 10, 6, 17, 19, 14, 11, 24]. This section reviews PAMs and gradient-based methods for the efficient alignment of high dimensional deformation models.

2.1. PAMs

PAMs [4, 10, 6, 19, 14, 5, 24] build the objects' appearance/shape representation from the principal components of training data. Let d_i ∈ ℜ^m×1 (see notation ¹) be the i^th sample of a training set D ∈ ℜ^m×n and U ∈ ℜ^m×k the first k principal components [13]. Once the model has been constructed (i.e. U is known), tracking/alignment is achieved by finding the motion parameter p that best aligns the data w.r.t. the subspace U, i.e.

$$\underset{\mathbf{\text{c}},\mathbf{\text{p}}}{\text{min}}{\Vert \mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))-\mathbf{\text{Uc}}\Vert }_2^2$$ (1)

Here x = [x₁, y₁, …x_l, y_l]^T is the vector containing the coordinates of the pixels to track. f(x, p) is the function for geometric transformation; denote f (x, p) by [u₁, v₁, …, u_l, v_l]^T. d is the image frame in consideration, and d(f(x, p)) is the appearance vector of which the i^th entry is the intensity of image d at pixel (u_i, v_i). For affine and non-rigid transformations, (u_i, v_i) relates to (x_i, y_i) by:

$$\left[\begin{array}{c}{u}_i\\ v_i\end{array}\right]=\left[\begin{array}{cc}{a}_1& a_2\\ a_4& a_5\end{array}\right]\left[\begin{array}{c}{x}_i^s\\ y_i^s\end{array}\right]+\left[\begin{array}{c}{a}_3\\ a_6\end{array}\right]$$ (2)

with ${[x_1^s,y_1^s,\dots x_l^s,y_l^s]}^T=\mathbf{\text{x}}+{\mathbf{\text{U}}}^s{\mathbf{\text{c}}}^s$, where U^s is the non-rigid shape model learned by performing PCA on a set of registered shapes [7]. a, c^s are affine and non-rigid motion parameters respectively, and p = [a; c^s].

2.2. Optimization for PAMs

Given an image d, PAM tracking/alignment algorithms optimize (1). Due to the high dimensionality of the motion space, a standard approach to efficiently search over the parameter space is to use gradient-based methods [1, 7, 17, 4, 8, 25]. To compute the gradient of the cost function given in (1), it is common to use Taylor series expansion to approximate d(f(x, p + δp)) by d(f(x, p)) + J^d(p)δp, where ${\mathbf{\text{J}}}^{\mathbf{\text{d}}}(\mathbf{\text{p}})=\frac{\partial \mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))}{\partial \mathbf{\text{p}}}$ is the Jacobian of the image d w.r.t. to the motion parameter p [16]. Once linearized, a standard approach is to use the Gauss-Newton method for optimization [2, 4]. Other approaches learn an approximation of the Jacobian matrix with linear [7] or non-linear [20, 15] regression.

Over the last few years, several strategies for improving the fitting performance have been proposed. For examples, Black & Anandan [4] and Cootes & Taylor [7] proposed using multi-resolution schemes, Xiao et al [26] proposed using 3D models to constrain 2D solutions, de la Torre et al proposed learning filters to achieve robustness to local minima, de la Torre & Black [8], and Baker & Matthews [1] learned a PCA model invariant to rigid and non-rigid transformations. Although these methods show significant performance improvement, they do not directly address the problem of learning a cost function with no local minima. In this paper, we deliberately learn a cost function which has local minima at and only at the desired places.

3. Learning parameters of the cost functions

Gradient-based algorithms, such as the ones discussed in the previous section, might not converge to the correct location (i.e. correct motion parameters) for several reasons. First, gradient-based methods are susceptible to being stuck at local minima. Second, even when the optimizer converges to a global minimum, the global minimum might not correspond to the correct motion parameters. These two problems occur primarily because PCA has limited generalization capabilities to model appearance variation. This section proposes a method to learn cost functions that do not exhibit these two problems in training data.

3.1. A generic cost function for alignment

This section proposes a generic quadratic error function where many PAMs can be cast. The quadratic error function has the form:

$$E(\mathbf{\text{d}},\mathbf{\text{p}})=\mathbf{\text{d}}{(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))}^T\mathbf{\text{Ad}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))+2{\mathbf{\text{b}}}^T\mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))$$ (3)

Here A ∈ ℜ^m×m and b ∈ ℜ^m×1 are the fixed parameters of the function, and A is symmetric. This function is the general form of many cost functions used in the literature including Active Appearance Models [6], Eigentracking [4], and template tracking [16, 18]. For instance, consider the cost function given in (1). If p is fixed, the optimal c that minimizes (1) can be obtained using c = U^Td(f(x, p)). Substituting this back into (1) and performing some basic algebra, (1) is equivalent to: min_p d(f(x, p))^T(I_m − UU^T)d(f(x, p)). Thus (1) is a special case of (3), with A = I_m − UU^T, and b = 0_m.

For template tracking, the cost function is typically the sum of squared differences: ${\Vert \mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))-{\mathbf{\text{d}}}_{\mathit{\text{ref}}}\Vert }_2^2$, where d_ref is the reference template. This cost function is equivalent to: $\mathbf{\text{d}}{(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))}^T\mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))-2{\mathbf{\text{d}}}_{\mathit{\text{ref}}}^T\mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))$. Thus the cost function used in template tracking is also a special case of (3) with A = I_m and b = −d_ref.

3.2. Desired properties of cost functions

As discussed previously, it is desirable that the cost function have minima at and only at the ‘right’ places. In this section, we deliberately address this need as an optimization problem over A and b.

Let ${\left\{{\mathbf{\text{d}}}_i\right\}}_1^n$ be a set of training images containing the objects of interest (e.g. faces), and assume the landmarks for the object shapes are available (e.g. manually labeled facial landmarks as in Fig. 5a). Let s_i be the vector containing the landmark coordinates of image d_i. Given ${\left\{{\mathbf{\text{s}}}_i\right\}}_1^n$, we perform Procrustes analysis [7] and build the shape model as follows. First, the mean shape $\overline{\text{s}}=\frac{1}{n}{\sum }_i{\text{s}}_i$ is calculated. Second, we compute a_i the affine parameter that best transforms s̄ to s_i, and let ${\mathbf{\text{a}}}_{\mathit{\text{i}}}^{-1}$ be the inverse affine transformation of a_i. Third, ŝ_i is obtained by applying the inverse affine transformation ${\mathbf{\text{a}}}_{\mathit{\text{i}}}^{-1}$ on s_i (warping toward the mean shape). Next, we perform PCA on ${\{{\widehat{\mathbf{\text{s}}}}_i-\overline{\mathbf{\text{s}}}\}}_{i=1}^n$ to construct U^s, a basis for non-rigid shape variation. We then compute ${\mathbf{\text{c}}}_i^s$, the coefficients of ŝ_i − s̄ w.r.t. the the basis U^s. Finally, let ${\mathbf{\text{p}}}_i=[{\mathbf{\text{a}}}^i;{\mathbf{\text{c}}}_i^s]$, p_i is the parameter of image d_i w.r.t. to our shape model. Notably, the shape model and ${\left\{{\mathbf{\text{p}}}_i\right\}}_1^n$ are derived independently of the appearance model. The appearance model (i.e. the cost function E(d, p)) is what needs to be learned.

Figure 5.

(a) example of landmarks associated with each face (red dots), (b) example of shape distortion (yellow pluses), (c) example of patches for appearance modeling.

For E(d_i, p) to have a local minimum at the right place, p_i must be a local minimum of E(d_i, p). Theoretically, this requires ${\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}_i}$ to vanish, i.e.

$${\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}_i}=0\forall i$$ (4)

To learn a cost function that has few local minima, it is necessary to consider p_i's neighborhoods. Let _i = {p :lb ≤ ‖p − p_i‖₂ ≤ ub}, ${\mathcal{N}}_i^{-}=\{\mathbf{\text{p}}:{\Vert \mathbf{\text{p}}-{\mathbf{\text{p}}}_i\Vert }_2<lb\}$, ${\mathcal{N}}_i^{+}=\{\mathbf{\text{p}}:{\Vert \mathbf{\text{p}}-{\mathbf{\text{p}}}_i\Vert }_2>ub\}$. Here lb is chosen such that ${\mathcal{N}}_i^{-}$ is a set of neighbor parameters that are very close to p_i; it is satisfactory for a fitting algorithm to converge to a point in ${\mathcal{N}}_i^{-}$. ub is chosen so that the fitting algorithm is guaranteed to be initialized at a point in _i or ${\mathcal{N}}_i^{-}$. In most applications, such ub exists. For example, for tracking problems, ub can be set to the maximum movement of the object being tracked between two consecutive frames. Fig. 2 depicts the relationship between ${\mathcal{N}}_i^{-}$, _i, and ${\mathcal{N}}_i^{+}$.

Figure 2.

Neighborhoods around the ground truth motion parameter p_i (ret dot). ${\mathcal{N}}_i^{-}$: region inside the orange circle; it is satisfactory for fitting algorithms to converge to this region. ${\mathcal{N}}_i^{+}$: region outside the blue circle; alignment algorithm will not be initialized in this region. _i: shaded region, region to enforce constraints on gradients.

For a gradient descent algorithm to converge to p_i or a point close enough to p_i, it is necessary that E(d_i, .) have no local minima in _i. This implies that $\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}$ does not vanish for p ∈ _i. Notably, it is not necessary to enforce similar constraints for $\mathbf{\text{p}}\in {\mathcal{N}}_i^{-}\cup {\mathcal{N}}_i^{+}$ because of the way lb, ub are chosen. Another desirable property is that each iteration of gradient descent advances closer to the correct position. Because gradient descent walks against the gradient direction at every iteration, we would like the opposite direction of the gradient at point p ∈ _i to be similar to the optimal walking direction p_i − p. This quantity can be measured as the projection of the walking direction onto the optimal direction. Fig. 3 illustrates the rationale of this requirement. This requirement leads to the constraints:

Figure 3.

p_i: desired convergence location. Blue arrows: gradient vectors, red arrows: walking directions of gradient descent algorithm, orange arrows: optimal directions to the desired location. Performing gradient descent at p advances closer to p_i while performing gradient descent at p′ moves away from p_i.

$$\langle -{\left(\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}\right)}^T,\frac{{\mathbf{\text{p}}}_i-\mathbf{\text{p}}}{{\Vert {\mathbf{\text{p}}}_i-\mathbf{\text{p}}\Vert }_2}\rangle >0{\forall }_{\mathbf{\text{p}}}\in {\mathcal{N}}_i$$ (5)

Equations (4) and (5) specify the constraints for the ideal cost function. However, these constraints might be too stringent. Therefore, we propose to relax the constraints to get the optimization problem:

$$\begin{array}{l}\underset{\mathbf{\text{A}},\mathbf{\text{b}},\xi }{\text{min}}\frac{1}{2}\sum _i{\Vert {\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}_i}\Vert }_2^2+C\sum _i{\xi }_i\\ \text{s}.\text{t}.\langle -{\left(\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}\right)}^T,\frac{{\mathbf{\text{p}}}_i-\mathbf{\text{p}}}{{\Vert {\mathbf{\text{p}}}_i-\mathbf{\text{p}}\Vert }_2}\rangle >-{\xi }_i\forall i,p\in {\mathcal{N}}_i\\ {\xi }_i\ge 0\forall i\end{array}$$ (6)

Here ${\Vert {\frac{\partial E({\mathbf{\text{d}}}_i,\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}_i}\Vert }_2^2$ is required to be small instead of strictly zero. ξ_i's are slack variables for constraints in (5) which allows for penalized constraint violation. C is the parameter controlling the trade-off between having few local minima and having local minima at the right places.

The gradient of the function E(d, p) plays a fundamental role in the above optimization problem. To compute the gradient $\frac{\partial \mathit{\text{E}}(\mathbf{\text{d}},\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}$, it is common to use first order Taylor series expansion to approximate d(f(x, p + δp)) by d(f(x, p)) + J^d(p) δp, where ${\mathbf{\text{J}}}^{\mathbf{\text{d}}}(\mathbf{\text{p}})=\frac{\partial \mathbf{\text{d}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))}{\partial \mathbf{\text{p}}}$is the spatial intensity gradient of the image d w.r.t. to the motion parameter p [16]. This yields:

$${\left(\frac{\partial E(\mathbf{\text{d}},\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}\right)}^T\approx 2{({\mathbf{\text{J}}}^{\mathbf{\text{d}}}(\mathbf{\text{p}}))}^T(\mathbf{\text{Ad}}(\mathbf{\text{f}}(\mathbf{\text{x}},\mathbf{\text{p}}))+\mathbf{\text{b}})$$ (7)

Substituting (7) into (6), we obtain a quadratic optimization problem with linear constraints over A and b.

3.3. Practical issues and alternative fitting methods

In practice, there is an issue regarding the optimization of (6): the small components of $\frac{\partial \mathit{\text{E}}(\mathbf{\text{d}},\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}$ tend to be neglected when optimizing (6). This occurs due to the magnitude difference between some columns of J^d(p). For example, in (2), the magnitudes of the Jacobians of d(f (x, p)) w.r.t. to a₁, a₂, a₄, a₅ can be much larger than the magnitudes of the Jacobians of d(f (x, p)) w.r.t. to a₃, a₆.

To address this concern, we consider an alternative optimization strategy where the update rule at iteration k^th is:

$$\begin{array}{l}{\mathbf{\text{p}}}^{k+1}={\mathbf{\text{p}}}^k+{\Delta }^{\mathbf{\text{d}}}({\mathbf{\text{p}}}^k)\\ \text{with}{\Delta }^{\mathbf{\text{d}}}({\mathbf{\text{p}}}^k)=-\frac{1}{2}{\mathbf{\text{H}}}^{\mathbf{\text{d}}}{({\mathbf{\text{p}}}^k)}^{-1}{\left({\frac{\partial E(\mathbf{\text{d}},\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}^k}\right)}^T\\ {\mathbf{\text{H}}}^{\mathbf{\text{d}}}({\mathbf{\text{p}}}^k)={\mathbf{\text{J}}}^{\mathbf{\text{d}}}{({\mathbf{\text{p}}}^k)}^T{\mathbf{\text{J}}}^{\mathbf{\text{d}}}({\mathbf{\text{p}}}^k)\end{array}$$ (8)

The update rule of the above algorithm is a variant of Newton iteration. Intuitively, H^d(p^k) is similar to the Hessian of E(d, p) at p^k, and it acts as a normalization matrix for the gradient. This algorithm is indeed a reasonable optimization scheme for cost functions in which A is symmetric positive semidefinite with all eigenvalues less than or equal to 1. See Theorem 1 in the Appendix for the proof.

Similar to the case of gradient descent, requiring the incremental updates to vanish at only at the places corresponding to acceptable solutions yields the following optimization problem:

$$\begin{array}{l}\underset{\mathbf{\text{A}},\mathbf{\text{b}},\xi }{\text{min}}\frac{1}{2}\sum _i{\Vert {\Delta }^{{\mathbf{\text{d}}}_i}({\mathbf{\text{p}}}_i)\Vert }_2^2+C\sum _i{\xi }_i\\ \text{s}.\text{t}.\langle {\Delta }^{{\mathbf{\text{d}}}_i}(\mathbf{\text{p}}),\frac{{\mathbf{\text{p}}}_i-\mathbf{\text{p}}}{{\Vert {\mathbf{\text{p}}}_i-\mathbf{\text{p}}\Vert }_2}\rangle >-{\xi }_i\forall i,{\forall }_{\mathbf{\text{p}}}\in {\mathcal{N}}_i\\ {\xi }_i\ge 0\forall i.\end{array}$$ (9)

A is also constrained to be a symmetric positive semidefinite matrix where eigenvalues are less than or equal one. By incorporating the ideas of maximal margin and regularization, we obtain:

$$\begin{array}{l}\underset{\mathbf{\text{A}},\mathbf{\text{b}},\xi }{\text{min}}\frac{1}{2}\sum _i{\Vert {\Delta }^{{\mathbf{\text{d}}}_i}({\mathbf{\text{p}}}_i)\Vert }_2^2+C\sum _i{\xi }_i+C_2\Omega (\mathbf{\text{A}},\mathbf{\text{b}})\\ \text{s}.\text{t}.\langle {\Delta }^{{\mathbf{\text{d}}}_i}(\mathbf{\text{p}}),\frac{{\mathbf{\text{p}}}_i-\mathbf{\text{p}}}{{\Vert {\mathbf{\text{p}}}_i-\mathbf{\text{p}}\Vert }_2}\rangle \ge C_3-{\xi }_i\forall i,{\forall }_{\mathbf{\text{p}}}\in {\mathcal{N}}_i\\ {\xi }_i\ge 0\forall i\&\mathbf{\text{A}}\in {\mathcal{H}}_m,\end{array}$$ (10)

where _m denotes the set of all m × m symmetric matrices of which all eigenvalues are non-negative and less than or equal to one. Ω (A, b) is the regularization term for A and b, C₂ is the weight for the regularization term, and C₃ is the user-defined margin size. Since Δ^d_i(p_i) is linear in terms of A and b, this is a quadratic programming problem with linear constraints, provided the requirement A ∈ _m can be described by linear constraints.

Of course, one can derive a similar learning problem for A and b where the Newton method is the optimizer of choice. The incremental update in Newton iteration is:

$$-\frac{1}{2}{[{\mathbf{\text{J}}}^{\mathbf{\text{d}}}{({\mathbf{\text{p}}}^k)}^T{\mathbf{\text{AJ}}}^{\mathbf{\text{d}}}({\mathbf{\text{p}}}^k)]}^{-1}{\left({\frac{\partial E(\mathbf{\text{d}},\mathbf{\text{p}})}{\partial \mathbf{\text{p}}}|}_{{\mathbf{\text{p}}}^k}\right)}^T$$ (11)

However, each Newton iteration has to invert J^d(p^k)^T AJ^d(p^k). As a result, learning A and b becomes much harder because the optimization problem is no longer quadratic with linear constraints.

4. Special cases and experiments

Sec. 3.3 proposes a method for learning generic A and b. However, in specific situations, A and b can be further parameterized. The benefits of further parameterization are threefold. First, the number of parameters to learn can be reduced. Second, the relationship between A and b can be established. Third, the constraint that A ∈ _m can be replaced by a set of linear constraints. This section provides the formulation for two special cases, namely weighted template alignment and weighted-basis AAM alignment. Experimental results on synthetic and real data are included.

4.1. Weighted template alignment

As shown in Sec. 3.1, template alignment is a special case of (3) in which A = I_m, and b = −d_ref. In template alignment, pixels of the template are weighed equally; however, there is no reason why this is optimal. Here, we propose learning the weights of template pixels to avoid local minima in template matching.

Consider the weighted sum of squared differences: (d(f(x, p)) − d_ref)^Tdiag(w)(d(f(x, p)) − d_ref), where, w is the weight vector for the template's pixels. This cost function is equivalent to (3) with A = diag(w) and b = −diag(w)d_ref. The constraint A ∈ _m can be imposed by requiring 0 ≤ w_i ≤ 1. Furthermore, in this setting, ${\Vert {\Delta }^{{\mathbf{\text{d}}}_i}({\mathbf{\text{p}}}_i)\Vert }_2^2=0\forall i$. Thus (10) becomes a linear programming problem with linear constraints over w.

To demonstrate this idea, we create a synthetic template of an isotropic Gaussian (Fig. 4a). Suppose the task is to locate the template inside an image containing the template (Fig. 4c), starting at an arbitrary location. Fig. 4d plots the error surface of the naive cost function (sum of squared differences). The value of this error surface at a particular pixel (x, y) is calculated by computing the sum of squared differences between the template and the circular patch centered at (x, y). Similarly, the error surface of the learned cost function (weighted sum of squared differences) is calculated and displayed in Fig. 4e. The learned template weights are shown in Fig. 4b; brighter pixels mean higher weights. As can be seen, the naive cost function has a fence of local maxima surrounding the template location. This prevents alignment algorithms from converging to the desired location. The learned cost function is convex, and therefore, is more suitable for this particular template.

Figure 4.

Learning to weight template's pixels. (a) synthetic template of an isotropic Gaussian. (b) the learned weights, brighter pixels mean higher weights. (c) an image containing the template. (d) error surface of the sum of squared differences. (e) error surface of the weighted sum of squared differences with the learned weights given in (b).

The template's weights given in Fig. 4b are learned by optimizing (10) with the following parameter settings: Ω(A, b) = 0, C₂ = 0, C₃ = 10⁻², C = 1. The linear constraints are reduced to a set of 5000 constraints obtained by random sampling. How to deal with infinitely many constraints is discussed in more detail in Sec. 4.2.

4.2. Weighted-basis for AAM alignment

As shown in Sec. 3.1, AAM alignment is a special case of (3) in which $\mathbf{\text{A}}={\mathbf{\text{I}}}_m-{\mathbf{\text{UU}}}^T={\mathbf{\text{I}}}_m-{\sum }_1^k{\mathbf{\text{u}}}_i{\mathbf{\text{u}}}_i^T$, and b = 0. U is the set of k first eigenvectors from the total of K PCA basis of the training data subspace. k (≤ K) is usually chosen experimentally. In this section, we propose to use all K eigenvectors, but weigh them differently. Specifically, we learn A which has the form: $\mathbf{\text{A}}={\mathbf{\text{I}}}_m-{\sum }_1^K{\lambda }_i{\mathbf{\text{u}}}_i{\mathbf{\text{u}}}_i^T$. To ensure that A ∈ _m, we require 0 ≤ λ_i ≤ 1. Let w = [λ^T b^T]^T. Substituting this into (10) we get a quadratic programming problem with linear constraints on w.

To demonstrate this idea, we perform experiments on the Multi-PIE database [12]. This database consists of facial images of 337 subjects taken under different illuminations, expressions and poses. We only make use of the directly-illuminated frontal face images under five expressions (smile, disgust, squint, surprise and scream). Our dataset contains 1100 images, 400 are selected for training, 200 are used for validation (parameter tuning), and the rest are reserved for testing. Each face is manually labeled with 68 landmarks, as shown in Fig. 5a. Images are down sampled to 120 × 160 pixels.

The shape model is built as described in Sec. 3.2. The final shape model requires 10 coefficients (6 affine + 4 non-rigid) to describe a shape. For object appearance, we extract intensity values of pixels inside the patches located at the landmarks (Fig. 5c).

The training data is further divided into two subsets, one contains 300 images and the other contains 100 images. U is obtained by performing PCA on the subset of 300 images. The second subset is used to set up the optimization problem (10). For better generalization, (10) is constructed without using images in the first training subset. To avoid _i being of infinite size, we restrict our attention to a set of 200 random samples from _i. The random samples are drawn by introducing random Gaussian perturbation to the correct shape parameter p_i.

Following the approach by Tsochantarisdis et al [22] for minimizing a quadratic function with an exponentially large number of linear constraints, we maintain a smaller subset of active constraints S and optimize (10) iteratively. We repeat the following steps for 10 iterations: (i) empty S; (ii) randomly choose 20 training images; (iii) for each chosen training image d_i, find the 100 most violated constraints from _i and include them in S; (iv) run quadratic programming with the reduced set of constraints.

Testing data are generated by randomly perturbing the components of p_i, the correct shape parameters of test image d_i. Perturbation amounts are generated from a zero mean Gaussian distribution with standard deviation PerMag × [0.05 0.05 1 0.05 0.05 1 2 2 2 2]^T. PerMag controls the overall difficulty of the testing data. The relative perturbation amounts of shape coefficients are determined to simulate possible motion in tracking, and this is estimated visually. Fig. 5b shows an example of shape perturbation, the ground truth landmarks are marked in red (circles), while the perturbed shape is shown in yellow (pluses).

Table 1 describes the experimental results with four difficulty levels of testing data (controlled by PerMag). The performance of the learned cost function is compared with four other cost functions constructed using PCA with popular energy settings (70%, 80%, 90%, and 100%). As can be observed, when the amount of perturbation is small, PCA models with higher energy levels perform better. However, as the amount of pertubation increases, PCA models with lower energy levels perform better. This suggests that cost functions using fewer basis vectors have less local minima while cost functions using more basis vectors are more likely to have local minima at the ‘right’ places. Thus it is unclear what the energy for the PCA model should be. On the other hand, the learned cost function performs significantly better than the PCA models for most difficulty levels. In this experiment, we use $\Omega (\mathbf{\text{A}},\mathbf{\text{b}})={\Vert \mathbf{\text{b}}\Vert }_2^2$, C = 2, C₂ = 0.1, and C₃ = 0.01. The parameters are tuned using the validation set.

Table 1.

Alignment results of different methods for four different difficulty levels of testing data (PerMag). Initial is the initial amount of perturbation before running any alignment algorithm. PCA e% is the cost function constructed using PCA preserving e% of energy. The table shows the means and standard deviations of mis-alignment (average over 68 landmarks and over testing data). The unit for measurement is pixel.

PerMag	0.75	1.00	1.25	1.5
Initial	0.75±.25	1.08±.38	1.37±.52	1.54±.54
PCA 100%	0.37±.18	0.41±.25	0.55±.45	0.60±.51
PCA 90%	0.36±.20	0.43±.33	0.47±.36	0.60±.65
PCA 80%	0.40±.23	0.43±.34	0.49±.37	0.57±.50
PCA 70%	0.41±.20	0.43±.25	0.47±.30	0.55±.46
Ours	0.37±.19	0.40±.25	0.43±.29	0.48±.39

5. Conclusion

In this paper, we have proposed a method for learning the cost functions for PAMs. We directly address the problem of learning cost functions that have local minima at and only at the desired places. The task of learning a cost function is formulated as optimizing a quadratic function under some linear constraints. To the best of our knowledge, this is the first paper that addresses this problem. Encouraging results have been achieved in the context of template matching and AAM fitting. Further work needs to address how to select the most interesting points in the error surface to reduce the number of constraints in the optimization.

Appendix

This section states and proves a theorem used to justify the optimization algorithm given in (8).

Theorem 1

Consider an m-dimensional function f(x) of p-dimensional variable x, and suppose we have to minimize the function: E(x) = f(x)^T A f(x) + 2b^T f(x), where A ∈ _m. Consider an iterative optimization method which has the following update rule:

$$\begin{array}{ll}\hfill & {\mathbf{\text{x}}}^{\mathit{\text{new}}}={\mathbf{\text{x}}}^{\mathit{\text{old}}}+\delta \mathbf{\text{x}}\hfill \\ \text{with}\hfill & \delta \mathbf{\text{x}}=-{\mathbf{\text{H}}}^{-1}{\mathbf{\text{J}}}^T(\mathbf{\text{A}}f(\mathbf{\text{x}})+\mathbf{\text{b}})\hfill \\ \text{and}\hfill & \mathbf{\text{J}}={\frac{\partial f}{\partial \mathbf{\text{x}}}|}_{{\mathbf{\text{x}}}^{\mathit{\text{old}}}},\mathbf{\text{H}}={\mathbf{\text{J}}}^T\mathbf{\text{J}}\hfill \end{array}$$ (12)

The above optimization method, when started sufficiently close to a regular local minimum, will converge to that local minimum. Here, a point x₀ is said to be regular if H is not singular and the Taylor series of f(·) converges for every point in the neighborhood of x₀.

Proving Theorem 1 requires two lemmas. We now state and prove those two lemmas.

Lemma 1

A ∈ _m if and only if I_m − A ∈ _m.

Proof

This lemma can be proven easily, based on:

$$0\le \frac{{\mathbf{\text{u}}}^T\mathbf{\text{Au}}}{{\mathbf{\text{u}}}^T\mathbf{\text{u}}}\le 1\iff 0\le \frac{{\mathbf{\text{u}}}^T({\mathbf{\text{I}}}_m-\mathbf{\text{A}})\mathbf{\text{u}}}{{\mathbf{\text{u}}}^T\mathbf{\text{u}}}\le 1\forall \mathbf{\text{u}}$$ (13)

Lemma 2

A ∈ _m if and only if there exists a positive integer k, scalars α_i's, and matrices B_i's such that:

1. ${\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i$ is invertible $\forall i=\overline{1,k}$.

2. ${\alpha }_i\ge 0\forall i=\overline{1,k}$, and ${\sum }_{i=1}^k{\alpha }_i\le 1$

3. $\mathbf{\text{A}}={\sum }_{i=1}^k{\alpha }_i{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T$

Proof for sufficiency conditions

Suppose there exist k, α_i's, and B_i's that satisfy all all three conditions above. Because A is a linear combination of symmetric matrices, A is also symmetric. We only need to prove that A is positive semidefinite of which all eigenvalues are less than or equal to 1. Consider v^T Av for an arbitrarily vector v ∈ ℜ^m:

$$\begin{array}{l}{\mathbf{\text{v}}}^T\mathbf{\text{Av}}=\sum _{i=1}^k{\alpha }_i{\mathbf{\text{v}}}^T{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}\\ =\sum _{i=1}^k{\alpha }_i{\mathbf{\text{v}}}^T{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}\\ =\sum _{i=1}^k{\alpha }_i{\Vert {\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}\Vert }_2^2\end{array}$$ (14)

We know that ${\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T$ is a projection matrix and ${\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}$ is the projection of v in the subspace B_i. Thus we have ${\Vert {\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}\Vert }_2^2\le {\Vert \mathbf{\text{v}}\Vert }_2^2\forall i$. Therefore:

$${\mathbf{\text{v}}}^T\mathbf{\text{Av}}\le (\sum _{i=1}^k{\alpha }_i){\Vert \mathbf{\text{v}}\Vert }_2^2\le {\Vert \mathbf{\text{v}}\Vert }_2^2$$ (15)

Furthermore, we have v^T Av ≥ 0 because ${\Vert {\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T\mathbf{\text{v}}\Vert }_2^2\ge 0$, and α_i ≥ 0 ∀i. Combining this with the inequality in (15), we have: 0 ≤ v^T Av ≤ v^T v Since these inequalities hold for arbitrary vector v ∈ ℜ^m, A must be an element of _m.

Proof for necessary conditions

Suppose A ∈ _m. Consider the singular value decomposition of A, A = UΛU^T. Here, the columns of U are orthonormal vectors. Λ is a diagonal matrix, Λ = diag([λ₁, …, λ_m]) with 0 ≤ λ_i ≤ 1∀i. Without loss of generality, suppose λ₁ ≥ λ₂ ≥ … ≥ λ_m. We have:

$$\begin{array}{l}\mathbf{\text{A}}={\mathbf{\text{UAU}}}^T=\sum _{i=1}^m{\lambda }_i{\mathbf{\text{u}}}_i{\mathbf{\text{u}}}_i^T\\ =\sum _{i=1}^{m-1}({\lambda }_i-{\lambda }_{i+1})(\sum _{j=1}^i{\mathbf{\text{u}}}_j{\mathbf{\text{u}}}_j^T)+{\lambda }_m(\sum _{j=1}^m{\mathbf{\text{u}}}_j{\mathbf{\text{u}}}_j^T)\end{array}$$ (16)

Let α_i = λ_i − λ_i+1 for i =1, …, m − 1, and α_m = λ_m. Let B_i = [u₁…u_i] for $i=\overline{1,m}$. Since ${\{u_i\}}_1^m$ is a set of orthonormal vectors, ${\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i={\mathbf{\text{I}}}_i$ an identity matrix. Therefore, ${\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T={\mathbf{\text{B}}}_i{\mathbf{\text{B}}}_i^T={\sum }_{j=1}^i{\mathbf{\text{u}}}_j{\mathbf{\text{u}}}_j^T$. Hence:

$$\mathbf{\text{A}}=\sum _{i=1}^m{\alpha }_i{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T$$ (17)

Finally, we have α_i ≥ 0 ∀i and ${\sum }_{i=1}^m{\alpha }_i={\lambda }_1\le 1$. This completes our proof for Lemma 1 D.

Proof of Theorem 1

From Lemmas 1 and 2 we know that ∃α_i ≥ 0, $\exists {\mathbf{\text{B}}}_i:{\mathbf{\text{I}}}_m-\mathbf{\text{A}}={\sum }_{i=1}^k{\alpha }_i{\mathbf{\text{B}}}_i{({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^T$ and ${\sum }_1^k{\alpha }_i\le 1$. To prove Theorem 1, let us first consider the optimization of the following function:

$$\begin{array}{l}{E}_2(\mathbf{\text{x}},\{{\mathbf{\text{c}}}_i\})=\sum _{i=1}^k{\alpha }_i{\Vert f(\mathbf{\text{x}})-{\mathbf{\text{B}}}_i{\mathbf{\text{c}}}_i\Vert }_2^2\\ +{\alpha }_0{\Vert f(\mathbf{\text{x}})\Vert }_2^2+2{\mathbf{\text{b}}}^{T}f(\mathbf{\text{x}})\end{array}$$ (18)

with ${\alpha }_0=1-{\sum }_{i=1}^k{\alpha }_i$. One way to optimize this function is using coordinate descent, alternating between:

1. minimizing E₂ w.r.t. x while fixing {c_i}.

2. minimizing E₂ w.r.t. {c_i} while fixing x.

To minimize E₂ w.r.t. x while fixing {c_i}, we can use the Newton method:

$${\mathbf{\text{x}}}^{\mathit{\text{new}}}={\mathbf{\text{x}}}^{\mathit{\text{old}}}-{\left(\frac{{\partial }^2{E}_2}{\partial {\mathbf{\text{x}}}^2}\right)}^{-1}{\left(\frac{\partial E_2}{\partial \mathbf{\text{x}}}\right)}^T$$

Using the first order Taylor approximation, we have f(x + δx) ≈ f(x) + Jδx with $\mathbf{\text{J}}=\frac{\partial \mathit{\text{f}}}{\partial \mathbf{\text{x}}}$

$$\begin{array}{l}\text{Thus}{E}_2(\mathbf{\text{x}}+\delta \mathbf{\text{x}},\{{\mathbf{\text{c}}}_i\})\approx E_2(\mathbf{\text{x}},\{{\mathbf{\text{c}}}_i\})+\delta {\mathbf{\text{x}}}^T{\mathbf{\text{J}}}^T\mathbf{\text{J}}\delta \mathbf{\text{x}}\\ +2\delta {\mathbf{\text{x}}}^T{\mathbf{\text{J}}}^T(f(x)-\sum _{i=1}^k{\alpha }_i{\mathbf{\text{B}}}_i{\mathbf{\text{c}}}_i+\mathbf{\text{b}})\end{array}$$ (19)

$$\begin{array}{cc}\text{Hence}& \frac{\partial E_2}{\partial \mathbf{\text{x}}}\approx 2{(f(x)-\sum _{i=1}^k{\alpha }_i{\mathbf{\text{B}}}_i{\mathbf{\text{c}}}_i+\mathbf{\text{b}})}^T\mathbf{\text{J}}\end{array}$$ (20)

$$\frac{{\partial }^2{E}_2}{\partial {\mathbf{\text{x}}}^2}\approx 2{\mathbf{\text{J}}}^T\mathbf{\text{J}}$$ (21)

Therefore, we have the Newton update rule:

$${\mathbf{\text{x}}}^{\mathit{\text{new}}}={\mathbf{\text{x}}}^{\mathit{\text{old}}}-{({\mathbf{\text{J}}}^T\mathbf{\text{J}})}^{-1}{\mathbf{\text{J}}}^T(f(x)-\sum _{i=1}^k{\alpha }_i{\mathbf{\text{B}}}_i{\mathbf{\text{c}}}_i+\mathbf{\text{b}})$$ (22)

When x is fixed, $\{{\mathbf{\text{c}}}_{\mathit{\text{i}}}^{\ast }(\mathbf{\text{x}})\}$ that globally minimize E₂ are:

$${\mathbf{\text{c}}}_i^{\ast }(\mathbf{\text{x}})={({\mathbf{\text{B}}}_i^T{\mathbf{\text{B}}}_i)}^{-1}{\mathbf{\text{B}}}_i^{T}f(\mathbf{\text{x}})$$ (23)

Combining (22) and (23), we have the update rule for minimizing E₂: x^new = x^old − (J^T J)⁻¹J^T[Af(x) + b] This update rule is exactly the same as the update rule given in (12). As a result, (12) will always lead us to a local minimum of E₂.

We now prove that a local minimum of E₂ obtained by (12) will be a local minimum of E. Suppose $({\mathbf{\text{x}}}_0,\{{\mathbf{\text{c}}}_{\mathit{\text{i}}}^{\ast }({\mathbf{\text{x}}}_0)\})$ is a local minimum of E₂, we have ∃ε₁ > 0 such that $E_2({\mathbf{\text{x}}}_0,\{{\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0)\})\le E_2({\mathbf{\text{x}}}_0+\delta \mathbf{\text{x}},\{{\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0)+\delta {\mathbf{\text{c}}}_i)\})\forall \delta \mathbf{\text{x}}$, $\delta {\mathbf{\text{c}}}_i:{\Vert \delta \mathbf{\text{x}}\Vert }_2^2+{\sum }_i{\Vert \delta {\mathbf{\text{c}}}_i\Vert }_2^2<{\epsilon }_1$. Because ${\mathbf{\text{c}}}_{\mathit{\text{i}}}^{\ast }(\mathbf{\text{x}})$ is a continuous function in terms of x, we can always find ε₂ > 0 small enough such that ∀δx if ${\Vert \delta \mathbf{\text{x}}\Vert }_2^2<{\epsilon }_2$ then ${\Vert \delta \mathbf{\text{x}}\Vert }_2^2+{\sum }_i{\Vert {\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0+\delta \mathbf{\text{x}})-{\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0)\Vert }_2^2<{\epsilon }_1$. Thus ∃ε₂ such that $E_2({\mathbf{\text{x}}}_0,\{{\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0)\})\le E_2({\mathbf{\text{x}}}_0+\delta \mathbf{\text{x}},\{{\mathbf{\text{c}}}_i^{\ast }({\mathbf{\text{x}}}_0+\delta \mathbf{\text{x}})\})\forall \delta \mathbf{\text{x}}:{\Vert \delta \mathbf{\text{x}}\Vert }_2^2<{\epsilon }_2$. On the other hand, one can easily verify that $E_2(\mathbf{\text{x}},\{{\mathbf{\text{c}}}_i\})\ge E_2(\mathbf{\text{x}},\{{\mathbf{\text{c}}}_i^{\ast }(\mathbf{\text{x}})\})=E(\mathbf{\text{x}})\forall \mathbf{\text{x}}$ Therefore, we have ∃ε₂ > 0 such that $E({\mathbf{\text{x}}}_0)\le E({\mathbf{\text{x}}}_0+\delta \mathbf{\text{x}})\forall \delta \mathbf{\text{x}}:{\Vert \delta \mathbf{\text{x}}\Vert }_2^2<{\epsilon }_2$. Hence, x₀ must be a local minimum of E.

To sum up, we have shown that (12) will converge to a local minimum of E₂. Furthermore, a local minimum of E₂ found by (12) is also a local minimum of E. Thus the update rule given in (12) is guaranteed to converge to a local minimum of E. This concludes our proof for Theorem 1 D.