Bayesian Stereo Matching Method Based on Edge Constraints

Abstract

A new global stereo matching method is presented that focuses on the handling of disparity, discontinuity and occlusion. The Bayesian approach is utilized for dense stereo matching problem formulated as a maximum a posteriori Markov Random Field (MAP-MRF) problem. In order to improve stereo matching performance, edges are incorporated into the Bayesian model as a soft constraint. Accelerated belief propagation is applied to obtain the maximum a posteriori estimates in the Markov random field. The proposed algorithm is evaluated using the Middlebury stereo benchmark. Our experimental results comparing with some state-of-the-art stereo matching methods demonstrate that the proposed method provides superior disparity maps with a subpixel precision.

1. Introduction

Binocular stereo vision infers 3D scene geometry from a pair of images with different viewpoints. Dense stereo matching is a key computational step for 3D reconstruction and has been one of the most active research topics in the stereo vision field. Given a pair of images, the aim of stereo matching is to determine the disparity values of the pixels belonging to the image selected as reference view. However, dense stereo matching is difficult for the following reasons: (1) light variations, optical blurring, and sensor noise corrupts the input images, (2) lacking texture in input images may lead to ineffectiveness in using intensity consistency constraints, (3) disparity discontinuities, caused by sudden disparity changes, often appear at object boundaries and are difficult to detect; the smoothness constraint may break-down in these boundaries, and (4) some portions of objects are visible by one camera but not by the other camera (occlusion), causing difficulties in computing reasonable disparity values in affected regions.

In general, depending on whether the matching method relies on local window-based computation or the minimization of a global energy function, dense stereo matching can be broadly classified into two categories: local methods and global methods. In local methods, the disparity computed at a given point depends only on intensity values within a finite window. Many effective local algorithms for stereo matching have been reported [1–3]. However, when local structures in an image are similar, it may be very difficult to find their correspondences in the other image without global reasoning. Although a high accuracy is not guaranteed in local methods, they have advantages in computational speed and parallelism over global methods.

A stereo algorithm belongs to global methods if there is a global cost function to be optimized, which considers the correlation of disparities in the neighborhood. Hence, the key of global algorithms is not only to define a good cost function but also to provide an effective computational method for global optimization. Among various global methods, the Bayesian approach [4] treats the stereo matching problem as finding the “best guessed” solution. Depending on the computation model, this approach can be classified into two categories: dynamic programming-based and MRFs-based.

The dynamic programming method [5] assumes that the occlusion cost is identical in each scanline. Ignoring the dependency between scanlines leads to the well-known “streaking” artifacts. In contrast, the MRFs-model based methods rely on energy minimization algorithms, such as Graph Cut (GC) [6], and Belief Propagation (BP) [4]. Both of these algorithms can be implemented efficiently to compute the minimum for an MAP-MRF whose energy function is Potts or Generalized Potts [4]. Experiments have shown [7] that, the solution produced by GC tends to be smoother than BP, but BP is significantly faster than GC, even up to real-time [8], while achieving a similar performance with GC.

Traditional global algorithms [4][6][9] usually give good results in most regions of an image. However, in the regions containing occlusions and disparity discontinuities, disparity errors often increase substantially. Hence, explicit smoothness assumptions have to be incorporated in these particular regions. Segment-based priors have been widely explored [4][10–13] due to their first-rank performance. The matching results greatly depend on whether the segmentation results are coincident in the locations where the occlusions and disparity discontinuities take place. But the consistency can hardly be achieved when tackling the highly-textured areas or the nonplanar surfaces with a uniform color. This problem can be solved by planar fitting after segmentation with impressive results [10][11]. However, the computational complexity increases.

Different from the methods mentioned above, our algorithm emphasizes on the edge information of the images. Assuming that discontinuities and occlusions occur on some edges, we explore the characteristics of the normal edges and the disparity discontinuities in reference image of the input image pairs. The edge information is incorporated into our MAP-MRF stereo matching model to obtain better stereo matching results. Another contribution of the proposed algorithm is that a local affine model is used to aggregate the matching costs. The aggregated matching costs are updated based on the current understanding of which pixels in the reference image are occluded.

This paper is organized as follows: Section 2 gives a mathematic model of the proposed algorithm. In Section 3, the framework of the proposed algorithm is described. Section 4 presents experimental results in which our algorithm and other algorithms are compared based on images in the Middlebury dataset [14]. Finally, conclusion and the future work are discussed in Section 5.

2. Related Works

In this section, we review the related models, the Maximum a posteriori Markov Random Field (MAP-MRF) model and the local affine model, which are used in the proposed algorithm.

2.1 MAP-MRF Model

The Bayesian approach provides a promising way for solving the problems of stereo matching, that is to find the best disparity map D given the observations I_L and I_R, according to maximizing the posterior probability P(D|I_L, I_R). By Bayes’ rule P(D|I_L, I_R) ∝ P(I_L, I_R|D)P(D), where P(D) is called the prior. Accordingly, the disparity problem can be modeled as pair-wise Markov Random Field (MRF), encoding the smoothness. If N(p) is the neighborhood of pixel p, pixel q belongs to N(p), and disparities of p and q are d_p and d_q, respectively. According to the Markovian property and Gibbs distribution, the basic Bayesian model can be expanded as:

$$P(D\mid I_L,I_R)\propto {\prod }_p\mathit{exp}\left(-\mathrm{\Phi }(d_p)\right){\prod }_p{\prod }_{q\in N(p)}\mathit{exp}\left(-\psi (d_p,d_q)\right)$$ (1)

where ψ(d_p, d_q) penalizes the different assignments of neighbor sites when no discontinuity exists between them. Φ(d_p) is the matching cost function of pixel p with disparity d_p. If (1) is written as a Gibbs distribution with energy function: E(D|I_L, I_R) = −log(P(D|I_L, I_R)), then, (1) has the familiar energy form:

$$E(D\mid I_L,I_R)=E_{\mathit{data}}+E_{\mathit{smooth}}$$ (2)

where E_smooth is defined to measure the disparity smoothness between neighboring segment pairs, and E_data measures the disagreement of segments and their matching regions based on the assumed disparity planes. Depending on the different solving methods, we can maximize the posterior probability P(D|I_L, I_R) or minimize the energy function E(D|I_L, I_R).

Sun et al [4] incorporated segmentation results into (1) as soft constraints under a probabilistic framework. They point out that low-level visual cues (e.g. edge, corner) can also be incorporated into the basic Bayesian model. Motivated by Sun’s method, we incorporate the edge classification result into (1) as a soft constraint. In this way, the probabilistic framework becomes:

$$P(D\mid I_L,I_R)\propto {\prod }_p\mathit{exp}\left(-\mathrm{\Phi }(d_p)\right){\prod }_p{\prod }_{q\in N(p)}\mathit{exp}\left(-\psi (d_p,d_q)\right)\mathit{exp}\left(-{\rho }_{\mathit{edge}}(d_p,d_q)\right)$$ (3)

where ρ_edge(d_p, d_q) is the edge constraint between neighbor sites, which can be defined as:

$${\rho }_{\mathit{edge}}({\text{d}}_{\text{p}},{\text{d}}_{\text{q}})=\{\begin{array}{c}0,\mathit{edge}(p)=\mathit{edge}(q)\\ {\lambda }_{\mathit{edge}},\mathit{edge}(p)\ne \mathit{edge}(q)\end{array}$$ (4)

where edge(p) is a label, which is used to judge whether the pixel p is in a discontinuity and λ_edge is a constant. In general, the larger the λ_edge is, the more difficult to pass the message between neighbor sites.

2.2 Local Affine Model

An aggregation step is needed to aggregate each pixel’s matching cost over a weight region to reduce the matching ambiguities and noises in the initial cost volume. In order to deal with disparity discontinuities and ambiguous regions (textureless areas, repetitive patterns, etc), an ideal cost aggregation strategy should modify its support at each position according to image content so that only those points with the same (unknown) disparity are included. Different from the traditional weight-based aggregation [10], which tries to change the color weight and distance weight to complete weight calculation, the proposed method uses a local affine model to aggregate the cost, which has been proven useful in image denoising [15] and image matting [16][17]. Assume there are two small local matching regions (Ω_P and Ω_q) centered on pixel p and pixel q in the two rectified images L and R (obtained from input pair I_L and I_R), respectively, an affine relation between the intensities of the local matching region is described as:

$$R_{k^{\prime }}=M_p{L}_k+B_p,k\in {\mathrm{\Omega }}_P,k^{\prime }\in {\mathrm{\Omega }}_q$$ (5)

where M_p and B_p are linear coefficients which are constants in Ω_p. Assuming that the left image is the reference image, we can get the disparity information from D_k = R_k′−L_k Combining with (5), D_k is given by

$$D_k=R_{k^{\prime }}-L_k=M_p{L}_k+B_p-L_k=A_p{L}_k+B_p$$ (6)

with A_p = M_p − 1 Eqation (6) gives the relationship between the disparity image D and input image L in local region Ω_p. Additionally, the local affine model satisfies D_k = A_p∇L_k, which ensures that the image D has an edge if image L also has an edge. In this way, disparity discontinuities are also parts of edges in the reference image.

In order to obtain A_P and B_P, we minimize the difference between input and output disparities p_k and disparity D_k respectively, using a least-square optimal estimator in a local window:

$$E(A_P,B_P)=\mathit{min}\left({\sum }_{k\in {\mathrm{\Omega }}_p}({(A_{\text{P}}{L}_k+B_{\text{P}}-p_k)}^2+\epsilon A_{\text{P}}^2)\right)$$ (7)

where p_k is a pixel in the window Ω_p in an initial input disparity map, denoted by D̂. ε is a factor which prevents A_p from being too large. A guided filter (GF) has been utilized to solve (7) using a least-squares estimator [15]. The optimal A_p and B_p are given by:

$$A_p=\frac{\frac{1}{\mid {\mathrm{\Omega }}_p\mid }{\sum }_{k\in {\mathrm{\Omega }}_p}{L}_k{p}_k-{\mu }_p\overline{p_p}}{{\sigma }_k^2+\epsilon },B_p=\overline{p_p}-A_p{\mu }_p$$ (8)

where μ_p and ${\sigma }_k^2$ are the mean and variance, respectively, of input image L in window Ω_p. |Ω_p| is the number of pixels in Ω_p, and $\overline{p_p}$ is the mean of p in Ω_p. Since p is included in many windows centered by its neighborhood, correspondingly, there are many A_p and B_p in different windows including p. Hence, the GF simply averages all possible values of D_k by

$$\begin{array}{l}{D}_k={\overline{A}}_k{L}_k+{\overline{B}}_k,\\ {\overline{A}}_k=\frac{1}{\mid {\mathrm{\Omega }}_k\mid }{\sum }_{p\in {\mathrm{\Omega }}_k}{A}_p,{\overline{B}}_k=\frac{1}{\mid {\mathrm{\Omega }}_k\mid }{\sum }_{p\in {\mathrm{\Omega }}_k}{B}_p.\end{array}$$ (9)

Additionally, from the (9), we found that all the summations can be calculated by box filtering. Therefore, these terms can be computed within the time proportional to N, i.e., O (N) time.

In the proposed algorithm, we expand (9) to disparity space image. In addition, we use a color image as the reference image in order to obtain a more reliable data term. Then, (9) is rewritten as:

$$D_k(x,y)={\sum }_{i\in {\mathrm{\Omega }}_k}({\mathit{A}}_{\mathit{k}}^T{\mathit{L}}_{\mathit{i}}+B_k)$$ (10)

where the bold faced L_i and A_k are both 3 × 1 vectors, and ${\mathit{A}}_{\mathit{k}}^T{\mathit{L}}_{\mathit{i}}=A_k^r{L}_i^r+A_k^g{L}_i^g+A_k^b{L}_i^b$, with r, g and b denoting color components.

3. Proposed Algorithm

In order to ease description, we partition our algorithm into three modules: initialization, global optimization and enhancement.

In the initialization module, the initial matching cost is first computed. Then, the reference image and local affine model are used to obtain aggregation matching costs. This process will be applied twice, because both the left and right input images are needed to be reference images. The initial data term Φ(d_p) is computed from the aggregation matching cost. Hence, this step generates the following outputs: the initial left disparity map D_L, initial right disparity map D_R, initial disparity image space $C_D^0$, aggregated disparity image space $C_D^1$, and data term Φ(d_p). Another important output of the initialization module is the labels of the edge classification from which ρ_edge is incorporated as the constraint in the basic Bayesian model. In the global optimization module, the method of accelerated belief propagation is employed for iterative optimization that trades off between the data term and the prior term. The prior is decreased at discontinuities using the edge constraint, since the discontinuities are likely to coincide with edges in the reference image. Finally, the goal of the enhancement module is to improve the disparity result in a subpixel level.

3.1 Initialization

Step1: Matching cost computation

Typically, a matching cost is computed at each pixel for all disparities under consideration, which constructs the disparity space image. Recently, Klaus proposed a simple self-adapting dissimilarity measurement which linearly combines SAD and a gradient for computing the matching cost [11]. Considering both the good performance and simplicity of Klaus’ method, in this paper, this measurement is used for constructing the disparity space image.

Assume that the left image is a reference image, C_CSAD is the absolute color difference and C_CGRAD is the color gradient difference. The matching cost C(x, y, d)can be defined by

$$C(x,y,d)=(1-\alpha )C_{\mathit{CSAD}}(x,y,d)+\alpha C_{\mathit{CGRAD}}(x,y,d)$$ (11)

where C_CSAD and C_CGRAD are given by:

$$\begin{array}{c}{C}_{\mathit{CSAD}}(x,y,d)={\sum }_{(\text{i},\text{j})\in \mathrm{\Omega }(\text{x},\text{y})}\mathit{min}(({\sum }_{c\in \{\text{r},\text{g},\text{b}\}}\mid I_l^c(i,j)-I_r^c(i+d,j)\mid )/3,{\tau }_c),\\ C_{\mathit{CGRAD}}(x,y,d)={\sum }_{(i,j)\in \mathrm{\Omega }(x,y)}\mathit{min}(\mid {\nabla }_x{I}_l(i,j)-{\nabla }_x{I}_r(i+d,j)\mid +\mid {\nabla }_y{I}_l(i,j)-{\nabla }_y{I}_r(i+d,j)\mid ,{\tau }_g)\end{array}$$ (12)

where C(x, y, d) is the matching cost of pixel (x, y) in the disparity d, α balances the color and gradient terms, ∇_x and ∇_y represent the horizontal and vertical gradients, respectively, and τ_c and τ_g are truncation values. In order to suppress noise, a small supporting window Ω(x, y) centered at (x, y) is used to calculate the matching cost. After this step, the initial disparity space image ${\text{C}}_{\text{D}}^0$ is constructed by C(x,y,d) for all the pixels in the reference image.

Step2: Aggregation

To obtain more accurate results in both smooth and discontinuous regions, the local affine model is used to initialize a reliable matching cost. The local affine model can be expanded to the RGB space as well by rewriting (10) as:

$$C_{\mathit{new}}(x,y,d)={\overline{\mathit{A}}}^T(x,y)\mathit{I}(x,y)+\overline{B}(x,y)$$ (13)

Ā(x, y) and B̄(x, y) are given by:

$$\begin{array}{l}\overline{\mathit{A}}(x,y)=\frac{1}{\mid \mathrm{\Omega }\mid }{\sum }_{(i,j)\in \mathrm{\Omega }(x,y)}\mathit{A}(i,j),\mathit{A}(x,y)=\frac{\frac{1}{\mid \mathrm{\Omega }\mid }{\sum }_{(i,j)\in \mathrm{\Omega }(x,y)}\mathit{I}(i,j)C(i,j,d)-\mu (x,y)\overline{C}(x,y,d)}{\mathrm{\Lambda }+\epsilon U},\\ \overline{B}(x,y)=\frac{1}{\mid \mathrm{\Omega }\mid }{\sum }_{(i,j)\in \mathrm{\Omega }(x,y)}B(i,j),B(x,y)=\overline{C}(x,y,d)-{\mathit{A}}^T(x,y)\mu (x,y).\end{array}$$ (14)

In (13) and (14), I is a 3 × 1 vector, μ is a mean and Λ is a 3 × 3 covariance matrix, respectively, of I in local window Ω. A and Ā are 3 × 1 coefficient vectors, U is a 3 × 3 identity matrix. C_new(x, y, d) constructs the new disparity space image $C_D^1$, which will replace the initial disparity space image $C_D^0$.

In a similar manner, we compute the disparity space image for the right reference image. When two disparity space images are obtained, the Winner-Take-All (WTA) [1] strategy is applied to form two initial disparity maps D_L and D_R.

In a reliable disparity map, any stable pixel is expected to satisfy the mutual consistency condition, which requires the pixels on the pixel grids in the left and right disparity maps to be perfectly consistent with each other (i.e., having the same disparity value). This is performed by a subsequent mutual consistency check (often called left-right checking [1]) that divides all the pixels into stable or unstable pixels:

$$D_L(x_l)-D_R(x_l-D_L(x_l))\le T_{lf}$$ (15)

where T_lf is a threshold. We mark pixels in the left disparity map as occluded pixels if (15) does not hold true.

After the left-right checking step, the data term can be defined by the occluded pixels and stable pixels as:

$$\mathrm{\Phi }(x,y,\text{d})=\{\begin{array}{c}{C}_{\mathit{new}}(x,y,d),(x,y)\in \mathit{stable}\\ 0,(x,y)\in \mathit{occluded}\end{array}.$$ (16)

The constant 0 in (16) reflects the fact that the occluded pixels need the most regularization.

Step3: Edge classification

This step is based on the assumption that an intensity edge in an image indicates a depth discontinuity in the scene. In our case, the edges in the reference image are detected by the Sobel edge detector [18]. If D̄_k is the mean disparity value of the window Ω_k centered at pixel k, and i is a pixel in the neighborhood of pixel k with disparity D_i, then, the standard deviation (Std) is computed to estimate the edge variation in the disparity map D_L:

$$\mathit{Std}(k)=\sqrt{{\sum }_{i\in {\mathrm{\Omega }}_k}{(D_i-{\overline{D}}_k)}^2},i\in \text{Sobel}\text{edge}.$$ (17)

Hence, if Std(k) is smaller than the threshold λ, the edge belongs to a normal edge. Otherwise, the edge is a discontinuity. The edge classification is thus given by

$$\mathit{edge}(k)=\{\begin{array}{cc}0,& \mathit{Std}(k)<\lambda \\ 1,& \mathit{Std}(k)>\lambda \end{array}.$$ (18)

Fig. 1 provides an example of edge classification using (17) and (18). It can be observed that most disparity discontinuities are successfully identified.

Figure 1.

(a) Left image in a pair of stereo images; (b) Edges detected by Sobel detector; (c) Edges corresponding to disparity discontinuities detected by Eq.(17) and Eq.(18).

3.2 Global Optimization

The data term Φ(d_p) is determined by updating the matching cost in eq.(16), and the prior term ψ(d_p, d_q) in (3) is defined by the Potts model. Assuming that two neighboring pixels p and q are likely to have the same disparity if their intensities satisfies I(p) ≈ I(q). This contextual information is incorporated into ψ(d_p, d_q), which is computed in the same fashion as that in [7]:

$$\psi (d_p,d_q)=\{\begin{array}{rr}\hfill 0,& \hfill {\text{d}}_{\text{p}}={\text{d}}_{\text{q}}\\ \hfill \beta ,& \hfill \text{otherwise}\end{array},\beta (\mathrm{\Delta }\text{I})=\{\begin{array}{cc}\text{s},& \text{otherwise}\\ \text{P}\times \text{s},& \text{if}\mathrm{\Delta }\text{I}<T\end{array}.$$ (19)

where T is a gradient threshold, and ΔI is the difference between I_p and I_q, s is a penalty term for violating the smoothness constraint and P is a penalty term when the gradient has a small magnitude. Note that T, P and s are constants over the whole image. Assuming that there is a matching cost, denoted by C, in order to use belief propagation, C should be converted to a compatibility form by calculating e^−C. For numerical reasons (e.g. e^−C is extremely small), we include a positive constant M and define the compatibility as e^−C/M.

Belief Propagation (BP) is an iterative inference algorithm that propagates messages in a network. Among several high performance algorithms, we adopted the max-product BP algorithm [4][7] which works by passing messages within a graph defined by an image with four-connected pixel grid. Each message is a vector of the dimension given by the number of possible labels. During a iteration, each node uses the messages received in the previous iteration from neighboring nodes. A new message is then calculated and sent to its neighbors.

A message update scheme determines when a message is sent to a node where it will be used to compute subsequent messages for neighbors of the node. One of the update schemes is to propagate messages in one direction and update each node immediately. In this work, we use the “accelerated” BP updating scheme (seen in Fig. 2) proposed by Tappen et al [7]. The “up-down-left-right” message passing scheme enables the BP algorithm to converge rapidly.

Figure 2.

Accelerated BP message updating scheme: (a) from right to left, (b) from left to right, (c) from down to up, (d) from up to down.

Let us consider the case where node p is located to the right side of node q (Fig. 2(a)). Let $m_{\mathit{right}}^t(d_q)$ be the message that node p sends to q in the current iteration t, which contains node p’s belief about each possible state of node q. This message is computed from the messages that p has received from its neighbors (up, down, right) at iteration t − 1. Then, the new message (for the right side case) is updated as:

$$m_{\mathit{right}}^t(d_q)\leftarrow {\mathit{max}}_{d_p}\psi (d_p,d_q)\times \mathrm{\Phi }(d_p)\times m_{\mathit{right}}^{t-1}(d_p)m_{up}^{t-1}(d_p)m_{\mathit{down}}^{t-1}(d_p)$$ (20)

where $m_{\mathit{right}}^{t-1}(d_p),m_{up}^{t-1}(d_p)$ and $m_{\mathit{down}}^{t-1}(d_p)$ are the messages received by p from the nodes’ right, above and below at iteration t−1. The new updated left, above and below messages (Fig. 2(b)–(d)) are computed similarly to Eq.(20). After iteration T, the beliefs at node q can be computed by

$$b(d_q)=\mathrm{\Phi }(q,d)m_{\mathit{left}}^T(d_p)m_{\mathit{right}}^T(d_p)m_{up}^T(d_p)m_{\mathit{down}}^T(d_p).$$ (21)

Then, the best disparity d_q^MAP in pixel q is obtained from the maximum belief:

$${d_q}^{\mathit{MAP}}={\mathit{argmax}}_{d_q}\left(b(d_q)\right).$$ (22)

3.3 Enhancement

To reduce the discontinuities error caused by the quantization effect in the disparity, a method producing subpixel precision is proposed. Assuming that d_c is the disparity after accelerated BP. The final disparities d_f can be achieved by:

$$d_f=d_c+0.5\ast \frac{C(x,y,d_c-1)-C(x,y,d_c+1)}{C(x,y,d_c-1)+C(x,y,d_c+1)-2C(x,y,d_c)}.$$ (23)

Since a disparity map has been produced from the belief propagation module and the matching cost of the three candidates C(x, y, d_c − 1), C(x, y, d_c + 1) and C(x, y, d_c) are obtained from the aggregated disparity space image in the aggregation step, the enhanced disparity map can be easily computed.

Our algorithm was summarized in Fig. 3.

Figure 3.

Framework of the proposed method

4. Experimental Results

In this section, we evaluate the performance of our stereo matching algorithm by the Middlebury stereo benchmark dataset [14]. This dataset consists of four pairs of images: Tsukuba, Venus, Teddy and Cones. Our experimental study includes the following components: parameters setting, evaluation of results, and comparison with several state-of-the-art stereo matching methods [10][11][13][19][20].

4.1 Parameter Setting

We first describe the settings of parameter used in our algorithm. Note that the parameters are constant across all datasets. In order to improve stability, we normalize the pixel intensity to [0,1].

Cost computation

The computation of matching cost is parallel at each pixel and each disparity level. To obtain appropriate thresholds, we analyze the parameters statistically by experiments shown in Fig. 4. Then, the truncation values τ_c and τ_g are set, respectively, to 0.0028 and 0.016 in (12). The balance term α is set to 0.92. Ω is chosen to be a 3 × 3 local window.

Figure 4.

Intermediate results of four different standard test images by our algorithm compared to the ground truth. (a) Left images of the stereo image pairs;(b)edge detection result; (c) WTA disparity maps after local affine model; (d) disparity maps after applying left-right checking ;(e) disparity maps after BP propagation;(f) disparity maps after refinement;(g) ground truth.

Cost aggregation

We use the Eq. (13) for the aggregation. The parameters are chosen empirically as follows: Ω_k is a 15 × 15 window and ∈ is 0.0001.

Occlusion detection and edge constraint

Since there is a certain error in computation, we choose T_lf to be 1 in Eq. (15). After experiments, λ in Eq.(18) is set to 0.18 in order to distinguish the disparity discontinuity from normal edges.

Belief Propagation

Parameters s, T and P in Eq. (19) of the BP module are set to 0.000153, 0.06 and 8, respectively. The constant M is set to 0.1961 in the global optimization module of section II. The iteration time is set to 5 by experiments.

Fig. 4 shows the disparity maps obtained by our algorithm using the above parameters. The results after different intermediate stages provided visual explanations that how the different stages in the algorithm improve the results. The ground truths are also given for comparison.

4.2 Evaluation

The performance of stereo matching algorithms can be evaluated quantitatively. Two methods are often utilized based on the RMS (root-mean-squared) error and the percentage of bad matching pixels with respect to the ground truth data included in the Middlebury datasets [1]. In this work, the second method is utilized. First, we compute B which reflects the percentage of bad matching pixels:

$$B=\frac{1}{N}{\sum }_{(x,y)}(\mid d_c(x,y)-d_T(x,y)\mid >{\delta }_d)$$ (24)

where δ_d is the disparity error tolerance (Table 1 list results for δ_d =0.5, 0.75 and 1). The result of each image pair in the Middlebury set is computed by measuring B in pixels of three regions: nonoccluded pixels (denoted by nocc), discontinuity pixels (denoted by disc) and either nonoccluded or half-occluded (denoted by all). The value of B in the whole disparity image is considered (the second column marked by Ave. Error in Tables 1 and 2). It can be seen that the result of image pair Venus is most accurate when δ_d =1 (Table 1 in the rows with bold faces). This is because our algorithm performs well when the scene is mainly composed of planar objects. Note that, in both tables, the subscript of each number denotes the rank in the Middlebury stereo benchmark which contains approximately 140 algorithms.

Table 1.

Result of the proposed method with different error thresholds

Error Threshold	Avg. Error	Tsukuba			Venus			Teddy			Cones
		Nocc	All	Disc	Nocc	All	Disc	Nocc	All	Disc	Nocc	All	Disc
δd =1	5.88	1.2937	2.0852	5.7118	0.093	0.3819	1.314	7.1461	13.264	15.735	3.4946	9.6259	10.364
δd =0.75	8.26	7.1523	8.2722	12.67	0.4317	0.8322	3.6216	7.9138	14.753	17.625	4.0534	10.342	11.650
δd =0.5	10.5	7.159	8.2711	12.65	3.7628	4.3427	9.8319	10.313	17.524	22.715	4.646	11.815	13.114

Table 2.

Results of Comparison (δ_d=0.5)

Algorithm	Avg. Error	Tsukuba			Venus			Teddy			Cones
		Nocc	All	Disc	Nocc	All	Disc	Nocc	All	Disc	Nocc	All	Disc
Our method(final)	10.5	7.159	8.2711	12.65	3.7628	4.3427	9.8319	10.313	17.524	22.715	4.646	11.815	13.114
Our method(cons)	10.7	7.3610	8.4811	13.87	3.6927	4.2727	9.5119	10.413	17.422	23.016	4.798	12.015	13.316
Our method(no)	10.8	7.5310	8.7313	14.18	3.7228	4.3027	9.8819	10.313	17.725	23.016	4.9310	12.419	13.718
SubPixDoubleBP[19]	10.7	8.7818	9.4515	14.910	0.723	1.125	5.243	10.112	16.415	21.37	8.4945	14.743	16.541
AdaptOvrSegBP[13]	11.9	5.983	6.563	9.091	3.6625	3.9622	13.252	13.043	18.937	26.435	9.4853	14.945	17.248
OvrSegBP[20]	12.4	7.7514	8.179	13.87	4.3333	4.7330	16.884	13.244	19.341	27.540	6.5324	12.622	14.021
AdaptingBP[11]	13.6	19.169	19.365	17.427	4.8439	5.0835	7.8411	12.839	16.717	26.334	7.0230	13.228	14.020
DoubleBP[10]	15.7	18.764	19.162	15.818	7.8277	8.2273	11.332	14.454	19.947	24.424	11.877	17.668	19.768

4.3 Comparation

Table 2 summarizes the quantitative performance of our method and those of other global BP stereo methods [10][11][13][19][20]. The results are ranked roughly in descending order of overall performance. The evaluation of our final results after disparity enhancement is presented in the first row. Additionally, the evaluation results with and without edge constrain incorporated into stereo matching are shown in the second and third rows (in bold face), respectively. After incorporating the edge constraints, the matching result is improved, which is indicated by smaller average error in Table 2. Note that our algorithm produces the best performance in subpixel level (e.g. δ_d =0.5), and the rank is up to 5 in approximately 140 algorithms.

The Tsukuba image pairs contain some dark and noisy regions near the lamp and the desk, which usually lead to incorrect support regions for aggregation. Our method performs very well in Tsukuba, including high texture parts (such as the lamp stem in Figure 4.(f)) and textureless parts (such as the box, desk). That is because our method preserves edges while smoothing the relatively flat regions.

Although a reliable local affine model prevents edge smoothing obtaining an accurate affine model is very difficult. Therefore, in the proposed algorithm, we just average possible local affine model parameters, which lead to the inequality ∇D_k < Ā_k ∇ L_k in high texture regions. In this case, certain details are lost, which can be seen in Tsukuba of Figure 4(c) where the lamp holder appears to be broken.

Additionally, our algorithm performs well when the scene is mainly composed of planar objects. However, if the scene is mainly composed of smooth, curved surfaces, the performance of the proposed algorithm may decrease. This is because the smoothness prior used in the proposed method is the first-order prior [21][22], which indeed favors low-curvature fronto-parallel surfaces. Even in man-made scenes (seen in the “stair-case” in Venes of Fig. 4(g)), the prior is maximized by fronto-parallel planes, leading to inaccurate depth estimates.

In the left-right checking stage, all unstable disparities are replaced by 0. Since the noise causes inconsistency in the left and right disparity maps, some discarded unstable pixels may have a correct disparity. Therefore, it can be explained why the error increases after left-right checking (seen in Fig. 4(c)–(d)).

5. Conclusion

In this paper, stereo matching is formulated as a Bayesian inference problem. The edge information is integrated into the basic MAP-MRF stereo model as a soft constraint. Then, the accelerated belief propagation algorithm is adopted to this MAP-MRF problem effectively. In order to obtain more reliable cost volume, a local affine model is used to aggregate the stereo matching process. Our experimental results have demonstrated high performance of the proposed algorithm. Additionally, our algorithm performs well when the scene is mainly composed of planar objects. However, the performance of the propose algorithm may decrease if the scene is mainly composed of smooth and curved surfaces. Hence, combining more visibility reasoning and second-order smoothness in an optimization framework will be considered in the future work.