Algorithm 1 Overview of depth alignment procedure.

Input Captured depth image $D_c$, estimated depth image $D_e$;

Output: aligned depth map $D_{c1}$ for $D_c$;

1: Run RANSAC to find the metric scaling factor.

2: Extract patch $P{a}_c$ in $D_c$.

3: Calculate scaled depth values and normals of $P{a}_c$.

4: Initialize offset map O.

5: Find a patch $Q{b}_e$ in $D_e$ based on O.

6: for $a_i$ ∈ $P{a}_c$ and $b_i$ ∈ $Q{b}_e$ do

7:    Calculate depth difference $d{d}_i$ and normal angle $an{g}_i$ between $a_i$ and $b_i$.

8:    Run PatchMatch propagation to update offset map O.

9: Mapping $D_c$ to $D_{c1}$ based on O.