A robust real-time surface reconstruction method on point clouds captured from a 3D surface photogrammetry system

Abstract

Purpose:

To develop a robust and real-time surface reconstruction method on point clouds captured from a 3D surface photogrammetry system.

Methods:

The authors have developed a robust and fast surface reconstruction method on point clouds acquired by the photogrammetry system, without explicitly solving the partial differential equation required by a typical variational approach. Taking advantage of the overcomplete nature of the acquired point clouds, their method solves and propagates a sparse linear relationship from the point cloud manifold to the surface manifold, assuming both manifolds share similar local geometry. With relatively consistent point cloud acquisitions, the authors propose a sparse regression (SR) model to directly approximate the target point cloud as a sparse linear combination from the training set, assuming that the point correspondences built by the iterative closest point (ICP) is reasonably accurate and have residual errors following a Gaussian distribution. To accommodate changing noise levels and/or presence of inconsistent occlusions during the acquisition, the authors further propose a modified sparse regression (MSR) model to model the potentially large and sparse error built by ICP with a Laplacian prior. The authors evaluated the proposed method on both clinical point clouds acquired under consistent acquisition conditions and on point clouds with inconsistent occlusions. The authors quantitatively evaluated the reconstruction performance with respect to root-mean-squared-error, by comparing its reconstruction results against that from the variational method.

Results:

On clinical point clouds, both the SR and MSR models have achieved sub-millimeter reconstruction accuracy and reduced the reconstruction time by two orders of magnitude to a subsecond reconstruction time. On point clouds with inconsistent occlusions, the MSR model has demonstrated its advantage in achieving consistent and robust performance despite the introduced occlusions.

Conclusions:

The authors have developed a fast and robust surface reconstruction method on point clouds captured from a 3D surface photogrammetry system, with demonstrated sub-millimeter reconstruction accuracy and subsecond reconstruction time. It is suitable for real-time motion tracking in radiotherapy, with clear surface structures for better quantifications.

1. INTRODUCTION

Motion monitoring and tracking play an important role in radiotherapy. Surface photogrammetry-based systems are gaining increasing popularity on patient positioning and motion tracking¹ in radiotherapy. Such systems, e.g., VisionRT (VisionRT Ltd., London, UK) and C-RAD (C-RAD AB, Uppsala, Sweden), captures dynamic point clouds of patient surface in real-time and provides an intensive yet a noninvasive way for monitoring patient motion. VisionRT, for example, projects a speckle pattern of visible (red) light on the thoracoabdominal surface of a supine patient, and such pattern is captured at about 10–25 Hz via optical cameras. A common occurrence in surface photogrammetry systems is noisy or missing measurements, due to, e.g., chest hair or the presence of immobilizing devices and fixtures which may partially occlude the patient surface. Blindly registering such intrinsically discordant point clouds is subject to large errors in the order of centimeters.² Our group has recently developed a variational method to reconstruct a continuous surface from the point cloud by optimizing a regularized functional.³ The reconstructed continuous surface is robust to noise and missing measurements and permits better quantification of the surface structure and subsequent motion tracking. However, the variational method requires solving a partial differential equation (PDE), which is usually time consuming, as the time step is restricted by the Courant–Friedrichs–Lewy (CFL) condition. Studies have been conducted to speed up the PDE evolution by restricting computations to be within a small narrowband of the level-set function,⁴ using multiresolution,⁵ and utilizing convex optimization schemes.⁶ Despite these efforts, current approaches are still far from meeting the real-time reconstruction requirement, as desired in adaptive radiotherapy for managing respiratory motion. To bridge the gap, this study builds upon recent developments on sparse shape composition and representation^7–9 and proposes a fast yet robust method to reconstruct patient surface without explicitly solving the PDE. The overarching premise is that normal variations of patient motion (mostly respiratory) are largely repetitive with limited physiological variations, and any point cloud or its underlying surface can be embedded within a local submanifold. Given a sufficient set of training point clouds with their corresponding surfaces, we first approximate the target point cloud as a sparse linear combination from the training point clouds. The target level-set surface is subsequently reconstructed from the corresponding training surfaces using the same linear coefficients, assuming that alternating from point clouds to their corresponding surfaces does not change the topological relationship in their low-dimensional embeddings. To the best of our knowledge, this is the first time a real-time continuous surface reconstruction method has been developed in a photogrammetry system. The proposed method provides a new pathway to achieve fast and reliable patient position monitoring and serves as a robust prestep to facilitate real-time motion management during the treatment.

2. METHOD

2.A. Variational surface reconstruction method

The level-set method¹⁰ is especially suitable for point cloud representation, as it offers a continuous representation of underlying curves/surfaces and avoids constructing explicit point correspondences for capturing and tracking deformations. Specifically, given a surface C, it is implicitly represented as the zero level-set of its corresponding level-set function in a space that is one dimension higher. Given a point cloud p, the level-set surface is constructed by optimizing the following regularized functional:^5,11

$${\displaystyle E\left(\varphi \left(\mathbf{x}\right)\right)={\int }_{\mathrm{\Omega }}d(\mathbf{x},p)\delta \left(\varphi \left(\mathbf{x}\right)\right)\left|\nabla \varphi \left(\mathbf{x}\right)\right|\mathrm{d}\mathbf{x},}$$ (1)

where d(x, p) defines a distance function to the point cloud p, ϕ(x) represents the level-set function, and δ is the Dirac delta function. Equation (1) is usually solved with gradient descent by evolving the following PDE:

$${\displaystyle \frac{\partial \varphi }{\partial t}=\left|\nabla \varphi \right|\left(\nabla d\cdot \frac{\nabla \varphi }{\left|\nabla \varphi \right|}+d\nabla \cdot \frac{\nabla \varphi }{\left|\nabla \varphi \right|}\right).}$$ (2)

Solving this PDE is time consuming, since its time step is restricted by the CFL condition. An alternative fast and robust surface reconstruction method is needed to meet the real-time requirement in radiotherapy, where a subsecond reconstruction time is desirable for point clouds acquired under normal breathing (3–5 s/cycle) to meet the Nyquist sampling criterion.

2.B. Proposed fast surface reconstruction method

In this study, we propose a fast yet robust surface reconstruction method, taking advantage of the repetitive and overcomplete nature of the acquired point cloud set. Given sufficient number of training point clouds and their corresponding level-set surfaces, we approximate the target point cloud as a sparse linear combinations from the training set. This can be perceived as a local linear descriptor for the open subset of Euclidean space, as a small open set about the given point cloud is transformed by its coordinate chart. The target level-set surface is subsequently reconstructed by using the same linear coefficients, assuming that the target surface shares the same topological relationship as that from the target point cloud. The validity of this assumption is based on accurate reconstruction of the underlying continuous patient surfaces from the training point clouds using the variational method, which has been verified quantitatively in our previous study.³ The basic idea of the proposed method is illustrated in Fig. 1. Specifically, our proposed method consists the following four consecutive steps.

FIG. 1.

The basic idea of the proposed method: the target point cloud is first approximated as a sparse linear combination of the training point clouds with coefficient $\hat{\mathbf{w}}$; the target level-set surface is subsequently reconstructed from the corresponding training surfaces using the same coefficients, assuming that any given point cloud and its level-set surface shares similar local topological geometry.

2.B.1. Reconstruct level-set surfaces on training point clouds (offline)

We first reconstruct training level-set surfaces from the training point clouds, following steps from previously proposed variational reconstruction method described in Sec. 2.A.³ Specifically, given k training point clouds {p₁, p₂, …, p_k}, we reconstruct their corresponding surfaces as {ϕ₁, ϕ₂, ..., ϕ_k}, with ϕ_i ∈ ℝ^N being the level-set function representing each reconstructed surface. This step requires the most computational efforts and is performed offline.

2.B.2. Build training libraries on aligned point clouds and surfaces with iterative closest point (ICP) (offline)

The ICP algorithm is used to align the training point clouds.¹² Specifically, we arbitrarily choose a point cloud p₁ ∈ ℝⁿ in the training set as the reference and transform the rest of the point clouds into the same coordinate system with correspondences using ICP: {p₂∘T₂, ..., p_k∘T_k}, where T_i represents the rigid transformation of each training point cloud to the reference and ∘ represents the composition operator. Our training libraries for both the point clouds and their corresponding surfaces are built as follows: 𝒫 = [p₁, p₂∘T₂, ..., p_k∘T_k] ∈ ℝ^n×k and Φ = [ϕ₁, ϕ₂∘T₂, ..., ϕ_k∘T_k] ∈ ℝ^N×k.

2.B.3. Represent target point cloud on $\mathcal{P}$ through sparse linear regression (online)

Given the repetitive and overcomplete nature of the respiratory signals, it is reasonable to expect high-dimensional point clouds and the corresponding level-set surfaces to lie in a low-dimensional manifold. With sufficient training data, any target point cloud/surface can be approximated by a sparse linear combination of examples in the training set. Given any target point cloud p_t ∈ ℝⁿ, we propose to solve its sparse representation with

$${\displaystyle \underset{\mathbf{w}}{\text{min}}\Vert {\tilde{p}}_t-\mathcal{P}\mathbf{w}{\Vert }_2^2+\lambda \Vert \mathbf{w}{\Vert }_1,}$$ (3)

where ${\tilde{p}}_t=p_t\circ T_t$ is prealigned to the reference coordinates, and λ is the regularization coefficient that controls the sparseness of w. Note that Eq. (3) explicitly assumes that ICP finds relatively accurate point-to-point correspondences among different point clouds in $\mathcal{P}$, and the error of the built correspondences follows a Gaussian distribution. This is generally true if we have reasonably consistent point cloud acquisitions over time.

However, problems may arise when inconsistent noise levels or occlusions are present, which results in large errors caused by the many-to-one correspondences built by ICP. Inspired by Refs. 7 and 13, we propose a modified model by directly considering such large error,

$${\displaystyle {\tilde{p}}_t\doteq \mathcal{P}\mathbf{w}+\mathbf{e},}$$ (4)

where e ∈ ℝⁿ represents the error vector, whose nonzero entries indicate errors in ${\tilde{p}}_t$ that are either caused by inconsistent noise levels or occlusions. Assuming that the noise/occlusion in e is sparse, our modified sparse coefficient can be solved by

$${\displaystyle \underset{\mathbf{w},\mathbf{e}}{\text{min}}\Vert {\tilde{p}}_t-\mathcal{P}\mathbf{w}-\mathbf{e}{\Vert }_2^2+\lambda \Vert \mathbf{w}{\Vert }_1+\gamma \Vert \mathbf{e}{\Vert }_1,}$$ (5)

where γ is the regularization coefficient which controls the sparseness of the inconsistent noise level and missing measurements, with ${\hat{p}}_t=\mathcal{P}\hat{\mathbf{w}}$ approximates the clean point cloud. Both Eqs. (3) and (5) are convex and can be efficiently solved by various iterative algorithms.^14,15

2.B.4. Propagate the sparse coefficients from point clouds to level-set surfaces (online)

Assuming that training point clouds and their level-set surfaces share the same topological geometry (reflected in the local linear coefficients), we propose to reconstruct the target level-set surface as ${\hat{\varphi }}_t=\mathrm{\Phi }\hat{\mathbf{w}}\in {\mathbb{R}}^N$. Finally, the target surface is transformed back to its original coordinates ${\hat{\varphi }}_t\circ T_t^{-1}$. Algorithm I illustrates details of the proposed method.

ALGORITHM I.

Fast surface reconstruction.

Input: target point cloud pt and training point clouds {p1, p2, ..., pk}

Step 1: Reconstruct surfaces from training point clouds using the variational method (offline)

• {ϕ1, ..., ϕk}←PDE-based variational surface reconstruction ({p1, p2, ..., pk})

Step 2: Build training libraries on aligned point clouds and surfaces using ICP (offline)

• solve rigid transformation Ti, with p1 as reference: Ti←ICP(p1, pi)

• construct point cloud library: 𝒫 = [p1, p2∘T2, ..., pk∘Tk] ∈ ℝn×k

• construct surface library: Φ = [ϕ1, ϕ2∘T2, ..., ϕk∘Tk] ∈ ℝN×k

Step 3: Represent target point cloud on $\mathcal{P}$ through sparse linear regression (online):

• align target point cloud to the reference: ${\tilde{p}}_t\leftarrow p_t\circ T_t$

• represent ${\hat{p}}_t$ on $\mathcal{P}$ through sparse regression (SR): $\hat{\mathbf{w}}\leftarrow \underset{\mathbf{w}}{\text{min}}\Vert {\tilde{p}}_t-\mathcal{P}\mathbf{w}{\Vert }_2^2+\lambda \Vert \mathbf{w}{\Vert }_1$,

or through modified sparse regression (MSR): $\hat{\mathbf{w}},\hat{\mathbf{e}}\leftarrow \underset{\mathbf{w},\mathbf{e}}{\text{min}}\Vert {\tilde{p}}_t-\mathcal{P}\mathbf{w}-\mathbf{e}{\Vert }_2^2+\lambda \Vert \mathbf{w}{\Vert }_1+\gamma \Vert \mathbf{e}{\Vert }_1$

Step 4: Propagate $\hat{\mathbf{w}}$ from point clouds to level-set surfaces (online)

• reconstruct target surface as: ${\hat{\varphi }}_t=\mathrm{\Phi }\hat{\mathbf{w}}$

Output: final reconstructed surface ${\hat{\varphi }}_t\leftarrow {\hat{\varphi }}_t\circ T_t^{-1}$

3. EXPERIMENTS AND RESULTS

We tested the proposed method on 200 clinical point clouds, which were captured continuously from the thoracoabdominal surface of a supine subject by the VisionRT system at 15 Hz and spanned about 14 s. The temporal plot of discrepancies between each of the 200 continuous surfaces against the first surface is shown in Fig. 2, where clear respiratory cycles and motion variations can be observed. We chose the first 100 point clouds as the training set and evaluated the performance of our method on the rest of the 100 testing point clouds. The reconstruction accuracy was evaluated based on root-mean-squared-error (RMSE) by comparing our reconstructed surfaces against surfaces reconstructed by the variational method,³ which has validated accuracy and could be considered as benchmark in this scenario. We further evaluated both the proposed reconstruction models by including inconsistent occlusions to the testing point clouds. For all experiments, we evaluated the performance of both the sparse regression (SR) model and the modified sparse regression (MSR) model.

FIG. 2.

Temporal plot of discrepancies between each of the 200 continuous surfaces against the first surface based on RMSE.

3.A. Parameter settings

The PDE-based variational surface reconstruction was implemented on a rectangular grid with size 200 × 150 × 75, with time step Δ_t = 0.01. The proposed method was tested with matlab 2013b on Mac OS 10.9.4, running on a macbook laptop with quad-core I7 2.3 GHz, 8GB RAM. The original point cloud has a size of 5107 and was downsampled to 335 by keeping every 15th sample for speed-up. We solved both the proposed regression models using cvx, a package for solving convex problems.^14,15 We empirically chose regularization coefficients in our regression models to be λ = 1, γ = 0.05.

3.B. Surface reconstruction results and evaluations

Figure 3(a) shows one typical point cloud acquired by the VisionRT system. Figures 3(b)–3(d) compare the reconstruction results from the SR and MSR models with the result from the variational method. For the convenience of performance evaluation, we represent reconstructed surfaces as height functions in a Cartesian coordinates: z = f(x, y). Figures 4(a)–4(c) illustrate example height functions from our reconstructed surfaces against the benchmark. We evaluated the reconstruction error within selected ROI in the chest area, as shown in Figs. 4(d)–4(f). The reconstruction accuracy was evaluated based on RMSE. Specifically, on a regular grid with size m × n, we calculated RMSE and standard deviation as the following: $\text{RMSE}=\sqrt{(1/mn)\sum _i^m\sum _j^n\underset{i,j}{\overset{2}{e}}}$, and $\text{S.D.}=\sqrt{(1/mn)\sum _i^m\sum _j^n{\left(e_{i,j}-\left(\sum _i^m\sum _j^n{e}_{i,j}/mn\right)\right)}^2}$, where e_i,j is the point-wise error defined as e_i,j = z_var(i, j) − z_fast(i, j), with z_var and z_fast being the height functions of the surfaces reconstructed by the variational method and our method, respectively. We tested both SR and MSR models on 100 testing point clouds. The error statistics are reported and compared in Table I, where sub-millimeter reconstruction accuracy was achieved from both models. In this experiment, we observed similar reconstruction accuracy from both models, with relative consistent acquisitions from both training and testing point clouds. The computational time was compared in Table II, where both of our models achieved subsecond reconstruction speed, as opposed to the mean 82.2 s from the variational approach. The MSR model required more computational efforts, since it required solving a larger linear system by considering the errors caused by inconsistent noise levels or occlusions.

FIG. 3.

Example point cloud and reconstructed surfaces from both the variational method and our method: (a) example point cloud, (b) example surface reconstructed by the variational method, (c) example surface reconstructed by the SR model, and (d) example surface reconstructed by the MSR model.

FIG. 4.

Example reconstructed surfaces represented by height functions and their error maps: (a) height function of the surface reconstructed by the variational method, (b) height function of the surface reconstructed by the SR model, (c) height function of the surface reconstructed by the MSR model, (d) ROI in the chest area from (a), (e) ROI in the chest area from (b), (f) ROI in the chest area from (c), (g) error map within the ROI from the SR model, depicting difference between (d) and (e), and (h) error map within the ROI from the MSR model, depicting difference between (d) and (f).

TABLE I.

Error statistics of the reconstructed surfaces from 100 testing point clouds based on RMSE (in mm).

	Mean	S.D.	Median
SR model	0.65	0.63	0.01
MSR model	0.62	0.59	−0.02

TABLE II.

Comparison of computational time from different models (in second), with k = 100 (training size) and n = 335 (point cloud size).

	Mean	S.D.	Median
SR model	0.52	0.02	0.52
MSR model	0.94	0.13	0.94
Variational model	82.23	1.15	82.35

We further evaluated the effect of training size on the end reconstruction results for both the proposed models. As shown in Fig. 5, the proposed MSR model performed consistently better than the SR model. There was a sharp error drop at training size 40, which roughly spans two respiratory cycles and covers most of the physiological motion variations of this subject. In practice, we recommend the training acquisition to at least cover a few respiratory cycles of one subject to ensure accurate reconstruction results.

FIG. 5.

Effects of different training sizes on the reconstruction results evaluated on 100 testing point clouds based on RMSE.

3.C. Surface reconstruction with inconsistent occlusions

In the previous experiment, acquisitions from both our training and testing sets were relatively consistent. However, inconsistent noise level and/or occlusions might appear during the acquisition due to either changing of lighting conditions or patient movement. To further test the robustness of the proposed two models under such inconsistent acquisitions, we included additional occlusions in the chest area in all the 100 testing point clouds, as shown in Fig. 6. We tested both of our models on the new testing set and reported error statistics in Table III. Figure 7 illustrates and compares example reconstructed surfaces and their error maps from both models. The MSR model demonstrated its advantage in this experiment, despite the presence of the inconsistent occlusions, as opposed to the large accuracy drop from the SR model. The computational time was compared in Table IV.

FIG. 6.

Illustration of adding additional occlusions to the testing point cloud: (a) the original point cloud, (b) point cloud with additional occlusions.

TABLE III.

Error statistics of the reconstructed surfaces on 100 testing point clouds with inconsistent occlusions based on RMSE (in mm).

	Mean	S.D.	Median
SR model	1.06	0.89	−0.48
MSR model	0.65	0.61	−0.26

FIG. 7.

Example reconstructed surfaces from point clouds with inconsistent occlusions: (a) height function of the reconstructed surface by the variational method, (b) height function of the reconstructed surface by the SR model, (c) height function of the reconstructed surface by the MSR model, (d) error map depicting difference between (a) and (b), and (e) error map depicting difference between (a) and (c).

TABLE IV.

Comparison of computational time from different models with inconsistent occlusions (in second), with k = 100 (training size) and n = 335 (point cloud size).

	Mean	S.D.	Median
SR model	0.52	0.03	0.51
MSR model	0.94	0.17	0.94
Variational model	82.23	1.15	82.35

4. DISCUSSION AND CONCLUSION

We have proposed a fast and robust surface reconstruction method on point clouds acquired by the photogrammetry system, with two specific models to accommodate surface reconstructions under different noise/occlusion scenarios. Taking advantage of the overcomplete nature of the training point clouds, both of the proposed models solve and propagate a sparse linear relationship from the point cloud manifold to the surface manifold, assuming that any given point cloud and its level-set surface shares similar local topological geometry. The proposed SR model directly approximates the target point cloud as a sparse linear combination from the training set, assuming relatively consistent point cloud acquisitions, and the error from the point correspondences built by ICP follows a Gaussian distribution. This model has achieved sub-millimeter reconstruction accuracy, with subsecond reconstruction time from 100 clinical point clouds. To accommodate for the situations when there are changing noise levels and/or the presence of inconsistent occlusions during the acquisition, we have further proposed a MSR model that explicitly models potentially sparse and large one-to-one correspondence errors built by ICP. The MSR model has achieved both sub-millimeter accuracy and subsecond reconstruction time on all clinical testing points, and has demonstrated consistent and robust performance even on point clouds with inconsistent occlusions.

In the proposed method, both the regularization coefficients λ and γ have clear physical meanings, and they are easy to tune in practice. Specifically, λ controls the sparseness of the weighting coefficient for the training point clouds and γ controls the sparseness of the inconsistent noise levels and/or occlusions. In our experiments, we empirically chose and fixed those two parameters, and we find our algorithm robust to values of these parameters within reasonable ranges.

This study assumes that the given point cloud and its level-set surface share similar local topological geometry, which permits the subsequent propagation of the sparse linear relationship from the point clouds to the level-set surfaces. The validity of this assumption is based on accurate reconstruction of the underlying continuous patient surfaces from the training and sufficient coverage of the physiological variations by the training. The variational reconstruction approach we used in this study has been quantitatively validated to have sub-millimeter accuracy.³ Should an alternative method be used, its continuous reconstruction performance needs to be rigorously validated or the overall performance may be affected. The second assumption on sufficient coverage of motion variations is easily satisfied given the semiperiodic nature of the thoracoabdominal motion due to respiration and the fast acquisition rate of most photogrammetry systems. Note that our method does not require consistent acquisition of training point clouds, as consistent samples can be acquired from the corresponding reconstructed continuous surfaces (from inconsistent acquisition).

When the lighting condition is consistent during test acquisition, we expect consistent point cloud representations and recommend using the computationally economical SR model. Otherwise, the MSR model is suggested to provide more robust reconstruction, at the cost of solving a linear system that is (n + k)/k times larger than that from the SR model, where k is the number of training point clouds, n is the size of each point cloud. In this study, under the setup of k = 100 and n = 335, solving the regression of the SR model is roughly four times faster than that of the MSR. However, such efficiency is sacrificed by the additional cost of ICP alignment (the same for both models), and the overall computational cost of the SR model was about half of that from the MSR model in our experiment.