Hole Filling in 3D Scans for Digital Anthropometric Applications

Abstract

Anthropometric measurements have been used to assess an individual’s body composition, disease risk, and nutritional status. Three-dimensional (3D) optical devices can rapidly acquire body surface scans in the form of a triangular mesh which can then be used to obtain anthropometric measurements such as body volume, limb lengths, and circumferences; however, the meshes provided by some scanners may include missing data patches known as holes. These need to be repaired in order to obtain correct landmark detection and automatic calculation of anthropometric measurements-especially body volume. In this study, we present ScReAM (Scan Reconstruction for Anthropometric Measurements) which is a fully automated geometrical 3D reconstruction approach to find and fill these holes. We compare ScReAM with Alias and MeshFix which are well-known software used for triangular meshing. Evaluations are derived from a sample size of 47 subjects that were scanned by two different 3D optical scanners. Our results validate the accuracy of ScReAM for reconstructing a mesh for volume calculation.

I. INTRODUCTION

Anthropometry is used to assess body composition changes and evaluate disease risk and nutritional status factors-such as obesity-that may result from over-nutrition or emaciation caused by malnutrition [1]–[4]. Three-dimensional (3D) optical devices can rapidly acquire body surface scans in the form of a triangular mesh. These scanners come with proprietary software applications that provide a list of digital anthropometric measurements; however, these measurements are different among scanners and thus not comparable.

In a large study, we attempt to develop software that takes triangular meshes from different scanners and calculates digital anthropometric measurements-such as body volume, limb lengths, and circumferences-that are independent of the scanner and thus comparable among scanners. However, prior to using the meshes provided by some scanners, a mesh repair step is required for correct calculations of digital anthropometric measurements. This step, referred to as hole filling, repairs scans that include chunks of missing data, known as holes. These holes are often found under the arms, between the legs, on top of the head, or on the feet. This can occur due to lack of coverage or occlusions. Using different devices to acquire 3D optical scans of the same subject would result in holes at different locations with different sizes. These holes can result in failure to provide a water-tight mesh and can lead to difficulties in landmark detection and the automatic calculation of anthropometric measurements.

Although hole filling is straight-forward for simple holes-i.e., holes that are relatively flat and represent a convex shape-it can be very challenging for complicated holes with large curvature and a concave boundary. In such cases, ensuring convergence and creating a triangular surface with well-shaped faces requires a more sophisticated algorithm.

In this study, we propose a geometrical restoration approach to find and fill such complicated holes, independent from the acquisition device. To ensure consistency among devices, we do not assume any prior knowledge regarding the possible location of holes, and we do not use a template mesh to fill the holes. We call our proposed method ScReAM (Scan Reconstruction for Anthropometric Measurements). ScReAm creates a 3D triangular mesh surface which has similar curvature to its surroundings, thus providing a smooth triangle mesh surface.

II. RELATED WORK

Hole filling methods can be categorized into two major groups: point cloud-based [5]–[8] and mesh-based methods [9]–[14]. Cloud-based techniques involve three-dimensional surfaces formed by a multitude of 3-dimensional points called vertices. A surface-based mesh consists of an association of faces, which are composed of three edges that each connect two vertices. Cloud-based methods work well on a low quantity of well-disposed vertices, while mesh-based methods can be applied to more complex-shaped surfaces and produce a higher quality mesh [9]. Measuring accurate anthropometric data from 3D scanned bodies requires a high quality of the mesh.

Two types of reconstruction are well known in the literature: volume- and surface-based. Volume-based processing modifies the entire mesh by generating a volumetric grid from the surface. These methods are rather slow and noisy in comparison to surface-based methods which only consider the faces on the boundary or in a close neighborhood around a hole [15].

One of the leading surface-based, hole-filling methods was introduced by Liepa and incorporated in the earlier versions of the Alias software [10]. In [10], new faces are generated to fill a hole according to the density and shape of the faces previously existing in the boundary of the hole. After this step, fairing-the alteration of a face to make it smoother-is applied to rearrange the previously created faces. As is later shown in the results section, except for one erroneous case, the latest version of the Alias software performs well in filling the holes in terms of leading to accurate volume measurements, however, Alias is proprietary and the algorithm used in the software is not available.

A different approach to surface-based hole filling was proposed by Wang [9]. This approach resulted in the MeshFix software. This method first uses a preprocessing step that corrects topological errors such as intersecting faces or poorly-shaped faces. The hole filling then occurs based on the newly-created topology. The restructuring of the area around the hole is a potential cause of error for anthropometric applications and as is later shown in the results section, the meshing provided by MeshFix leads to volume calculations that are an order of magnitude less accurate than Alias.

Since Alias is proprietary, and Meshfix does not perform well for anthropometric volume measurements, we present ScReAM as an alternative preprocessing step for digital anthropometric calculations. ScReAM is an automated surface-based algorithm for filling holes on a triangular mesh. The meshing provided by ScReAM leads to volume measurements which are an order of magnitude more accurate than MeshFix and are comparable with the latest version of Alias.

III. METHODS

A. Subjects

A sample of 47 adult men and women were asked to change into form fitting shorts and, if female, a sports bra. Subjects with hair that extended below their chin were asked to wear a swim cap. They were asked to remove every piece of jewelry.

B. Scanners

Subjects were scanned by two different 3D optical scanners (ProScanner [Fit3D, Redwood City, CA] and Styku scanner [Styku, Los Angeles, CA])). For all scans, subjects were asked to stand with their hands in a downward V position. The ProScanner and the Styku scanner both use inexpensive, infrared sensors and contain a turntable where the subject stands during image capture. The turntable rotates for less than a minute to complete the 3D scan. The ProScanner uses three stationary cameras that are aligned vertically on a column, while the Styku scanner uses a single stationary camera on a tower.

C. Algorithms

1). Notation:

The data for each 3D scan contains a triangular mesh represented by a list of vertices, v, and a list of faces, f, where v contains x,y,z coordinates of all vertices and f contains indices of the three vertices generating each triangular face.

Each face has three edges, where each edge is represented by two vertices. Let A_i, B_i, represent the two vertices of the ith edge on the boundary of the hole, where 1 ≤ i ≤ N and N is the number of edges on the hole. Then, AiB_i denotes the ith edge on the hole, f_i denotes the face containing A_iB_i and E_i denotes the third vertex of f_i, i.e., f_i = △A_iB_iE_i (Fig.1).

Fig. 1.

A scan with a hole on the shoulder. The hole is outlined in red and the faces on the boundary of the hole are in blue. Each edge along the boundary of a hole, represented by A_iB_i, only belongs to one face, f_i.

2). Hole finding in ScReAM:

In a closed surface, every edge belongs to two distinct faces; however, in cases where there are holes in the mesh, the edges along the boundary of a hole are only used by one face. This is used to find all edges (pairs of vertices) belonging to the boundary of a hole. Since some scans have multiple holes, we then need to group the boundary edges such that each group creates a closed path.

3). Hole filling in ScReAM:

After finding the holes, the following hole filling procedure is performed on each hole:

• Step 1. Initial layer of creating faces

  • 1.1 Calculate the angle between every two adjacent edges along the boundary of the hole.
  • 1.2 For angles smaller than 20°, such as α_i in Fig.2a, replace B_i with A_i-1 as shown in Fig.2b.
  • 1.3 For angles smaller then 90° such as ${\alpha }_i^{*}$ in Fig.2a, create the edge A_i-1B_i and add the face △A_i-1B_iA_i.
  • 1.4 Redefine the hole as in the red dashed line of Fig.2c.

• Step 2. Define the polynomial $\mathcal{P}\triangleq \sum _i\sum _j{P}_{ij}{x}^i{y}^j$ to be the best fit to the faces in a neighborhood around the hole (as shown in Fig. 3).

• Step 3. Perform the iterative hole-filling algorithm, presented in Algorithm 1. In each iteration, a band of new faces (colored green in Fig. 4–f) is created and a smaller hole (dashed red shape in Fig. 4–f) is defined until the hole is entirely filled.

Fig. 2.

Pictorial representation of the stages involved in Step 1 of ScReAM.

Fig. 3.

Step 2 in ScReAM: the polynomial ($\mathcal{P}$) in blue is defined by the gray faces in a neighborhood around the hole, given by a dashed red line. Pictured is a hole with faces around it (left) and the polynomial (right) $\mathcal{P}$ shown on the area around the hole.

Fig. 4.

Pictorial representation of the steps involved in Step 3 of ScReAM.

Remark 1.

In Algorithm 1, ${\widehat{E}}_i$ is defined such that the angles of the triangle ${\widehat{f}}_i$ are limited to the range [0.1π,0.8π] and that ${\widehat{E}}_i$ is positioned at a predefined distance ε from A_iB_i where ε is the average length of all edges on the hole. This ensures that the edges of the newly created faces are all roughly of the same size. Moreover, to patch the hole with a smooth meshing surface, it is important that ${\widehat{E}}_i$ is defined inside the hole. To ensure this, we first find the mirror of E_i with respect to e_i. Then, this value is projected onto the polynomial $\mathcal{P}$, to obtain ${\widehat{E}}_i$ The projection onto the polynomial enforces the newly created faces to follow the curvature of the area around the hole.

Remark 2.

Creating new faces by following the basic idea presented above, problematic cases may occur which result in an non-smooth or poorly-meshed surface. Such cases occur when ${\widehat{f}}_i$ is either overlapping, intersecting or hovering on top of a previously existing or a previously created face. In such cases, Algorithm 1 includes adjustments such as merging vertices that are reasonably close. A detailed description of handling these cases are presented in Algorithm 1.

IV. RESULTS

To evaluate our method, we compare our hole-filling results against Alias and MeshFix, well-known software used for triangular meshing. We employ scans obtained from the Styku scanner and ProScanner separately in this analysis to judge both the ability of our algorithm to recover known anthropometric measurements and to accurately fill genuine holes in the data. Styku scans are initially without holes, thus anthropometric data such as volume can be extracted from the scan and compared to reliably. In contrast, ProScanner meshes commonly yield authentic holes on problematic regions of the human body, such as the crotch or top of the head, yet initial measurements cannot be reliable compared to. Thus, our analysis is two-fold. First, we create artificial holes in Styku scans and subsequently patch them using the various algorithm; volume, a mesh-sensitive measurement, can then be compared before and after creating and patching the hole. Next, we demonstrate the ability of our algorithm to patch authentic holes using ProScanner data, and we compare this to results obtained with Alias and MeshFix.

To begin, we first create holes in scans that are initially complete. We then have all three methods reconstruct the damaged mesh. For each scan, four holes were created on the arm, head, foot, and crotch respectively. Afterwards, we calculate the body volume for the repaired meshes as well as the original, then evaluate the three methods against the original mesh by comparing the calculated body volumes. This emulates the ability of our method to accurately recover anthropometric measurements even from damaged scans.

Results were evaluated with for 47 Styku scans. To illustrate this, in Fig. 5, we plot the linear regression of volumes calculated after hole filling (AHF) to the volumes calculated from the original scan before the artificial holes were created (original). Moreover, we plot the associated Bland-Altman graphs [16], where the x-axis is difference (AHF-original) and y-axis is mean $(\frac{\mathit{\text{AHF}}+\mathit{\text{original}}}{2})$.

Fig. 5.

Linear regression (top) and Bland-Altman (bottom) graphs for Styku scans. AHF refers to volumes obtained “After Hole Filling”. Results are plotted with regards to the original volumes obtained from the initial scan. Columns a, b, and c show results for ScReAM, Alias, and MeshFix, respectively.

A linear regression mapped the AHF volume to a function of the form y = ax + b, where x is the original volume and y is the AHF volume. All three methods yielded an R² = 1 and an intercept that is close to zero, which is ideal; however, the intercept of the MeshFix approach is an order of magnitude farther from the desired result than the two alternative methods.

When investigating Bland-Altman graphs, all three methods are similarly comparable. For each algorithm, the difference between the original and AHF volumes, Δ, is plotted against the average of the two volumes. The median and mean Δ should be as close as possible to zero to ensure accurate anthropometric measures. Our method yielded a mean Δ = 3.58 × 10⁻⁴ ± 7.7 × 10⁻³ and a median Δ = 5.22 × 10⁻⁴. Both values show an order of magnitude improvement over Alias and two orders of magnitude improvement over MeshFix.

Unlike the Styku device, the ProScanner scans often provided ample authentic holes. In this case, correct volumes are not available for comparison. Instead, we report if any of the hole filling methods patch the hole with a surface that is abnormal. Also, as in [9], we visually represent examples of hole-filling results by each method.

For all Proscanner scans, every method successfully filled the holes in the mesh, with a single exception. For one subject, the Alias software struggled to patch a hole on the top of the subject’s head, yielding the error seen in Fig. 6c. Also note that while MeshFix successfully patched each hole, the generated mesh was highly irregular, producing faces that were non-uniform and incongruous with the surrounding original mesh.

Fig. 6.

An overview of common holes found in ProScanner scans. Column a contains the holes outlined in red on the avatar. Columns b, c, and d show the hole filling results of ScReAM, Alias, and MeshFix, respectively.

In Fig. 6, we show various common holes found in ProScanner data followed by the meshes generated by each algorithm to cover them. Holes on the armpit region, sole of the foot, back of the arm, and top of the head are all common in the scans, due to the postures the subjects’ must assume. For each example, our algorithm provides more and smaller faces, yielding a detailed meshed that closely resembles the original mesh surrounding the hole. In comparison, the other two approaches yield much larger faces. This can potentially lead to errors and loss of accuracy.

In comparing these methods, we found that ScReAM performed on par with Alias software in both recovering accurate volumes and in creating uniform meshes over holes; however, the Alias software would at time yield errors that would not occur in other methods. Additionally, it should be noted that the Alias software is proprietary and not available for general use. MeshFix, when contrasted to the aforementioned methods, was a factor of ten less accurate than either Alias or ScReAM in recovering known anthropometric data. This could be due in part to MeshFix’s algorithm, which will redefine faces bordering the hole as well as covering the hole itself-a procedure which may destroy genuine features of interest in anthropometric applications. Accuracy can also be lost in the large, irregular faces of the mesh provided by MeshFix, as more detailed meshes retain more accuracy, especially when considering curved surfaces. In summary, ScReAM shows comparable accuracy with Alias without incurring the errors, and it preserves the mesh and anthropometric features much more accurately than MeshFix.

V. CONCLUSION

Three-dimensional optical scanners can quickly provide body surface scans useful for anthropometry applications. These quick, noninvasive, and efficient techniques yield data that can be used to identify health risk factors, such as obesity or emaciation. In particular, optical scanners provide methods for automatically acquiring measurements such as circumferences or volumes. However, the meshes obtained from these scanners can contain holes of missing data due to lack of coverage or occlusions. To achieve accurate anthropometric calculations from 3D scans, it is necessary to fill the holes in a manner that most closely preserves the original shape. In this paper, we presented ScReAM, which is an algorithm that is capable of filling holes by detecting the edges surrounding the hole and generating new, well-shaped faces and vertices to complete the mesh. We have demonstrated that our algorithm is comparable to existing triangular meshing software; furthermore, we have demonstrated a noticeable improvement over those methods in accurately recreating anthropometric data such as body volume.