A facial reconstruction method based on new mesh deformation techniques

ABSTRACT

This article presents a new numerical method for facial reconstruction. The problem is the following: given a dry skull, reconstruct a virtual face that would help in the identification of the subject. The approach combines classical features as the use of a skulls/faces database and more original aspects: (1) an original shape matching method is used to link the unknown skull to the database templates; (2) the final face is seen as an elastic 3D mask that is deformed and adapted onto the unknown skull. In this method, the skull is considered as a whole surface and not restricted to some anatomical landmarks, allowing a dense description of the skull/face relationship. Also, the approach is fully automated. Various results are presented to show its efficiency.

Introduction

Facial reconstruction aims at recovering the facial appearance of an individual from the sole datum of the underlying skull. The facial reconstruction problem arises in various application fields like forensics, anthropology, archeology or history. In forensic science, facial reconstruction comes in the process of identification of deceased people. When all the usual methods of identification have failed and the skeletal remains are the sole data available for leading to a positive identification, facial reconstruction might be considered as an enhancing tool for “recognition”, producing a short list of candidates from which the individual may be identified by other endorsed methods of identification [1]. In archaeological investigations, facial reconstruction is employed with the purpose of identifying skeletal remains of famous people from the past.

The creation of the face from the skull is a procedure of approximation: from the observation of the cranium, one will not be able to recover a large amount of face features (eyes, hair, lips and ears). Moreover, the facial likeness of a given individual changes considerably depending on factors like nutrition or aging. This flexibility may not be fully reflected on the subjacent skull. Any facial reconstruction tool is expected to account for the uncertainty related to the ill-posedness of the problem, no matter what method is employed (artistic, parametric, statistic, mechanical, etc.). From a mathematical point of view, this issue leads to at least two important difficulties: on the one hand, it raises the question of how to correctly characterize the solution, which might be a continuum spectrum of all the faces that are “consistent” with a given skull rather than a single exemplar; on the other hand, it poses the problem of how to rigorously assess the accuracy of the result. Despite the intrinsic difficulty of the problem, the media are full of facial images that have been constructed on the basis of a single given skull. A fascinating survey of such cases can be found in the book [2]. The work presented in this paper is part of the ongoing multi-disciplinary project FaciLe, grouping together maxillo-facial surgeons, anthropologists, computer scientists and mathematicians from Sorbonne Universités.

The traditional facial reconstruction methods are based on manual procedures, producing 2D portraits or 3D sculptures. These methods basically consist of three common steps: (1) equip (a replica of) the raw skull with a sparse set of anatomical landmarks; (2) apply an average soft tissue thickness to each skull landmark in order to estimate a corresponding landmark on the face; (3) draw up or sculpt a face fitting the estimated landmarks. Most practitioners add a face muscle model in order to enrich the anatomical accuracy of the reconstruction, leading to the so-called Manchester method described in [1]. The results obtained from forensic art are often quite plausible, as the medical artists may take anatomical, historical, archaeological or other types of expertise into account, giving the observer a feeling of coherence. However, the final result of a manual reconstruction depends on the subjectivity of the artist. Additionally, one single reconstruction requires several days of work of a well-experienced forensic artist, making impracticable the realization of multiple instances and feature variations. In order to alleviate these difficulties, several computer graphic software packages have been developed. These animation software packages use the same methodology as manual methods, allowing the expert to tune some modelling parameters and combine the human expertise with the flexibility of the software [3]. However, this approach does not eliminate the subjectivity in the reconstruction. During the last 30 years, a large amount of work has been devoted to the conception of objective fully automated methods. The common pipeline of modern facial reconstruction software is described [4]. First, an expert examines the unknown skull in order to determine anthropological parameters like age, gender and ethnicity. Then, a virtual replica of the dry skull is produced and represented according to the modelling parameters. A craniofacial template encoding face, skull and soft tissue information are derived from a head database. Then, an admissible geometric transformation drives the adaptation of the craniofacial template onto the unknown skull, according to the “proximity” between the skulls. As a result, the template face is deformed onto the predicted face associated with the unknown skull, linking together information coming from both the database and the examination of the unknown skull. Finally, a skin texture and hairiness are added to the reconstructed face.

The approach proposed in this article combines classical features – as the anthropological expertise for classifying the unknown skull and the use of a head database – with mathematical and computational skills as 3D geometric modelling. The contribution of this work is threefold. First, a method for generating a closed surface mesh model of the skull template is introduced. The method relies on an original mesh evolution technique. A template shape is iteratively deformed, producing a sequence of shapes that get “closer and closer” to the source skull. Second, the elastic shape matching method first introduced is used [5]. This technique is used to link skulls (or faces) with each other and learn about their similarities. This procedure is the heart of the reconstruction method and its most original feature. In particular, it allows the method to be fully automated by removing the need of landmarks. Third, the shape matching tool is combined with soft tissue deformation techniques from computational surgery for transporting the skull/face templates onto the unknown skull. The common denominator of the three parts is the use of elasticity equations for driving the shape deformation.

The paper is organized as follows: the first section deals with the acquisition and the construction of the database; the second section describes the representation of the skull templates; the third section presents the method for matching skull and face templates; the fourth section describes the process of reconstruction of the unknown face. Eventually some facial reconstructions are presented to prove the efficiency of the approach. Figure 1 summarizes the various steps of the procedure.

Figure 1.

From CT scans to densely matched surface templates.

Data acquisition

Some recent numerical facial reconstruction methods rely on the a-priori information contained in a database of coupled skull and face templates. The acquisition of both skull and face is accomplished by head CT scans of living subjects, allowing a good visualization of hard tissues. Standard segmentation tools and 3D reconstruction algorithms lead to the definition of dense surfaces of both skull and face from CT data. Unfortunately the invasiveness of this technique causes serious legal and ethical problems, preventing the constitution of a large database of healthy subjects. Due to this difficulty, several studies have proposed to exploit the relationship between soft and hard tissues by means of average soft tissue thickness measurements [6–8].

Soft tissue depth tables are usually used in combination with a large database of face templates [4]. Facial templates can be acquired by non-invasive techniques such as stereophotogrammetry [9], making the constitution of a large database painless and easy. However, despite the acquisition of dense surface templates for describing the outer face, the average values of soft tissue thickness are systematically measured on a sparse set (<53) of (manually positioned) anatomical landmarks. See [10] for a review of recent 3D landmark-based facial reconstruction methods. Since the manual measurement is time consuming and requires expertise in correctly identifying the landmarks, it is actually infeasible to extend these measurements to a dense distribution of points. See [11] for a discussion about the use of sparse soft tissue measurements for facial reconstruction purposes. Several authors have claimed the importance of using a dense representation of the soft tissue information, for example by describing the intra-subject correlation in terms of the volume between the two boundary surfaces representing the face and the underlying skull [12,13]. The results presented in this study are based on a collection of 26 head CT scans of female healthy subjects aged between 20 and 40 years. The CT images used have been provided by the statistical facial reconstruction project of Paris Descartes University presented in [14]. The 3D reconstruction from tomographic data described in this section, combined with an original wrapping method, allows to define closed surface models of both face and skull.

Image segmentation

Data segmentation consists in identifying the bone and soft tissues on stacked 2D grey-level images. This procedure has been carried out in a semi-automated way with the help of the software Amira [15]. First, the CT slices are automatically pre-segmented using a multi-threshold technique. This step consists in partitioning the original images into subdomains whose boundaries are identified by given intensity values. The bone and soft tissues threshold values used are described [14]. From the pre-segmentation step, two sets of binary images – respectively, for bone and soft tissues – are obtained. By stacking these slices one can essentially detect the 3D structures. However, the intensity-based segmentation is not enough to ensure a correct separation of the tissues of interest, due to the presence of noise on the data and artefacts occurring during the acquisition process. The binary images are then cleaned by removing the so-called islands (very small structures whose contours are defined by only a few pixels). These structures can be external to the tissues of interest (noise added during the acquisition process) or very thin internal structures (small bones inside the cranium). This action helps denoising the images. Moreover in most of the subjects, large artefacts due to dental filling are observed on the images. These defects need to be manually removed on each affected slice (Figure 2).

Figure 2.

General procedure for image segmentation. (A) Reference grey-level image; (B) pre-segmentation of the bone using a threshold technique (red); (C) segmentation of the bone after denoising and artefact removal (red).

Geometric mesh processing

From the segmented 2D slices, 3D mesh models of the skull and the face are generated by a marching cube algorithm [15]. These initial meshes contain, in general, a prohibitive number of elements that are redundant and oversampled to correctly describe the geometry of the model. These dense meshes are simplified thanks to the surface remeshing tool mmgs [16], using a local modification approach. The remeshing procedure aims at providing (1) a correct and accurate geometric approximation of the underlying 3D model (geometric mesh) and (2) a computational mesh of high quality elements suitable for finite element simulations (computational mesh). To ensure vicinity between the original and the remeshed triangulations, the remesher controls the discrete Hausdorff distance between the two sets of triangles. The element size of the simplified mesh is locally adapted to the surface curvatures, ensuring a correct approximation of the surface geometry. As illustrated in Figures 3 and 4, this procedure also removes the “staircase” artefacts due to the spatial discretization and connectivity.

Figure 3.

Example of surface remeshing. (A) 3D mesh of the skull after segmentation (2 179 332 triangles); (B) 3D mesh of the skull after geometric remeshing (156 301 triangles).

Figure 4.

Example of surface remeshing. (A) 3D mesh of the face after segmentation (1 286 542 triangles); (B) 3D mesh of the face after geometric remeshing (90 949 triangles).

Generation of surface model of the skull

The human skull is characterized by a complex structure, showing small details that are difficult to both acquire and handle numerically. Due to these difficulties, several authors opt for describing the skull through underlying anatomical or geometrical substructures. The most popular choice leads to the definition of a sparse set of anatomical landmarks, possibly coupled with a dense representation of the skull [17–19]. Some authors use automatically detected continuous crest-lines [13]. The process of matching skull templates is then driven by the outlined feature structures, by requiring their best alignment.

The purpose of this section is to characterize the skull template in terms of a bounded domain, known by a closed mesh of its boundary. This issue is related to the more general problem of surface reconstruction from sample points. Surface reconstruction methods have been extensively investigated in the context of interface evolution via level set methods [20] or via deformable surfaces [21,22]. The specific task, already evoked [6,23], is to define a closed surface of the skull. In order to achieve this goal an original wrapping algorithm based on mesh deformation techniques is proposed. First, a (possibly invalid) source triangulation ${\mathcal{T}}_S$ of the skull geometry is generated by standard segmentation and 3D reconstruction tools. Then, a closed surface, for example a sphere, is iteratively deformed, producing a sequence of surfaces that are “closer and closer” to the source triangulation ${\mathcal{T}}_S$. The final deformed surface partitions the ambient space into two sub-regions defining unambiguously an interior (respectively, exterior) domain. This preliminary step is of crucial importance for matching shapes among the head database.

Presentation of the method

Let Γ₀ (respectively, Γ⁰) be the sphere of radius r > 0 (respectively, R > r) and let Ω⁰ be the volume between Γ₀ and Γ⁰. Suppose that Ω⁰ encircles ${\mathcal{T}}_S$ and is filled with a linear elastic material whose deformation is driven by the Elasticity equations recalled as follows:

Elasticity equations: The displacement field of a shape Ω clamped at a part Γ^D of its boundary Γ and deflated under the effect of internal pressure p ∈ H^−1/2(Γ^N) on Γ^N = Γ\Γ^D is achieved as the unique solution $u\in H_{{\Gamma }^D}^1{\left(\Omega \right)}^3:=\{w\in H^1{\left(\Omega \right)}^3,w=0\text{on}{\Gamma }^D\}$ of the following variational problem:

$${\int }_{\Omega }\sigma \left(u\right):\epsilon \left(v\right)dx={\int }_{{\Gamma }_N}pv·nds,$$ (1)

for all $v\in H_{{\Gamma }_D}^1{\left(\Omega \right)}^3$, where the stress tensor σ follows the Hooke's law:

$$\sigma \left(u\right)=2\mu \epsilon \left(u\right)+\lambda \text{tr}\left(\epsilon \right(u\left)\right)I,$$ (2)

with λ, μ the Lamé coefficients and $\epsilon \left(u\right)=\frac{1}{2}(\nabla u+\nabla u^T)$ the linearized strain tensor.

Hence, starting from Ω⁰ deflated on its interior boundary Γ₀, a sequence of shapes Ω^k – with interior boundaries Γ_k – is produced. Points of Ω^k are advected according to equation (1) until they intercept the triangulation ${\mathcal{T}}_S$. Doing so, at each step k the intersection between the vector displacement and the triangulation ${\mathcal{T}}_S$ is checked. In case of multiple intersection points, the closest point is retained. Whenever a contact between the advecting shape and the source triangulation occurs, the first is clamped and forced not to cross the boundary of the latter.

The advecting sequence of internal boundaries Γ_k gets closer and closer to the source mesh ${\mathcal{T}}_S$ thanks to the strict inequality:

$$d_H({\Gamma }_{k+1},{\mathcal{T}}_S)<d_H({\Gamma }_k,{\mathcal{T}}_S),$$ (3)

where d_H( ·, ·) is the Hausdorff distance between the two surfaces. Finally, for k sufficiently large, Γ_k defines a closed boundary that wraps the source triangulation ${\mathcal{T}}_S$.

Numerical issues

From the numerical point of view, the solution of the Elasticity equation (1) is computed by the finite elements method on a volume mesh, that is Ω⁰ is filled with tetrahedra. For its part the boundary Γ₀ is discretized as a surface mesh ${\mathcal{T}}_0$. The iterative algorithm is performed in order to get a sequence ${\mathcal{T}}_k$ of meshes with decreasing values of $d_H({\mathcal{T}}_k,{\mathcal{T}}_S)$. Given a fixed integer N, the procedure ends if all the points in ${\mathcal{T}}_k$ have reached ${\mathcal{T}}_S$ or if N iterations of the process occur without registering a new intersection. The latter condition deals with the potential presence of holes in ${\mathcal{T}}_S$.

Figures 5 and 6 depict the wrapping of a skull triangulation. The parameters used for equation (1) correspond to a very soft and compressible material. The procedure ends after 35 iterations with N = 5, running in a few minutes on a standard laptop computer. About 85% of points on the triangulation ${\mathcal{T}}_{35}$ are clamped on the surface mesh ${\mathcal{T}}_S$.

Figure 5.

Deformation of a template model onto the skull. (A) Initial surface Γ₀; (B) advecting surface Γ₁₀; (C) final surface Γ₃₅.

Figure 6.

Generation of a closed model of the skull. (A) 3D model of the skull: profile view; (B) 3D model of the skull: surface clipping; (C) 3D model of the skull: base of the skull view; (D) wrapped skull: profile view; (E) wrapped skull: surface clipping; (F) wrapped skull: base of the skull view.

Inter-subject shape matching

Shape morphing or matching arises in a wide variety of situations in areas ranging from biomedical engineering to computer graphics. Beyond the specific stakes to each particular application, the general issue is to find one transformation from a given “template” shape Ω₀ into a “target” Ω_T (Figure 7). Such a transformation may be used as a means to appraise how much Ω₀ and Ω_T differ from one another – for instance in shape retrieval, classification or recognition – or to achieve physically the transformation from Ω₀ to Ω_T (in shape registration, reconstruction or shape simplification). See for instance [24] and references therein for an overview of several related applications. In the facial reconstruction context, shape matching is a key ingredient for studying the shape database, making possible the generation of average shapes – possibly weighted according to their “similarity” – as well as for driving the registration of the craniofacial template onto the unknown skull.

Figure 7.

Elastic shape matching allowing to deform a shape onto another.

The problem is stated as follows: given a “template” shape Ω₀ and a “target” shape Ω_T, the aim is to deforme the mesh of Ω₀ onto a computational mesh of Ω_T.

To achieve this purpose, a method that has much in common with that of [25] is proposed, borrowing techniques from shape optimization, and more generally optimal control. Under the assumption that Ω₀ and Ω_T share the same topology, the desired transformation from Ω₀ to Ω_T is obtained through a sequence of elastic displacements, which are obtained by minimizing an energy functional based on the distance between Ω₀ and Ω_T. In this section, the mathematical framework and some numerical issues related to the method are briefly presented. Refer to [5] for more details and for 2D and 3D numerical examples.

Presentation of the method

The discrepancy between a reference shape Ω and a target shape Ω_T is measured by the following functional J(Ω) of the domain:

$$J\left(\Omega \right)={\int }_{\Omega }{d}_{{\Omega }_T}\left(x\right)dx,$$ (4)

which involves the Euclidean signed distance function$d_{{\Omega }_T}$ to Ω_T, defined as

$$d_{{\Omega }_T}\left(x\right)=\left\{\begin{array}{ll}-d(x,\partial {\Omega }_T)& \text{if}x\in {\Omega }_T,\hfill \\ 0& \text{if}x\in \partial {\Omega }_T,\hfill \\ d(x,\partial {\Omega }_T)& \text{if}x\in {}^c\overline{{\Omega }_T}.\hfill \end{array}\right.$$ (5)

In the above formula, d(·, ∂Ω_T) denotes the usual Euclidean distance function to the boundary ∂Ω_T.

In order to decrease the value of J(Ω), the domain Ω must expand in the regions of the ambient space where $d_{{\Omega }_T}$ is negative (that is, in the regions comprised in Ω_T), and to retract in those where it is positive. Note that the functional J(Ω) has a unique, global minimizer Ω = Ω_T, and no extra local minimum point provided Ω_T is connected. It is then expected that an iterative algorithm for minimizing J(Ω), starting from Ω₀, will lead to an interesting way to transform Ω₀ into Ω_T. This paves the way for an iterative algorithm, producing a sequence (Ω_k)_{k = 0, ...} of shapes, which are “closer and closer” to Ω_T: at each step, Ω_k is updated according to

$${\Omega }_{k+1}=(I+u)\left({\Omega }_k\right),$$ (6)

where u is a suitable descent direction for J(Ω).

Now, imagine that all the considered shapes Ω are filled with a linear elastic material. One can compute the unique solution u of the elasticity equation (1) where the pressure p is taken equal to $-d_{{\Omega }_T}$. This vector field u is naturally a descent direction for J(Ω) since, by a classical calculation, the shape derivative of the function J(Ω) satisfies

$$J^{\prime }\left(\Omega \right)\left(u\right)\le 0.$$ (7)

Numerical issues

For numerical implementation, on the one hand, the template shape Ω₀ is filled with tetrahedra and the Elasticity equations are solved on this volume mesh by the finite element method. On the other hand, the target shape Ω_T is only supplied through its signed distance function, e.g. as a piecewise affine function on a large computational box (Figure 8).

Figure 8.

Implicit description of shapes. (A) Target shape Ω_T as the zero-level set of the signed distance function; (B) adaptive remeshing of the computational box.

Starting from the template shape Ω₀, a gradient descent algorithm with adaptive step size is performed in order to get a sequence (Ω_k)_{k = 0, ...} of domains with decreasing values of J(Ω_k). The algorithm stops when the step size is smaller than a fixed tolerance. The global mapping from Ω₀ to Ω_T is easily recovered by the composition of the local displacements between each iteration.

In the proposed examples, the calculation of the signed distance function to Ω_T is performed using the algorithm described in Dapogny and Frey [26]. The error is computed as the L²-norm of the distance $d_{{\Omega }_T}$ calculated on the boundary of the final shape.

Figure 9 depicts the matching of two skull shapes. The convergence of the gradient descent procedure is obtained in 300 iterations and the final error equals 0.1 mm (much smaller than the minimal mesh size), revealing an excellent matching of Ω₃₀₀ with Ω_T.

Figure 9.

Example of skull matching. (A) Template shape Ω₀ and discrepancy with respect to the target shape; (B) deformed shape Ω_k for k = 300 and discrepancy with respect to the target shape; (C) target shape.

Next, a face example is considered. Fourhundred iterations of the gradient descent algorithm have been performed to achieve convergence, running in a few minutes on a standard laptop computer (Figure 10). The error calculated on the boundary of the final shape Ω₄₀₀ is 0.2 mm (again, much smaller than the minimal mesh size).

Figure 10.

Example of face matching. (A) Template shape Ω₀ and discrepancy with respect to the target shape; (B) deformed shape Ω_k for k = 400 and discrepancy with respect to the target shape; (C) target shape.

Remark:

The ears are not included in the face-matching process, since these structures are not linked with the underlying skull morphology. This procedure is simply done by defining patches on the face template and including only the interesting patches in the minimization of J(Ω).

Registration of the craniofacial template onto the unknown skull

The parametrization of skull, face and soft tissue information depends strongly on the nature of the database. Using CT scans of living subjects allows a dense representation of the craniofacial information. If closed surface models of both skull and face are available, then the craniofacial template may be described in terms of the 3D domain delimitated by the outer face and the inner skull boundaries, generating a 3D “mask” that incorporates the soft tissue information. A single craniofacial template may be used for producing templates of all individuals in the database. The procedure relies on the shape matching technique, generating a set of vector fields, one for each individual, each of them allowing to deform the template shape onto a specific shape of the database.

Various methods have been proposed for deforming the craniofacial template onto the unknown skull. On the one hand, template deformation methods use generic mathematical transformations such as radial basis functions [13]. Care has to be taken because no knowledge of facial anatomy is incorporated in these transformations leading sometimes to awkward looking faces. On the other hand, face-specific methods use principal component analysis on a facial database to generate a statistical shape model [19,27–30]. The advantage is that faces are then deformed in a face-specific manner within statistical boundaries. The disadvantage is that the deformations are dependent on the database and can be too restrictive in case of small databases.

Here, the method lies on the “physical” deformation of the craniofacial template onto the unknown skull. This approach allows to overcome the drawbacks which affect existing template deformation methods. The mask is deformed according to a displacement field prescribed on the skull boundary, measuring the deformation of the skull template onto the unknown skull. Under the effect of boundary changes, the mask is allowed to deform as an elastic material, resulting in a deformed face shape that is now adapted to the unknown skull. The use of elasticity for soft tissue deformation is largely customized in the field of computational maxillofacial surgery [31,32].

Craniofacial template generation

Let (S_i)_{i = 0, ⋅⋅⋅, N} and (F_i)_{i = 0, ⋅⋅⋅, N} be the collection of skulls and faces of the database. Thanks to the procedure described above, theses shapes can be at this point considered as closed surfaces. Then, the volume between S_i and F_i can be defined, generating a 3D mask M_i (called craniofacial template).

The craniofacial templates are first roughly aligned by an iterative closest point algorithm. Then, a reference craniofacial template, say M₀, is chosen and filled with an elastic material (Figure 11). It is matched onto all the M₁, …, M_N (with global displacements u₁, …, u_N) thanks to the shape matching process (Figure 12), that is

$$M_i=(I+u_i)\left(M_0\right)\text{for}\text{all}i=1,\cdots ,N.$$ (8)

Figure 11.

Generation of a 3D mask encoding soft tissue information.

Figure 12.

Matching skull and face templates within the database.

Note that this process generates meshes for all the craniofacial templates M₁, …, M_N that share the same number of elements (vertices and tetrahedra) and the same connectivity. The computation of average shapes therefore turns into trivial average of the vector fields. Any convex combination w of the form

$$w=\sum _{i=1}^N{\alpha }_i{u}_i,\text{with}{\alpha }_i\in [0,1]\text{and}\sum _{i=1}^N{\alpha }_i=1,$$ (9)

defines a new shape W through the mapping:

$$W=(I+w)\left(M_0\right).$$ (10)

See Figure 13 for an example of generation of new shapes as a convex combination of three templates.

Figure 13.

The corners of the triangle display three faces in the database. The top template is chosen as reference shape and matched onto the bottom right template and the bottom left template. The middle of the triangle and the mid-edges display the “barycenter” faces computed by averaging the displacement fields.

Face-to-skull mapping via skull matching

Now let u₀ be the global displacement mapping the reference skull S₀ onto the target skull S_T. The 3D mask M_i is elastically deformed under the effect of the boundary changes (Figure 14). The deformation of the template M_i is achieved as the solution U_i of an elastic problem similar to (1), in which the displacement condition U_i = −u_i + u₀ is imposed on the skull boundary and zero traction conditions on the face boundary. The soft tissue deformation step produces N new faces, each one of them obtained by linking a specific craniofacial template M_i with the unknown skull. These facial items are then combined together to predict the most plausible face associated to the unknown skull.

Figure 14.

Elastic deformation of the mask under the effect of skull changes. (A) Initial view; (B) final view.

Results and discussion

Use of facial reconstruction techniques in forensic science

An essential deal of the work of a forensic scientist regards the identification of postmortem bodies. In the special case in which the facial remains are no longer available, the forensic scientist can resort to various techniques to achieve a positive identification of the dead: DNA assessment, dental record analysis, the identification of a tattoo, the serial number of a prosthesis and other accepted methods of identification. All these techniques are based on a principle of comparison (with respect to a sample or to a database) and can indeed fail in the absence of comparative elements. For instance, at the Forensic Medicine Institute in Paris, this situation shows up for 2 up to 5 cases per year. In this context, a facial reconstruction is then the last resort for reaching a positive identification. The key idea is to scan the skeletal remain to generate a virtual replica of the dry skull and to use a facial reconstruction method to produce a virtual estimation of the victim's face. Then, the produced output is compared with a pool of facial photos which contains the actual target and other individuals of similar age and ethnicity. Then, a group of volunteers is asked to choose which face of the pool is the most resemblant to the reconstruction (face pool assessment). In the context of a murder investigation, a call for witnesses can also be demanded. Indeed the reconstructed face should sufficiently resemble to the original victim's face to allow a recognition and enhance a positive identification. There are some forensic cases which have employed facial reconstruction techniques [2,4,33]. Actually in France, forensic facial reconstructions are produced using 2D reconstruction techniques and are rarely successful. Since a few years our multi-disciplinary team works on the conception and development of a open-source software, based on the 3D deformation techniques presented in this paper, which will be used during criminal investigations in addition to other endorsed methods. The main concern of this project is to develop a software fully automatic, usable even by non trained users, and allowing to produce with short time consuming a facial output (or a range of possible outputs) starting from the scan of the unknown skull.

Validation

The validation of a method for facial reconstruction is of tremendous importance to legitimate its use during a criminal investigation. The natural way to address this problem is the “leave-one-out scheme”. One individual is removed from the database and the method is employed to reconstruct his/her face given the sole skull. Then, the predicted face is compared with the available “unknown face”. The discrepancy between the two shapes can be evaluated mathematically by computing distances between them. However, since the final purpose of the method is a positive identification, a recognition test can be also employed for revealing the power of prediction of the method, as it is done in [4]. A recognition test consists in showing the predicted face together with a sample of faces that contains the unknown face. Then, the human volunteer indicates the face (or faces) that is (are) closest to the predicted one. The positive outcome will then correspond to the identification of the unknown face among the sample. Such a study for the method proposed in this article is under process and results will be published in a forthcoming paper.

Hereafter three examples of reconstruction are presented. According to the leave-one-out scheme, one individual is removed from the database and a prediction of his face is generated. Figures 15–17 show the results obtained by computing the mean of all the faces transported onto the unknown skull. Figure 18 shows the distribution of the error (signed distance function) over the surface for the three reconstructions. Remark that in the regions of interest, excluding ears, neck and eyes in particular, the error is less than 1 mm for most points. Also notice that in these regions, the thickness of the tissue is underestimated or overestimated. Using the information coming from the body mass index (BMI) of the individuals in the database could be a way to propose different reconstructed faces corresponding to different BMI of the unknown individual.

Figure 16.

Test case 2. (A) original face; (B) reconstructed face.

Figure 15.

Test case 1. (A) original face; (B) reconstructed face.

Figure 17.

Test case 3. (A) original face; (B) reconstructed face.

Figure 18.

Discrepancy between the reconstructed and the original faces: (A) for the test case 1 of Figure 15; (B) for the test case 2 of Figure 16; (C) for the test case 3 of Figure 17.

The discrepancy $\mathcal{D}$ between a surface Γ and a reference surface Γ₀ can also be evaluated by the following mean error:

$$\mathcal{D}={\left(\frac{1}{\left|\Gamma \right|}{\int }_{\Gamma }{d}^2(x,{\Gamma }_0)dx\right)}^{\frac{1}{2}}.$$ (11)

where |Γ| is the measure of Γ and d(·, Γ₀) is the Euclidean distance to Γ₀. This error estimator is used for evaluating the vicinity between:

• (1)the skull templates of the database and the unknown skull;
• (2)the face templates of the database and the unknown face associated with the unknown skull;
• (3)the face templates after deformation onto the unknown skull and the unknown face associated with the unknown skull;
• (4)the face templates of the database and the reconstructed average face.

The minimal, maximal and mean values of $\mathcal{D}$ for the 25 templates of the database are reported in Tables 1–3 in each of the third cases. In particular, the discrepancy between a face template and the unknown face is smaller after deformation for all the individuals. Thus, the elastic transformation used in the method seems to be a good tool to transport the faces close to the unknown face. Moreover, the discrepancy between the unknown face and the predicted face is smaller than the discrepancy with any individual in the database, so this measure $\mathcal{D}$ can be used for an automatic numerical identification.

Table 2.

Discrepancy between shapes for the test case of Figure 16. (mm)

Item	Min	Mean	Max
Discrepancy between skulls of the database and the unknown skull	4.01	7.01	11.10
Discrepancy between faces of the database and the unknown face	4.41	7.16	11.40
Discrepancy between faces after deformation and the unknown face	2.60	5.30	8.69
Discrepancy between faces of the database and the predicted face	3.50	7.65	13.39

Table 1.

Discrepancy between shapes for the test case of Figure 15. (mm)

Item	Min	Mean	Max
Discrepancy between skulls of the database and the unknown skull	5.44	7.86	13.46
Discrepancy between faces of the database and the unknown face	5.02	8.94	15.69
Discrepancy between faces after deformation and the unknown face	2.93	5.80	10.35
Discrepancy between faces of the database and the predicted face	4.54	7.03	11.28

Table 3.

Discrepancy between shapes for the test case of Figure 17. (mm)

Item	Min	Mean	Max
Discrepancy between skulls of the database and the unknown skull	4.46	9.57	17.30
Discrepancy between faces of the database and the unknown face	4.92	7.82	13.50
Discrepancy between faces after deformation and the unknown face	3.04	5.01	10.60
Discrepancy between faces of the database and the predicted face	3.63	8.20	15.15

Comparison with previous works

A comparison with previous facial reconstruction methods is a quite hard task. The reliability of two methods should be compared starting from the same database and the same test cases, to avoid the dependencies of the methods on the data. Such a comparison is out of the scope of this paper. Here, we limit ourselves to compare the results of the test cases with the outcomes of some statistical facial reconstruction methods [6,17,19]. A statistical method is developed for the purpose of estimating a facial outcome using landmarks [19]. The experiment is carried out from a database of 118 individuals. The mean reconstruction error registered equals 1.4 mm, with a standard deviation of 1.04 mm. A statistical method based on implicit surfaces was employed for the facial reconstruction test using a database of 20 individuals [17]. The mean error registered equals 1.9 mm with a standard deviation of 1.7 mm. A local technique based on surface patches is used for statistically predicting some facial regions [6]. The experiment is carried out using a database of 47 individuals. For both chin region and nasal region, the mean reconstruction equals 0.99 mm, ranging from 0.58 to 1.83 mm for the nose, and from 0.21 to 2.41 mm for the chin. The mean error registered with the method proposed in this paper is reported in Tables 1–3. Note that most of the error is concentrated on the ears, the neck, the eyes and lips regions. The distribution map of Figure 18 shows that for the three studied test cases the mean error over the remaining regions is smaller than 1 mm. The result is similar to the one reported in [6] (which only deals with the reconstruction of two facial regions) and improves the outcomes of [17,19], at least when considering the regions of interest. Also, note that all the experiments were conducted on a smaller database and with no need of landmarks, resulting in a significant improvement in the fastness and feasibleness of the overall process, and avoiding manual intervention.

Conclusion

The proposed reconstruction method lies on the “physical” deformation of templates of coupled faces and skulls onto the unknown target skull. In practice, the acquisition of full head scans of healthy subjects is still a difficult process. Most of the time one can have access to clinical data, meaning that the patients present morphological anomalies or that the scans are only partial (in the case of maxillo-facial examination for surgery purposes). The access to an adequate database of full heads of healthy subjects would enormously improve the final product of the method. Even if the experiments were carried out on a small collection of 26 individuals, the preliminary results produced are very promising. The proposed method for shape matching allows an accurate registration. The method is simple to implement and does not require any a-priori landmark correspondence, allowing for an automatic processing of the database.

Funding Statement

This work received a financial support by Idex Sorbonne Universités under the French funds “Investissements d’Avenir” [grant number ANR-11-IDEX-0004-02].

Acknowledgments

The authors would like to kindly thank F. Tilotta, Y. Rozenholc and J. Glaunes for providing the CT head data [14], L. Uro and S. Slimani for segmenting part of the DICOM images, P. Frey, M. Friess and P. Deuflhard, whose comments improved the quality of the manuscript, C. Dapogny, L. Norgeot, the ISCD team and the partners of project FaciLe for fruitful meetings and D. Baum, S. Zachow and the ZIB team for interesting discussions.

Compliance with ethical standards

This article does not contain any studies with human participants or animals performed by any of the authors.

Disclosure statement

The authors declare no conflicts of interest.