C3I-SynFace: A synthetic head pose and facial depth dataset using seed virtual human models.

Abstract

This article presents C3I-SynFace: a large-scale synthetic human face dataset with corresponding ground truth annotations of head pose and face depth generated using the iClone 7 Character Creator “Realistic Human 100” toolkit with variations in ethnicity, gender, race, age, and clothing. The data is generated from 15 female and 15 male synthetic 3D human models extracted from iClone software in FBX format. Five facial expressions - neutral, angry, sad, happy, and scared are added to the face models to add further variations. With the help of these models, an open-source data generation pipeline in Python is proposed to import these models into the 3D computer graphics tool Blender and render the facial images along with the ground truth annotations of head pose and face depth in raw format. The datasets contain more than 100k ground truth samples with their annotations. With the help of virtual human models, the proposed framework can generate extensive synthetic facial datasets (e.g., head pose or face depths datasets) with a high degree of control over facial and environmental variations such as pose, illumination, and background. Such large datasets can be used for the improved and targeted training of deep neural networks.

Specifications Table

Subject	Computer Science
Specific subject area	Computer Vision, Head Pose Estimation, Monocular Depth Estimation
Type of data	Images Annotations
How the data were acquired	The initial 3D virtual human models are collected from the low-cost commercially available software iClone [15] and Character Creator [16]. The data has been produced with a 3D graphics rendering pipeline using the open-source Computer Graphics (CG) software Blender [17]. The source of the 3D virtual models and the generating Python scripts are included in this paper.
Data format	Raw
Description of data collection	Rendering - The face images and the corresponding raw depths have been rendered with the following camera parameters - camera near and far the clip is set to 0.001 and 5.0 meters. Camera sensor size and field of view (FOV) are set to 36 millimeters and 60°, respectively. The yaw, pitch, and roll of a head are constrained to ±30°. To render the face images and raw depths, the RGB and the Z-Pass compositor nodes of the Blender [17] are used with the cycle rendering engine. Background - For complex backgrounds, two scenes (Classroom and Barbershop) from the Blender [17] website have been used.
Data source location	Laboratory: C3Imaging, University of Galway Institution: School of Computer Science, University of Galway City/Town/Region: Galway Country: Ireland
Data accessibility	Direct URL to data: Synthetic Head Pose Datasets: https://data.mendeley.com/datasets/jd4jm3jpp2[7] https://data.mendeley.com/datasets/mc9fzhkvwp[8] https://data.mendeley.com/datasets/vfrfb56sh4[9] https://data.mendeley.com/datasets/pttvxjcmpd[10] Synthetic Face Depth Datasets: https://data.mendeley.com/datasets/z4454fyd8b[4]https://data.mendeley.com/datasets/yzjdjj5w39[5]https://data.mendeley.com/datasets/tbt46rs4y6[6]https://data.mendeley.com/datasets/33kjk7mj7y[1]https://data.mendeley.com/datasets/5wpj8nh2cv[2]https://data.mendeley.com/datasets/2c2r7998vs[3] Virtual Human Models: Due to licensing issues, we cannot release the virtual human models. But the models can be purchased from the Reallusion website from the following link. These models need to be extracted in fbx format and put in a folder structure as described in the ‘Experimental Design’ section. https://www.reallusion.com/contentstore/iClone/pack/Realistic_Human_100/default.html Code: https://github.com/shubhajitbasak/blenderDataGeneration
Related research article	S. Basak, P. Corcoran, F. Khan, R. Mcdonnell and M. Schukat, Learning 3D Head Pose From Synthetic Data: A Semi-Supervised Approach in IEEE Access, vol. 9, pp. 37557-37573, 2021, https://doi.org/10.1109/ACCESS.2021.3063884[11]

Value of the Data

• •The data can be used to train and evaluate computer vision models for head pose estimation, face depth estimation, and face reconstruction. The different lighting conditions and camera positions make the data set robust and capable of generalizing the learning model. Finally, the large scale of the dataset makes it an ideal candidate for deep learning training. For head pose estimation, only two datasets, 300WLP(real) [13] and Nvidia Synhead (synthetic) [14], are available for training. There is no real large-scale dataset available for face depth estimation tasks.
• •As the dataset is generated synthetically, both the head pose distribution and the depth data cover a wide range of angles and a wide variation of background which can be challenging to acquire in a constrained laboratory condition.
• •Both real head-pose and depth data are acquired by inertial measurement unit (IMU) sensors or depth sensors, which are both prone to sensor noise. For example, often, real depth data has missing depth values or holes in it. The most common head pose dataset, Biwi [12], has an average error of 1 degree [14]. These errors in ground truth data eventually pass to the trained model and affect the model performance. On the contrary, the synthetic head pose and depth data generated by our pipeline are pixel-accurate and do not have any of these issues.
• •As both the acquisition of real head pose and face depth data required human subjects, they fall under different data protection and privacy regulations like GDPR, which makes them difficult to collect and use for research purposes. Synthetic data can does not fall under any of these rules, so they are easy to use and generate without any restrictions.
• •Apart from the raw data, we have also provided the source for synthetic human models and open-sourced the data generation scripts. Using this code and the open-source CG software Blender [17], one can generate an unlimited amount of pixel-perfect data by changing the camera parameters, scene illumination, and background scenes.

1. Objective

Recent advancement of deep learning makes it the de-facto choice for facial analysis tasks. But the human face has a complex structure and requires high-quality ground truth data to learn the features. Collecting real ground truth 3D face data either requires expensive 3D scanners or depth sensors, which are prone to noise. Also, as this data acquisition involves human subjects, they often fall under data privacy restrictions and other ethical regulations. So, creating synthetic face datasets can be an alternative that can provide the freedom to generate high-quality, large, and diverse face datasets without any such restrictions. A key element of face analysis is the face alignment information as well as the face depth to get the 3D cues. So, in this work, with the help of low-cost 3D human assets and an open-source CG tool, we have created a large face dataset with their corresponding head pose and raw depth annotations. Further, we used this dataset to train a model for head-pose estimation and face-depth estimation to validate the generated synthetic data.

2. Data Description

The data set contains two parts: The synthetic faces with their ground truth depth and another set of synthetic faces with their ground truth head pose annotations. The following section will describe those two parts in detail:

• •Face Depth dataset:

  Directory Structure - This part of the dataset contains all the rendered face images, their corresponding raw depth in *.exr format, and the head poses data in *.txt files. The root folder contains two subfolders, ‘male’ and ‘female’, with all the identity folders with labels from ‘0001’ to ‘0045’. Each identity folder has a subfolder called ‘Complex’, which signifies the complex background of the rendered images. There are two different background scenes in two subfolders; the first folder contains the Barbershop scene, and the second folder has the Classroom scene. Each of these folders contains the ground truth files with five expressions - angry, happy, neutral, sad, and scared. For each expression, there are three different datasets based on the rendered settings. We have collected the ground truth for three different camera and head movements. The ‘CameraTran’ folder contains the ground truths when the virtual human models are at the origin of the scene, and a random translation motion is added to the camera. The ‘HeadCameraRotTran’ folder has all the ground truths where the models are in the scene origin, and a head rotation {-30°, +30°} is applied to the head bone of the model, and a simultaneous rotation and translation are added to the camera. The last folder, ‘HeadRot,’ contains the ground truth, where the camera is placed in front of the face in a fixed location, and a rotation {-30°, +30°} is applied to the head of the model. Fig. 1a shows the directory structure, and 1b shows some examples of rendered ground truth face images and their corresponding raw normalized depth visualized in grayscale and plasma color maps.

  Ground Truth Annotations - Each of these final folders contains three type of ground truth data –

  • I.
    rgb_<id>.jpg - The rendered ground truth face image in RGB (*.jpg) format.
  • II.
    depthExr_<id>.exr - The raw depth (distance of the face point from the camera center) values in *.exr format.
  • III.
    data_<id>.txt - The ground truth annotations in *.txt files. Each text file contains four ground truth annotations - camera location (x,y,z coordinates), head point location (x,y,z coordinates of the head bone), camera rotation (yaw, pitch, and roll of the camera) and head rotation (yaw, pitch, and roll of the head bone). Fig. 2 shows a sample annotation text file.

Fig. 1.

(a) Folder structure for the facial depth data (b) Sample data rendered in Blender – 1^st column is the RGB image, 2^nd and 3^rd are the normalized depth data (*.exr) visualized in colormap and grayscale.

Fig. 2.

Sample annotation data consists of the camera position, head point location, camera rotation, and head rotation.

The depth dataset contains a total of 37670 sets of ground truth images and their corresponding annotations (.exr and .txt) with a total size of around 45 GB.

• •Head Pose Dataset:

  Directory Structure – This part of the dataset contains the ground truth face images with varying head pose and their corresponding head pose annotations. The root folder contains two subfolders, ‘male’ and ‘female’, with all the identity folders with labels from ‘0001’ to ‘0045’. Each of these folders contain ground truth RGB images and their corresponding head pose data annotations in a text file. Fig. 3a shows the directory structure and 3b shows some samples of ground truth RGB images with varying head poses.

  Ground Truth Annotations - Each of these folders contains the ground truth data –

  • I.
    rgb_<id>.jpg - The rendered ground truth face image in RGB (*.jpg) format.
  • II.
    data_<id>.txt - The ground truth annotations in *.txt files. Similar to the depth data each of these text file contains four ground truth annotations - camera location (x,y,z coordinates), head point location (x,y,z coordinates of the head bone), camera rotation (yaw, pitch, and roll of the camera) and head rotation (yaw, pitch, and roll of the head bone). Fig. 2 shows a sample annotation text file.

Fig. 3.

(a) Folder structure for the head pose data (b) Ground truth synthetic face data rendered in Blender with varying head pose.

This part of the dataset contains a total of 72060 pairs of ground truth images and their corresponding annotations (.txt) with a total size of around 32 GB.

3. Experimental Design, Materials and Methods

This section describes the methodology for generating the data set, including the details of the FBX models and the rendering pipeline. It also provides the details of the Python code that has been used to generate the ground truth data with the help of Blender [17].

• • 3D Model Generation

  Though we are not able to release the virtual model publicly, here we provide the detailed methodology to create this part of the data.

  • I.
    The models can be generated from the ‘Realistic Human 100’ package in iClone [15] software. It provides the functionality to add expressions to the face morphs. The models can be exported in FBX formats from the iClone Character Creator [16].
  • II.
    The iClone [15] tool provides a feature to add different facial expressions to a morph to enhance the facial mesh's diversity. We have added random changes in the face morph and added different clothing to the model. Then we added four different expressions angry, happy, sad, and scared. The default model is the neutral one.
  • III.
    Then we export the models in fbx format through the iClone Character Creator [16] export pipeline1.
  • IV.
    The FBX files need to be structured in the following manner to run the python scripts provided to generate the ground truth data. The root folder contains the two subfolders, ‘male’ and ‘female,’ which have the male and female model files within them. In each of these folders, there are identity folders for female and male models, starting from ‘0001’ up to ‘0100’. Each identity folder has a subfolder called ‘Simple’ with five subfolders containing the FBX model files with five different expressions - angry, happy, neutral, sad, and scared. Each of these folders has the texture (in textures and fbm folders) and other image files associated with the FBX models. Fig. 4a shows the directory structure of this data set.
  • V.
    These models are then imported into the existing Blender scene. We have used two different Blender scenes – Barbershop and Classroom collected from the Blender [17] free library. Fig. 4b shows the sample models (in FBX format) imported into Blender [17] with their armatures (bones) and textures. The imported FBX models are rigged with armatures, later used to add head movements.

• • Rendering Ground Truth

  To render the ground truth, we followed the following steps. All these steps are performed in batch through python scripts that will be explained in the next section.

  • I.
    A camera is added to the scene in perspective mode, and adequate illumination (e.g., area, sun, point, and spotlight) is added on top of the existing light in the Blender [17] scenes.
  • II.
    To apply the head pose, the neck bone is selected. An empty object is added to the center of the two eyeballs, chosen as the center of the head. The camera's optical axis is set as normal to the plane of the two eyeballs to select the initial head pose. The neck bone's translation and rotation have been copied to the empty object, which adds constraints to the empty object to follow the neck bone.
  • III.
    Once the initial setup is completed, a uniform rotation is applied to the neck bone in the sequence of PRY (pitch, roll, and yaw), and the keyframes are saved.
  • IV.
    After the final design, the ground truth face RGB images with their corresponding raw depths and annotations are generated by a Python script with the help of Blender [17]. Blender's [17] in-build Python support is used to run these scripts. Details about the rendering parameters used while generating the data are shown in Table 1.
  • V.
    We have also provided a separate dataset emphasizing head pose annotations only. We have not used virtual environments because we have not collected the depth information for this part of the data. Instead, we have set real images as the background and put the virtual models in front of them to render the ground truths.
  • VI.
    We have added discrete head rotations to the models in an interval of 3°. The yaw, pitch, and roll ranges are ±80°, ±70°, and ±55°, respectively. Though these rotations cover an extensive range of angles, as these are discrete linear sequences, these do not cover some cross-rotation angles. So, to cover all the practical human head pose angles, we have also applied the rotations collected as the ground truth of a real dataset called Biwi [12].
  • VII.
    After applying the head rotation and saving the keyframes, we render each frame through the Blender [17] cycle rendering engine.

• • Explanation of Generating Code -

Fig. 4.

(a) Folder structure for the virtual human model data (b) Samples of the synthetic human models in fbx format after exporting in Blender.

Table 1.

Rendering parameters used to generate the ground truth in Blender.

Parameters	Values
Camera center and model head center distance	30 centimeters
Camera Near Clip	0.001 meter
Camera Far Clip	10.0 meters
Camera Sensor Size	36 millimeters
Camera Field of View (FOV)	60 degrees
Blender Rendering Engine	CYCLES
Cycles Progression	BRANCHED PATH
Cycles AA samples	256
Cycles Min transparent bounces	32
Cycles Light sampling threshold	0
Cycles Sample clamp indirect	0
Cycles Max bounces	32
Cycles Diffuse bounces	0
Cycles Glossy bounces	0
Cycles Transparent max bounces	16
Cycles Transmission bounces	16
Rendering Resolution X	640
Rendering Resolution Y	480

The following section elaborates on the main components of the Python code to generate the ground truth data. The complete code for ground truth generation with complex background is attached to this paper as supplementary material. Also, additional codes for simple and textured background generation can be found on the GitHub page mentioned in the specification table.

importFbx.py: In the first step, the FBX models are imported to the Blender [17] scene (classroom.blend or barbershop.blend), and the imported model is scaled in proper scale to match the Blender [17] scene.

importMisFileBlender.py: First, we set the blender properties. Next, a script is run to add missing texture files (if any) to the *. blend file. sceneSetup.py: In the next step, the Blender [17] scene with the model is set up and head movements are added. Also, the other rendering parameters, like camera properties, are set. In the next step, the midpoint of the two eyeballs is computed by setting empty objects in the eyeball positions and calculating the midpoint of those object locations in global coordinates.

After setting the midpoint of the eyeballs, which will be the center of the head, another empty object is placed at that point. It will provide the head pose ground truth data. Also, the camera is positioned perpendicular to this point as its initial position.

Finally, the neck bone is chosen on which the head rotation is applied. The rotation is applied while inserting a keyframe to save the animation. The following shows a sample example of adding the animations:

compositorSetup.py: Using this script, the compositor nodes in Blender [17] are declared to set the output ground truth paths. It also sets the output data format (e.g., JPEG for RGB and EXR for raw depth data). Finally, the rendering parameters are established, as stated in Table 1.

Following is a code snippet where we set the Blender [17] scene properties.

captureWithGaze.py: Finally, the ground truth is rendered by running this script. As stated in 1.2, the ground truth is generated with three different camera and head rotation settings:

• - Head Rotation: In this scenario, the camera is placed in its initial location and applied to the frames saved during the above scene setup with the head rotation.

• - Camera Translation: In this scenario, the camera is placed in its initial location, and translation is applied while keeping the head model stationary at its initial location. The sample code to apply this translation is given below:

• - Head Rotation and Camera Rotation & Translation: In this scenario, firstly, a virtual half sphere centering the empty object (center of the eyeballs) is constructed before randomly generating distributed points on that half sphere to where the camera is moved. In contrast, the camera axis is pointed to the empty object. At the same time, the previously saved frames are applied to the head model to rotate the head linearly. The sample code to generate the points in a half sphere and apply them to the camera position is shown below:

renderFromCMD.py: To run correctly, we must execute these individual scripts within the Blender [8] python console. So, to generate the ground truth in batch, we pass these individual scripts as a command line argument to the Blender [17] executable while iterating through all the fbx files from the model root folder. We execute the scripts in the same order as discussed above - importFbx.py → importMisFileBlender.py → sceneSetup.py → compositorSetup.py → captureWithGaze.py. A sample execution (for importFbx.py) is shown in the following code snippet.

Ethics Statements

The work did not involve human or animal subjects or data from social media platforms.

CRediT authorship contribution statement

Shubhajit Basak: Conceptualization, Methodology, Software, Writing – original draft. Faisal Khan: Data curation, Resources, Investigation. Hossein Javidnia: Conceptualization, Formal analysis, Investigation. Peter Corcoran: Formal analysis, Validation, Supervision. Rachel McDonnell: Supervision, Funding acquisition. Michael Schukat: Writing – review & editing, Supervision, Project administration.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability

• SyntheticFaceDataset_Male_Part2 (Original data) (Mendeley Data).

• SyntheticFaceDataset_Male_Part3 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Female_Part1 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Female_Part2 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Female_Part3 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Male_Part1 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Male_Part2 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Female_Part1 (Original data) (Mendeley Data).

• SyntheticHeadPoseDataset_Female_Part2 (Original data) (Mendeley Data).

• SyntheticFaceDataset_Male_Part1 (Original data) (Mendeley Data).