FaceFinder: A machine learning tool for identification of facial images from heterogenous datasets

Abstract

Purpose:

To develop an algorithm to automate the organization of large photo databases using the Haar cascade algorithm for face and eye detection and machine learning tools in Python.

Design:

Retrospective study for the purposes of clinical tool development.

Methods:

We developed an algorithm, termed FaceFinder, to identify front facing images in a large dataset of facial, orthodontal and miscellaneous images. FaceFinder works by detecting the presence of faces and at least two eyes using the Haar cascade. Execution time was recorded using different-sized datasets. A total of 895 images were analyzed by FaceFinder using various thresholds for face and eye detection. Precision, recall, specificity, accuracy, and F1 score were computed by comparison to ground truth labels of the images as determined by a human grader.

Results:

Using medium thresholds for face and eye detection, FaceFinder reached recall, accuracy, and F1 score of 89.3%, 91.6%, and 92.9%, respectively with an execution time per image was 0.91 s. Using the highest threshold for face and eye detection, FaceFinder achieved precision and specificity values of 98.3% and 99.2% respectively.

Conclusions:

FaceFinder is capable of sorting through a heterogenous dataset of photos of patients with craniofacial disease and identifying high-quality front-facing facial images. This capability allows for automated sorting of large databases that can facilitate and expedite data preparation for further downstream analyses involving artificial intelligence and facial landmarking.

Precis

FaceFinder is a machine learning based tool for the sorting of large medical facial datasets.

Introduction

Two- and three-dimensional imaging is widely utilized in surgical specialties for diagnostic, planning, and postoperative assessment purposes. Sophisticated techniques, such as photogrammetry, machine learning, and artificial intelligence, facilitate these by creating predictive models, and providing personalized surgical plans.^1–4 From pre-operative simulations to post-operative evaluations, computational image analysis is driving a more precise and individualized approach to patient care.^3,5 For instance, automated detection of facial landmarks has been leveraged in craniofacial surgery. However, the utility of artificial intelligence (AI) far exceeds the surgical domains.^6–8 RetinaFace and MediaPipe are two algorithms developed for robust face detection and facial landmark identification.^9,10 Previous reports have demonstrated the use of models like OpenFace to detect pain action units for automatic assessment and grading of pain.^11,12 Similar technology has been implemented for quantification of disease severity in Parkinson’s disease, Alzheimer’s disease, and cerebral palsy.^13–15 With respect to ophthalmology, much work has been devoted to automating the prediction of periorbital distances and quantifying the state of disease using facial scene recognition via AI.^16–19

Despite the promise of AI with regard to image analysis in these fields, the difficulty in streamlined organization and curation of large image datasets still hinders the efficiency of research. Given the volume and variety of facial images obtained in the craniofacial and oculoplastic clinic, considerable manual effort and time is required to prepare the dataset for computational analysis, thereby slowing down the research process and potentially leading to inaccuracies. Authors have previously proposed database schemes for the storage of large datasets.²⁰ Additionally, various AI systems exist to help curate image datasets from large open-source data repositories or via scraping the web.^21,22 However, these models were not developed for use in a clinical setting which encompasses thousands of pictures, ranging from intraoperative photos to paperwork. To date, no automated system exists which can parse through and organize facial photographs of patients with craniofacial pathologies.

To address this issue of dataset management for downstream artificial intelligence experiments, we designed an open-source, novel machine-learning pipeline, FaceFinder, to automate the identification and categorization of front-facing facial images from large datasets. FaceFinder is a time-saving tool that will expedite the development of novel insights and therapeutic strategies in the craniofacial and oculoplastic fields via the reduction of data preparation time. FaceFinder has been open sourced to allow further engagement and utility within the community.

Methods

Dataset

This is a retrospective study conducted at a single institution. Approval was obtained from the Institutional Review Board and the study was performed in adherence with the Health Insurance Portability and Accountability Act guidelines and tenets of the Declaration of Helsinki (IRB Number: STUDY2022-0030). All images used in this study were collected from the University of Illinois at Chicago. The original dataset (n = 895) was manually sorted into true front-facing images (n = 388), and negative images (n = 507). The dataset contained photographs from clinical visits of healthy patients mixed with those having craniofacial pathologies affecting the eyes (Supplemental Table 1). The patients were majority female (55%) with an average age of 17.4 ± 5.4 years old. The other images from orthodontia visits, paperwork, intraoperative photographs, and other non-facial images were included. The criteria for a true front facing (positive image) were the presence of a single face and two eyes being observable within the image. An image not meeting these criteria was defined as a negative image. Three human graders created the designations of ground truth labels. A subset of representative images showcasing positive and negative images from the dataset is shown in Fig. 1. An additional dataset of healthy faces, the Chicago Facial Dataset, was also used for external validation.^23–25 This open-source dataset consisted of 827 true positive and zero true negative images.

Fig. 1.

A-C) Cropped examples of human designated true front facing (positive image) and d-F) negative images. The criteria for a positive image were the presence of a single face and two eyes being observable within the image. An image not meeting these criteria was defined as a negative image.

Algorithm development

FaceFinder is a Python-based image processing tool developed to automatically identify and manipulate images of faces from a large dataset, streamlining the process of data preparation for further computational analysis. In addition to standard Python libraries, FaceFinder primarily leverages the Haar Cascade for face and eye detection.^26–28 The Haar cascade is one of the original machine learning face detection techniques proposed in 2001.²⁹ It works by detecting faces and eyes using the handcrafted Haar features. Through a cascading mechanism described previously, highly accurate face detection can be achieved.³⁰ While convolutional neural networks (CNN) have emerged as state-of-the-art for face detection, the Haar cascade is still widely used due to its high accuracy and low computational cost.^21,23,24,31

A complete flow diagram of FaceFinder can be seen in Fig. 2A. The user first defines a source directory containing the images and two destination directories to save the predicted facial images as well as predicted non face images. FaceFinder removes all non-alphanumeric characters from the folder and file names to standardize the inputs to the algorithm. It then identifies photos with faces in them via face detection using the Haar cascade.²⁹ If one or more faces are detected, FaceFinder then applies eye detection via the Haar cascade to each identified face. If at least two eyes are detected in the image, it is deemed as a true front-facing image. For all images meeting these criteria, the face bounding box is expanded by 20% in all directions to include a broader area around the face and the image is cropped using this box, working on the assumption that more relevant content (such as the forehead or ears) is located outside the initial bounding box. Importantly, no further training of the Haar Cascade was performed, other than what was previously published.²⁹ As such, the entire dataset was treated as a testing set and individual train and test sets were not used which is common practice in other machine learning settings.

Fig. 2.

A Flow Diagram of the complete FaceFinder algorithm. A) The files are standardized to have appropriate naming. The algorithm predicts the location of the face and the eyes. It will subsequently crop the image and then copy the EXIF metadata and save the image in the appropriate folder. B) The graphic user interface to set up FaceFinder experiments. Home directory designates the directory containing the image database. A popup window to select the directory is displayed when ‘Select Directory’ is clicked. The minimum eye and face neighbors can be chosen from the dropdown window and ‘Run FaceFinder’ initiates the algorithm. The ‘Stop Processing’ Terminates the program execution.

In the Haar cascade, detections are grouped by proximity, and this hyperparameter determines the number of detections per group required to deem a detection valid. That is, for every group of faces or eyes detected, if there are less than the specified minimum neighbor value of detections, the candidate object will be discarded.^28,32 Multiple combinations of face and eye minimum neighbors were evaluated, and the results were recorded. This hyperparameter can be tuned by others based on their specific use case and database size.

The processed images are then saved to the output folder, with the directory structure and original EXIF data of each image preserved. Additionally, if an image has EXIF data, it is used to orient the image in the ‘portrait’ orientation. This protocol ensures the provenance and other contextual information of the images are maintained for subsequent analysis. A simple user interface was designed to input the paths to folders and modify the minimum neighbors value for both eye and face detection (Fig. 2B). This interface can be expanded to include other hyperparameters of the Haar cascade as the code is freely available.

Evaluation metrics

FaceFinder was evaluated by comparing the predictions to the ground truth label of every image in the initial dataset (n = 895). For all combinations of thresholds evaluated, precision, recall, specificity, accuracy, and F1-score were calculated. These metrics were calculated according to Eq. (1). To evaluate the speed of FaceFinder, the algorithm was run on multiple datasets of different sizes and the time to completion was logged.

$$\mathit{\text{Acc}}=\frac{TP+TN}{TP+TN+FP+FN},\mathit{\text{Pr}}=\frac{TP}{TP+FP},\mathit{\text{Re}}=\frac{TP}{TP+FN}\mathit{\text{Spec}}=\frac{TP+TN}{TP+TN+FP+FN},F1=\frac{2\times \mathit{\text{Pr}}\times \mathit{\text{Re}}}{\mathit{\text{Pr}}+\mathit{\text{Re}}}$$ (1)

Additionally, to assess whether or not the predictions from FaceFinder were likely to be achieved by chance, Cohen’s Kappa was calculated according to Eq. (2).³³

$$\kappa =\frac{2\left(\left(TP*TN\right)-\left(FP*FN\right)\right)}{\left(\left(TP+FP\right)*\left(FP+TN\right)\right)+\left(\left(TP+FN\right)*\left(FN+TN\right)\right)}$$ (2)

Hardware

All timing experiments were performed on a 2020 M1 MacBook air using the M1 chip (8 core CPU) with 8GB of RAM.

Results

FaceFinder processed a total of 895 images successfully using different combinations of threshold values. A combination of 1,3,5 and 8 minimum neighbors were evaluated to explore the effect of a large range of values on FaceFinder’s performance. In the standard implementation of the Haar Cascade, the default minimum neighbor value is 3.³⁴ The maximum precision (98.3%) and specificity (99.2%) was achieved when using the highest threshold (minimum neighbors of 8 for both face and eye detection). The maximum recall (89.3%), accuracy (91.6%), and F1-Score (92.9%) was achieved using eye and face minimum neighbor values of 3 and 5 respectively. At this combination of thresholds, the precision and specificity were also reasonably high at 96.8% and 95.3% respectively (Table 1). The eye and face minimum neighbor combination of 3 and 5 also achieved the highest Cohen Kappa value of 0.878.

Table 1:

Results of FaceFinder using various minimum neighbor values in the Haar Cascade. Bold indicates the highest percentage for any given category. A higher minimum neighbor value increases the threshold of detection. We evaluated combinations of 1,3,5 and 8 minimum neighbors to explore the effect of a large range of values on FaceFinder’s performance. In the standard implementation of the Haar Cascade, the default minimum neighbor value is 3. The utilization of 3 minimum eye neighbors and 5 minimum 5 face neighbors resulted in the highest recall, accuracy, and F1 Score. F1 score is the harmonic mean of precision and recall. The equations for all metrics can be seen in Equation 1 and 2.

Eye Min Neighbors	1	1	3	5	8	8
Face Min Neighbors	1	8	5	3	1	8
Precision	80.6%	97.9%	96.8%	96.1%	95.1%	98.3%
Recall	82.8%	68.8%	89.3%	67.1%	68.1%	58.6%
Specificity	83.8%	98.8%	95.3%	97.8%	97.2%	99.2%
Accuracy	83.4%	85.4%	91.6%	84.0%	84.1%	81.0%
F1 Score	81.7%	80.8%	92.9%	79.0%	79.4%	73.4%
Cohen’s Kappa	.671	.701	.878	.672	.675	.606

To further validate FaceFinder, the pipeline on an open-source dataset (Chicago Facial Dataset) of healthy faces was evaluated with all in the front-facing position.^23–25 Minimum neighbor values of 3 and 5 were used for eye and face detection, respectively. With this, the FaceFinder precision, recall, and accuracy score was 1, 0.99, and 0.99, respectively (Supplemental Figure 1).

A selection of false positives is shown in Fig. 3 with the misidentified face and eye regions marked. When using the lowest thresholds (minimum neighbors of 1 for both face and eyes) for detection, features such as braces were detected to be faces and eyes within the image (Fig. 3A). Using minimum neighbors of 3 and 5 for face and eye detection respectively, scars and corners of mouths were erroneously predicted to be eyes resulting in the misclassification of side profile images (Fig. 3B) as front-facing.

Fig. 3.

Examples of false positive images from FaceFinder. A) Orthodontal image where braces were predicted to be faces (green box) and eyes (blue box). 1 minimum neighbor (lowest threshold) was used with for face detection on this image. (B-C) Side profile photographs where face prediction was accurate, but where the corners of the mouth, chin, and a scar were predicted to be eyes (blue box) A minimum neighbors value of 3 for eye detection was used on this image. (blue box),.

Following successful face detection, FaceFinder executed facial cropping and metadata preservation with 100% success rate. FaceFinder had an average processing time of 0.91 s per image determined by evaluating the time to completion on multiple sized datasets (Fig. 4).

Fig. 4.

Time to completion for FaceFinder process datasets with various size image datasets. Average processing rate was 0.91 s/image.

Discussion

FaceFinder demonstrates the capability to efficiently sort through large, heterogeneous datasets of craniofacial photographs with a high degree of accuracy, specificity, precision, and recall. In clinical research settings, relevant front-facing photographs are often mixed in with other miscellaneous biomedical image data acquired over multiple clinical visits. Even in scenarios where clinical datasets only contain pictures of the face, the logistic process of renaming files, identifying only front facing images (as is standard in much of the literature related to facial scene understanding via AI), and organizing file hierarchies still presents a large time burden.^16–19 The ability to parse these datasets facilitates more efficient data management and research, especially for analysis of soft tissue facial features.

Multiple studies have evaluated and validated the Haar cascade for facial detection.^29,31 Here, we validate these previous studies and show that highly accurate parsing of patients with rare craniofacial pathology can be achieved using simple feature extraction techniques from the Haar cascade, which permit short run times and avoid the reliance on graphic processing units (GPUs) or additional model development. Furthermore, we demonstrate the utility on craniofacial images, a subset that differs from standard evaluation cases in the literature. Also in line with prior literature, we have validated FaceFinder on an open-source dataset of healthy eyes and achieved near perfect performance. FaceFinder is, to our knowledge, the first tool purpose-built to facilitate data for craniofacial and oculoplastic research. Once a dataset has been properly curated and cleaned, further use of AI has been described in the detection and analysis of craniofacial and oculoplastic disease.^2,17,35–37 Given the wide variety of dysmorphology present in these patients, development of a program to accurately identify patient images regardless of underlying condition is of paramount importance.

The best results for precision and specificity using FaceFinder were obtained using the largest threshold combination of minimum neighbors for face and eye detection. This is an intuitive finding as higher thresholds will increase the chance of returned predicted positive images being truly positive while avoiding the misclassification of negative images. However, this high threshold combination does come at the cost of recall (58.6%) as many true positive images were not detected. Of all the threshold combinations evaluated, we determined that a lower threshold for eye detection with a medium threshold for face detection (minimum neighbors of 3 and 5 respectively) is the optimal combination. FaceFinder achieved the maximum recall (96.8%), accuracy (91.6%), and F1-Score (92.9%) while still maintaining very high precision and specificity when run with these values. However, the best hyperparameters for FaceFinder will depend on the research needs of downstream users, and the thresholds can easily be modified within the graphical user interface (GUI). If having a very pure end dataset is of importance, a higher minimum neighbors values would be better, but for best overall performance, values in the middle of the range may be best. In this study, only the minimum neighbors hyperparameter was explored as previous literature has suggested this is sufficient to control the number of true positives and negatives in face and eye detection.³⁸ However, there are other hyperparameters in the Haar cascade which can be tuned on an individual basis to achieve optimal results on user-specific datasets in the future.

There are certain limitations in this algorithm that should be discussed. From a clinical logistic point of view, storage strategies of imaging data may need to be altered to be most effectively compatible with FaceFinder. Integration with tools such as FiftyOne could further increase the ease of use the algorithm across multiple clinical sites.³⁹ From an algorithmic point of view, the simple feature extraction leveraged by the Haar cascade makes it prone to misidentifying objects such as gaps in teeth, braces, or scars (Fig. 3). This is most pronounced at lower threshold values, but we observed false positives even at the best threshold combination evaluated. More sophisticated methods, such as CNNs, could result in higher accuracy compared to the Haar cascade which has been reported previously.³⁰ However, the training of CNNs is more computationally expensive and relies on training or finetuning a network on representative image data.^30,40 As training data for patients with rare craniofacial pathology is generally limited and GPUs are not always accessible in clinical environments, FaceFinder can serve as a preprocessing step to facilitate these research goals through curation of high-quality representative datasets without the need for excessive finetuning. Future experiments could be performed to incorporate CNNs trained for face detection that can be run on a CPU, such as RetinaFace or Mediapipe, into the pipeline to organize and curate large medical datasets.^9,10 Usage of more advanced algorithms may allow for further options in the process of dataset creation such as only selecting side profile images. It would also be interesting to study potential time savings from using such algorithms. Additional limitations are that FaceFinder has not been validated on images of oculoplastic diseases not included in our dataset and we did not evaluate the robustness of FaceFinder to noise that may be present in images captured from different clinical imaging devices.

In conclusion, FaceFinder can aid in efficiently sorting through large, heterogeneous datasets with a high degree of accuracy, specificity, precision, and recall. As such, FaceFinder may be leveraged as a clinical tool for automated sorting of large databases to facilitate downstream analysis of large datasets and expediting data preparation for further analyses involving artificial intelligence and facial landmarking.