Determining the threshold for asymmetry detection in facial expressions

Abstract

Objective

To quantify the threshold for human perception of asymmetry for eyebrow elevation, eye closure and smile, and to ascertain whether asymmetry detection thresholds and perceived severity of asymmetry differ in distinct facial zones.

Study Design

Online survey.

Methods

Photographs of a female volunteer performing eyebrow elevation, eye closure, and smile were digitally manipulated to introduce left-to-right asymmetry in 1mm increments from 0-6mm. One-hundred forty-five participants viewed these photographs using an online survey, measuring accuracy of asymmetry detection and perceived expression unnaturalness (on a scale of 1-5).

Results

Photographs of facial asymmetries were correctly judged as asymmetric over 90% of the time for 2mm or more of asymmetry in eyelid closure, and 3mm or more of asymmetry during smiling. Identification of eyebrow elevation asymmetry gradually rose from 23% correct to 97% correct across the range of 1-6mm of asymmetry. Greater degrees of asymmetry were ranked as significantly more unnatural across all expressions (3 tests; X² (6, N = 145) = 405.52 to 656.27, all P<0.001).

Conclusion

Thresholds for asymmetry detection vary across different zones of the face, and once detected, asymmetry in eyelid position is perceived as more unnatural than asymmetries in either brow elevation or smile. These data will inform counseling of patients with segmental facial weakness and may provide more objective goals for facial reanimation procedures.

Introduction

Patients suffering from facial paralysis develop multiple functional deficits: corneal exposure, epiphora, oral incompetence, and nasal obstruction, among others. Frequently, they also experience psychosocial difficulty based upon the issue of altered appearance.¹ The face is the outward manifestation of a person’s inner state, expressing both voluntary and involuntary emotion; thus, changes in its appearance necessarily evoke changes in how an individual is perceived.² One common chief complaint of facial paralysis patients is that their emotions are frequently misread by others. Patients are mistakenly perceived as angry, especially when smiling, because positive emotions are normally expressed with symmetric facial expressions, whereas an asymmetric expression, particularly an asymmetric smile, can be misinterpreted as a sneer, conveying contempt.

This disconnect between the emotion a patient wants to express and what is recognized by observers is readily apparent to the patient, and causes emotional distress, leading to loss of self-esteem, and may ultimately affect how they view themselves.^{1, 2} To avoid misrepresenting their emotions, patients will often turn the “good side” of the face toward others, or cover the paralyzed side of the mouth when speaking, avoid smiling, or avoid social contact altogether. Because these changes in appearance and behavior can have a profoundly detrimental effect on patients, we have undertaken a study to characterize perception of facial asymmetry. By quantifying the unnaturalness of an asymmetric face, based on the extent and location of asymmetry, we will develop a better understanding of how varying degrees of segmental facial weakness are perceived, in turn allowing us to predict how facial paralysis patients will likely be affected in their social interactions. Not only will these data inform how we counsel facial paralysis patients, but they will also potentially provide objective goals to improve symmetry in both static and dynamic reanimation procedures.

While other authors have previously addressed observers’ perception of facial asymmetry in repose and with movement,^{3, 4, 5} to our knowledge, this is the first study to examine observers’ detection and perception of systematically manipulated degrees of asymmetry in facial expressions. The assessment of the human face is a complex and dynamic process, taking into account position at rest and with movement, as well as excursion timing and velocity, and the issue of detection threshold for asymmetry in the distance a facial feature travels during expressions has not been addressed in the literature. For this reason, we propose to prospectively evaluate the accuracy with which volunteers are able to identify segmental weakness in facial expressions, as well as determine the severity of the perceived asymmetry.

Materials and Methods

High resolution digital photography of a healthy 31 year old female performing the following facial movements was performed: eyebrow elevation, eye closure, and smile (figure 1). A digital camera (Nikon D90 with Micro Nikkor 105mm lens, Nikon Inc., Melville, NY) was used for image capture, and umbrella lights (StudioMax II 320, Photogenic Inc., Chicago, IL) were used for illumination. Informed consent for photography was obtained under a study protocol approved by the Institutional Review Board at the Massachusetts Eye and Ear Infirmary.

Figure 1.

The 3 facial expressions, each shown with 0mm, 3mm, and 6mm of asymmetry. A) Eyebrow elevation, B) Eyelid closure, and C) Smile.

Each image was assessed for naturally occurring asymmetries in periocular and perioral regions, and these were corrected to provide a symmetrical baseline from which incremental asymmetries could be added for comparison using Photoshop CS software (Adobe Systems, Inc, San Jose, CA). Because the volunteer had no history of facial weakness, only subtle digital manipulations (<1mm) were required to establish the symmetrical baseline image set. The corrected images were then digitally altered to introduce asymmetries in increments of 1mm, up to a maximum of 6mm, in order to simulate segmental facial weakness. In the raised eyebrows series, the right brow was lowered from a maximally elevated position by 1mm in each successive image, while the left remained maximally elevated. In the eye closure series, the right side remained closed while the left upper eyelid was elevated in 1mm increments, using eyelid images copied from a series of photographs taken during varying degrees of eye closure. In the smile series, the right oral commissure remained in full lateral excursion while left side excursion was reduced in 1mm increments.

Photographs were viewed by participants using an internet-based questionnaire service (surveygizmo.com, Widgix, LLC, Boulder, CO). Both non-physicians and physicians, including otolaryngologists and plastic surgeons, were invited to participate in the study via an email that stated the aims of the project, and provided a link to the online survey. Following the link provided an introduction and instructions page, shown in figure 2. The next page displayed a set of sample photographs demonstrating symmetric brow elevation, eye closure, and smile in our healthy volunteer, and explained that each photograph would be presented for 5 seconds. Subsequently, sample questions were provided, asking participants to specify which area of the face appeared asymmetric (brows, eyes, or mouth), as well as to characterize the severity of perceived asymmetry on a 5 point Likert scale, with 1 being natural and 5 unnatural, as previously published by Kim et al.⁶ Additional information was collected on the age and gender of participants. Fields were also provided to allow comments on technical problems and questions about the study.

Figure 2.

Screen shot of the online survey, showing the introduction and instructions as presented to study participants.

Twenty-one images (with 0mm, 1mm, 2mm, 3mm, 4mm, 5mm, and 6mm of asymmetry for each of 3 facial movements) were then presented for 5 seconds each, in random order. Photographs were viewed at a resolution of 842 × 368 pixels. Participants were instructed to click “next” after selecting the area of greatest perceived asymmetry, and rating the unnaturalness of the expression on a 5 point scale (figure 3).

Figure 3.

Screen shot of the online survey interface for selection of the perceived asymmetric area, and the 5 point unnaturalness Likert scale.

At the conclusion of the study, data were compiled into an Excel spreadsheet (Microsoft Corp., Redmond, WA) for analysis. Statistical testing of asymmetry detection and unnaturalness ratings was conducted to determine whether there was 1) a difference among facial regions for asymmetry detection at each degree of asymmetry, 2) a difference across facial regions in unnaturalness score for each level of asymmetry, 3) a difference across levels of asymmetry in unnaturalness score for each manipulated facial region, 4) a difference between genders in asymmetry detection at each degree of asymmetry, and 5) a correlation between asymmetry detection and the age of the observer. Specifically, a Friedman non-parametric test was conducted to determine if significant differences were present among unnaturalness ratings of the images across different degrees of asymmetry. Post-hoc analysis with the Wilcoxon signed-rank test and Bonferroni correction was performed for pairwise analysis of each degree of asymmetry (0mm to 6mm) versus all other degrees within each manipulated facial region (i.e. brow, eyelid, or smile). Pairwise analysis was also conducted across facial region within a given degree of asymmetry. An alpha level of P<0.002 (0.05/21 comparisons) was used when examining pairwise comparisons of degrees of asymmetry within a given expression, in order to mitigate alpha inflation. When examining differences between expressions, at a given degree of asymmetry, an alpha level of P<0.02 (0.05/3 comparisons) was used.

Results

One-hundred forty-five online surveys were completed. There were 56 male (39%) and 89 female (61%) participants, with ages ranging from 14 to 75 (mean 43.6 ± 16.7) years.

Digitally altered facial photographs were presented to participants in random order. When the baseline symmetric images were presented, participants correctly identified symmetry 60% of the time for the smiling photograph, 72% of the time for the brow elevation photograph, and 83% of the time for the eye closure photograph (figure 4). In the brow elevation photographs, accurate identification of asymmetry increased in a generally linear fashion, where 23% of participants correctly noted an asymmetry at 1mm difference, up to a maximum of 97% at 6mm of difference. In the smiling photographs, 58% of participants correctly noted an asymmetry at 1mm and 79% correctly noted an asymmetry at 2mm. From 3mm to 6mm, nearly all participants correctly noted an asymmetry, with detection accuracy ranging from 92% to 100% (figure 4). In the eye closure photographs, 1mm differences were accurately identified as asymmetric by 14% of participants. In eye closure photographs with differences greater than 1mm, detection of asymmetry displayed an abrupt increase. Almost all the participants noted an asymmetry, with detection accuracy of asymmetry ranging from 92% to 98%.

Figure 4.

Percentage of respondents reporting symmetry decreases with increasing degree of asymmetry for each facial expression; note the abrupt decrease in percentage of respondents reporting symmetry as eyelid closure asymmetry increases from 1 to 2mm.

When a participant reported seeing asymmetry, the survey web page presented a 5-point Likert scale for judging facial expression unaturalness, ranging from 1 (natural) to 5 (unnatural). There was a strong correlation between unaturalness ratings and amount of asymmetry for each of the manipulated facial regions (smile R=0.978, brow elevation R=0.916, and eye closure R=0.907), with greater degrees of asymmetry being judged progressively less natural (figure 5). The relationship between excursion asymmetry and unnaturalness ratings was statistically examined with the Friedman non-parametric test, which generated a rank ordering for each subject based on Likert scale values (perception of unnaturalness) versus the 7 levels of asymmetry within each expression, and compared rankings among participants in relation to a chi squared distribution. There was a significant difference in the ranks of facial feature asymmetry amount for all expressions (3 tests; X² (6, N = 145) = 405.52 to 656.27, all P<0.001), confirming the impact of asymmetry on perceived unnaturalness. The only pairs where this pattern was not significant are listed in table 1. Rankings were averaged across participants and are presented in figure 6revealing a consistent relationship between facial feature asymmetry and unnaturalness score for each of the 3 facial regions. A Friedman test was also performed to determine whether each level of asymmetry significantly differed in unnaturalness scores from all other levels among all facial feature positions. There was a statistical difference in the ranks of all expressions at all ranks of asymmetry (6 tests; X² (2, N = 145) = 37.69 to 179.73, all P<0.001). A Wilcoxon signed-rank test revealed that, at all levels of asymmetry, unnaturalness scores of any facial region differed from every other facial region, with the exception of eyelid versus forehead at 0mm (P>0.02) and lip versus forehead at 6mm (P>0.02).

Figure 5.

Median unnaturalness scores increase as the degree of feature asymmetry increases.

Table 1.

Nonsignificant (P >0.002) Comparisons of Steps in Asymmetry Degree According to Facial Feature.

Eyebrow Elevation	Eyelid Closure	Smile
0–1 mm	0–1 mm	0–1 mm
0–2 mm	3–5 mm	3–4 mm
0–3 mm	4–5 mm	5–6 mm
2–3 mm	4–6 mm
	5–6 mm

Figure 6.

Mean ranks of unnaturalness scores increased with degree of asymmetry for each facial expression.

Analysis of gender-based perception of asymmetry revealed that females are significantly more likely than males to rate an asymmetry of 2mm as natural appearing (P=0.004) for all expression ratings combined; there were no other statistically significant differences in asymmetry perception or unnaturalness ratings between sexes for any expression or degree of manipulated asymmetry. No statistically significant correlation was found between age of observer and asymmetry detection either by zone of asymmetry or by degree.

Discussion

It is not difficult to identify an asymmetric face as abnormal; however, pinpointing exactly what aspect of that face makes it seem out of the ordinary is something that has not been studied in detail to date. Prior work by Chu et al³ addressed the question of how much asymmetry of the perioral and periocular areas is required before observers will identify asymmetry of a face in repose, determining that the majority of observers could identify facial asymmetry with 3mm of ptosis at the oral commissure or 3.5mm at the brow. A study by Kim et al⁶ investigated temporal thresholds for detection of facial asymmetry using digitally altered video recordings of facial expressions, finding that most observers are likely to notice a 33ms delay between eyes during a blink, but that a 99ms delay is required before asymmetry of eyebrow elevation, lip depression, or smile is noticed greater than 50% of the time. These data are consistent with the findings in the present study with regard to the higher incidence of asymmetry perception in eye closure when compared to detection thresholds for other areas of the face. Our results and those of Kim et al should be considered in the context of protocols that requested volunteers to search specifically for facial asymmetry, which may not reflect a “real life” situation, wherein bystanders would not necessarily be looking for anomalies. That said, data from the present study parallel those of Chu et al, whose volunteers were naïve to the study goals.

While numerous aesthetic proportions that define a normal or attractive face are well-described,⁷ by quantifying the threshold for human perception of abnormality, we can better, and perhaps more realistically, target reconstructive or rehabilitative goals for facial paralysis patients. For example, instead of attempting to recreate an oral commissure that is absolutely symmetric at rest and possesses smile excursion identical to the unaffected side, we can aim for the more achievable objective of an oral commissure with less than 3mm of asymmetry in repose, less than 2mm of excursion difference when smiling, and less than a 99ms delay in motion, knowing that these asymmetries in smile features and movement will likely go unnoticed and/or not substantially affect perceived naturalness.^{3, 6} Along the same lines, chemodenervation to weaken the normal brow may be guided by the knowledge that up to 3mm of asymmetry in either repose or with elevation, and less than a 99ms delay in motion will likely remain unnoticed, thus complete paralysis of the mobile side is not necessary to restore the appearance of facial balance.^{3, 6} Interestingly, the present study indicates that observers are most sensitive to asymmetries of eye closure, as >90% of study participants noticed any asymmetry >1mm. This parallels the findings published by Kim et al, in which eye closure was also found to have the lowest asymmetry detection threshold; a 33ms delay in eye closure resulted in 67% of participants perceiving asymmetry, while a 97% perceived an asymmetry with a 66ms delay. The other facial movements examined had >90% asymmetry detection thresholds of 99 to 132ms.⁶ Despite the consistency of the findings in the current paper with those of Kim et al, there may be some bias in the results due to the fact that only one model, a young and healthy female, was used in each of the studies; additionally, the asymmetries were produced with digital manipulation for the sake of accurate measurement, perhaps making them seem more unnatural than they otherwise would if occurring in the setting of organic pathology. Further work, presenting photographs of multiple asymmetric males and females, will elucidate whether the patient’s age and gender have any effect on asymmetry perception by observers.

Although patients typically present with a desire to return to “normal,” and have often excessively examined themselves in mirrors, counseling them that restoration of symmetry is a more achievable goal than restoration of a premorbid facial appearance may be the first step towards recovery. Patients habitually exhibit strong emotional attachments to their pre-paralysis appearances, especially with regards to the size and shape of their smiles, and they are frequently resistant to the idea that it is not always reasonable to expect a return to “the way they used to look.” Therefore, helping patients to understand that the key to a normal facial appearance is improved symmetry is often the first step in convincing patients to invest time and effort in the treatment plan. Historically, the literature has demonstrated good correlation between facial attractiveness, or normalcy, and symmetry,^{8, 9} although some studies have demonstrated that computer-generated perfectly symmetric facial images are not necessarily perceived as more attractive than unaltered facial photographs.^{10, 11} Ishii et al established that asymmetric faces (House-Brackmann grade IV-VI/VI) are found to be less attractive at rest and when smiling than normal, symmetric faces, and up to 70% of observers perceived them as demonstrating a negative affect, even when smiling.^{4, 5} They also noted that, while lay observers are perceptive with regards to noticing facial asymmetry, they do not necessarily identify the location of the asymmetry accurately. Facial asymmetry was, nevertheless, strongly associated with a higher unnaturalness rating in both the present study and in the literature.⁶ We have demonstrated that thresholds for asymmetry detection vary by facial feature, and that the degree of asymmetry correlates to the rate of asymmetry detection, up to the point of consistent detection. Thus, restoration of symmetry up to a facial feature-specific threshold should be the primary priority when devising a treatment plan for a patient with facial paralysis.

Conclusion

Herein we have reaffirmed the correlation between facial asymmetry and perception of unnaturalness. Additionally, we have provided some objective guidelines for rehabilitating specific zones of the paralyzed face, that, when taken with data from other studies^{3, 6} may function as a coordinate set to define reconstructive goals in terms of asymmetry at rest, asymmetry in maximal excursion, and delay during motion. While the generalizability of the results we report may be limited based upon the fact that participants were cued to search out asymmetries in the photographs, all of which were digitally manipulated images of a single model, the data presented may be used in conjunction with future work evaluating images of older and younger males and females to guide not only reconstructive procedures, but also physical therapy, chemodenervation, and future therapeutic options, such as biomimetic prostheses.^{12, 13}