Skip to main content
PMC home page
Dialogues in Clinical Neuroscience logoLink to Dialogues in Clinical Neuroscience
. 2015 Dec;17(4):457–462. doi: 10.31887/DCNS.2015.17.4/kwolf

Measuring facial expression of emotion

Medición de la expresión facial de la emoción

Mesure de l'expression faciale des émotions

Karsten Wolf 1,*
PMCID: PMC4734883  PMID: 26869846

Abstract

Research into emotions has increased in recent decades, especially on the subject of recognition of emotions. However, studies of the facial expressions of emotion were compromised by technical problems with visible video analysis and electromyography in experimental settings. These have only recently been overcome. There have been new developments in the field of automated computerized facial recognition; allowing real-time identification of facial expression in social environments. This review addresses three approaches to measuring facial expression of emotion and describes their specific contributions to understanding emotion in the healthy population and in persons with mental illness. Despite recent progress, studies on human emotions have been hindered by the lack of consensus on an emotion theory suited to examining the dynamic aspects of emotion and its expression. Studying expression of emotion in patients with mental health conditions for diagnostic and therapeutic purposes will profit from theoretical and methodological progress.

Keywords: facial expression of emotion, emotion theory, measurement

“We respond to gestures... in accordance with an elaborate and secret code that is written nowhere, known by none and understood by all.” Edward Saphir, 19211

Introduction

Interest in both emotions and their regulation has grown substantially in recent years, particularly in the neurosciences and specifically in psychiatry. In experimental studies measuring facial expressions, researchers have sought to identify certain patterns of expression.

A vast quantity of data is available; some of these data have been confirmed and some contradicted, depending on the emotion examined and the method used to measure the emotion. A major problem stems from interpreting data which have not always been fully confirmed, and are based on Paul Ekman's theory of six basic patterns of expression (happiness, anger, disgust, fear, sadness, and surprise).2-8

Only the first of Ekman's “basic emotions,” the feeling of happiness with its expressive feature of the “smile,” is observably related to the underlying physiological and facial pattern of expression. Considerable scientific controversy exists regarding Ekman's other basic patterns of expression.

Universality of facial expression

Investigations into the expression of emotion have raised the question of whether basic emotions are shared across cultures. If so, it is important to examine which types of feelings can be described as “basic emotions.” Over the last 50 years, many apparently comprehensive and plausible emotion theories have been proposed. These theories provide different terms to denote “basic emotions.”

Among these theories, two approaches to the universality of facial expressions have evolved. One is the leading neurocultural theory, and the other the behavioral ecology theory, complemented by a compromise between these two theories proposed by Frijda.9,10 The neurocultural theory of facial expression is based on the investigations by Charles Darwin,11 the first scientist to systematically examine the universality of facial expression by carrying out intercultural comparisons. His methods included observing the emotional expressions of young children, comparing emotional expressions in dual interactions, observing the expression of emotions of people born blind, and analyzing the emotional expressions of persons suffering from a mental illness.

From his observations, Darwin found strong similarities in facial expression between people of different cultures (as well as between animals and men), from which he drew conclusions about the universality of expression of emotion. Based on Darwin's scientific works, Ekman completed a great number of studies demonstrating the universality of facial expression, and subsequently identified the types of emotions that he considered universal, ie, observable basic emotional expressions in unconnected cultures. Ekman assumed that similarity between emotional forms of expression indicated their genetic basis. In 1982, he postulated six basic emotions: anger, disgust, fear, happiness, sadness, and surprise, and supplemented these in the 1990s with 11 additional emotions (amusement, contempt, contentment, embarrassment, excitement, guilt, pride in achievement, relief, satisfaction, sensory pleasure, and shame).2-8

Ekman's findings were followed by other scientific contributions on the psychological development of the newborn and people who were born blind. In the newborn, Sroufe12 found reactive crying and native smiling. In an experimental study by Steiner,13 newborn babies who were exposed to a sugar solution with vanilla flavoring reacted with a smile, compared with an expression of disgust in response to a strong smell of rotten eggs.

Izard14 found the facial expression of irritation to be present from the 6th month of life, while the facial expression of fear presents later. Charlesworth15 found a similarity between children who could see and those who were blind with respect to the expression of emotions such as happiness, fear, sadness, anger and trouble, or surprise.

In contrast to the neurocultural theory, Fridlund,16 in his biological behavior-based mimic theory assumed that facial behavior is not automatically related to basic emotions, but rather originates from social contexts and is used more or less indiscriminately. Although the behavioural ecology theory has not been strongly supported, and related evidence is still limited, Frijda9 has suggested a synthesis between these conflicting theories, in that phylogenetic expressions related to types of emotions do exist, and represent basic emotions which, however, are influenced by lifetime experience. The “Facial Feedback” theory assumes that emotions originating from facial behavior are identical to those derived from emotional experience.17

Measuring facial expression of emotion

Three methods are used to measure facial expression of emotion. Each has specific advantages and disadvantages.

Facial Action Coding System; Emotion Facial Action Coding System

These methods involve an analysis of changes in expression in the face of a person. This can range from real-life observation of a person interacting in a group to videotaped interactions in which facial expressions are documented under laboratory conditions upon experimental triggering of a specific emotion.

The pioneer of this approach to measuring facial expression of emotion was Ekman, who, based on his theory of six basic emotions, developed the Facial Action Coding System (FACS).8 This method allows the identification of basic emotions over time, using the image of a videotaped face for image analysis and documenting specific expression changes (called “Action Units”). The advantage of this method is that facial expression of emotion can be studied without biases by the investigator or the technique. Its disadvantages are the dependence on relatively strong emotions and the time needed to analyze the fixed images one by one using the Facial Action Coding System (EMFACS) method. The time required to use this method is one of the reasons that so few studies have been conducted on expression, compared with those on recognition of emotion.

One contribution of EMFACS research is the identification of different forms of smile: expressions of happiness have been extensively investigated as an expression of emotion. Firstly, different studies have shown that M. zygomaticus is the reference muscle for every type of expression of happiness. Secondly, various types of happiness and different types of laughing as indirect indicators of activation of facial muscles have been described and investigated.3-5

The “felt smile” (also “real laughing”) involves the M. zygomaticus and M. orbicularis oculi muscles, which results in spontaneous involuntary smiling/laughing as an expression of perceived happiness. Another type of smiling is the well-known “phoney smile” or “social smile” which involves the isolated use of M. zygomaticus major and is important in social contexts and personal relationships.8

Electromyography method

An electromyography (EMG) method has been developed to recognize activation of facial muscles as accurately and distinctly as possible by using surface electrodes. Due to technical difficulties, for a long period, only two different facial muscles, the M. zygomaticus and M. corrugator supercilii, could be investigated. The methodological validity remained limited because it was unclear whether the recorded muscle activities actually reflected the muscle over which the electrodes had been fixed or another neighboring muscle. From 2005 onwards, the technical performance of systems has reached an acceptable level due to highly selected amplifiable sensitivity. This technical progress allowed identification and independent recording of the activities of subtle visible face muscles. Using a new EMG system, Wolf and colleagues identified the specific facial muscle patterns used to display disgust, appetite, relaxed joy, and aroused joy.18 In validating the system correlations were also found between hungry versus super-satiated states and between the intensity of activation of facial muscles and subjective emotional state. The validity of another EMG system was demonstrated by Schumann,19 who identified emotion-related activations of facial muscles.

The advantages of an EMG system to detect subtle visible facial muscle activity are undisputed for basic research. EMG systems may be useful to develop a consensus-based, stable, empirically founded theory of emotion. The disadvantages of an EMG system are its technical complexity and the restriction of its use in an experimental context. It is therefore not possible to use this system to examine the facial expression of a person in his/her natural social situation.

Automatic face recognition

Commercially available methods for automatic face recognition have improved substantially in quality. These methods have the advantage of allowing an analysis of facial expression of emotion in different settings, where people are studied in their natural social environment without the influence of technical recordings (eg, the Sophisticated High-speed Object Recognition Engine [SHORE™] system).20

The SHORE™ system, as the world's leading system in face detection, is the culmination of years of developments in the field of intelligent systems. SHORE™ resulted in the creation of a highly-efficient real-time C++ software library. The software enables the detection of objects and faces as well as detailed facial analyses based on Ekman's EMFACS-System. Because of the high degree of optimization, it can be adapted to nearly any platform and operating system, especially mobile devices, which makes it suitable for research in real-life settings. An outstanding advantage of SHORE™ is its ability to function in real time with Google Glass.

This opens up an entire spectrum of new smart eyewear applications, including communication aids and even biofeedback aids for patients with disorders like autism or schizophrenia, many of whom have different kinds of difficulty interpreting emotions through facial expression. The missing information could be superimposed in the person's field of vision with data glasses.

To conclude, SHORE™ might supersede EMG systems extensively, and FACS/EMFACS completely.

Facial expression of emotion in metal illnes

Most information on facial expression of emotion is obtained from schizophrenic patients, who have been found to exhibit a paucity of facial expression, particularly in the muscles involved in laughter. This deficit correlates with the amount of negative symptoms.21-26 Hyperactivity of the levator labii muscle is seen in patients with schizophrenia and correlates with positive symptoms (on the Positive and Negative Symptom Scale).23 To date, the importance of M. levator hyperactivity has not yet been conclusively established. Ekman2-8 found M. levator to be the characteristic muscle involved in disgust, as well as in basic aggressive emotional aspects.18 M. levator activity in patients suffering from schizophrenia, however, may not reveal any specific significant emotion, but rather an emotion-specific cophenomenon of “mimic disintegration”.21-32

Mimic disintegration is defined as the inability to organize specific facial muscle movements as an integrated whole, thus making it difficult for observers to decode the emotional state and establish contact or develop a relationship. Using an EMG system, two signs of mimic disintegration were identified: undefined mimic reactions and lack of mimic consistency.

Facial expression of emotion was also examined in depressed patients by Trémeau,33 who studied their capacity to express emotions via facial expressions, compared with both schizophrenic patients and healthy subjects. The participants were asked to create faces showing different expressions of emotions (anger, disgust, fear, happiness, sorrow, and surprise) by using their own verbal expressions. In a second stage, the participants were asked to talk for 2 minutes about each of the six emotions, specifically about their sensations or experience, and to describe how these emotions were present in themselves. During both these tasks, participants* facial expressions were recorded by video and then analyzed using the Ekman FACS.6,8 Both patient groups (schizophrenic and depressed patients) exhibited less spontaneous facial expression of emotion than healthy people and, compared with the schizophrenic patients, depressed patients showed a greater deficit. A discrepancy between minimal facial expression of sorrow and an intense subjective feeling of sorrow was found in depressed patients.33,39

The future of measuring the facial expression of emotion

As a result of the complex measurement methods used to analyse facial emotional expression (FACS/EMFACS and EMG), only a few studies involving small numbers of cases have been carried out, with many basic and applied research questions left unanswered.

This situation might change, as some methods providing a fully automatic face recognition system are already commercially available and provide sufficiently valid data, the quality of which is improving rapidly (eg, the aforementioned SHORE™ system).20 In particular, dynamic aspects of emotion regulation are expected to be more amenable to investigation. A fundamental problem is a limitation in interpreting experimental data because of a lack of a scientifically consensual emotion theory, which has great repercussions on the analysis of facial expression.

The emotion theory forwarded by Hans Lungwitz40,41 offers a persuasive argument for five basic feelings and for both mixed and subsidiary feelings, providing a richly detailed analysis of the dynamic of emotions. Lungwitz defined five “basic emotions” (hunger, anxiety, pain, sorrow, joy) as biologically encoded and occurring in a predetermined sequence. Based on these five sequential emotions, the subjective dynamic reality with unique mixed emotions and feelings is constructed by the individual brain. The term “sequence” means that every individual's action needs an initiating emotion, called “hunger,” which is followed by the emotion “anxiety,” psychologically known as activation preparing the individual for action. The cycle is then completed with the emotion of joy after fulfilling the concrete action. This theory can be considered as a step towards modern theories of action, combining motion and motivation.

Conclusions

A detailed clarification of how emotions are regulated and how the dynamics of facial expression of emotion can be explained could contribute to improved knowledge about basic research in a social setting. Even more importantly, it could improve understanding of, and therapeutical intervention on, the interactive and social consequences of emotional expression deficits in persons with mental illness. Innovative studies in the field of facial expression of emotion may provide detailed answers to unresolved questions in emotion research.

REFERENCES

  • 1.Sapir E. Language: an introduction to the Study of Speech. New York, NY: Harcourt, Brace and Company. 1921 [Google Scholar]
  • 2.Ekman P., Friesen WV., Ellsworth P. What emotion categories or dimensions can observers judge from facial behavior? In: Ekman P, ed. Emotion in the Human Face. 2nd ed. Cambridge, UK: Cambridge University Press. 1982:39–55. [Google Scholar]
  • 3.Ekman P., Friesen WV. Measuring facial movement with the facial action coding system In: Ekman P, ed. Emotion in the Human Face. 2nd ed. Cambridge, UK: Cambridge University Press; 1982:178–211. [Google Scholar]
  • 4.Ekman P., Friesen WV. Pictures of Facial Affect. Palo Alto, CA: Consulting Psychologists Press. 1976 [Google Scholar]
  • 5.Ekman P. Basic emotions. In Dalgleish T, Power MJ, eds. Handbook of Cognition and Emotion. New York, NY: John Wiley & Sons Ltd; 1999:45–60. [Google Scholar]
  • 6.Ekman P. Emotional and conversational nonverbal signs. In: Messing LS, Campbell R, eds. Gesture, Speech, and Sign. New York, NY: Oxford University Press: 1999:45–55. [Google Scholar]
  • 7.Ekman P. Facial expressions. In Dalgleish T, Power MJ, eds. Handbook of Cognition and Emotion. New York, NY: John Wiley & Sons Ltd; 1999 [Google Scholar]
  • 8.Ekman P., Rosenberg EL., eds What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). New York, NY: Oxford University Press; 1999 [Google Scholar]
  • 9.Frijda NH., Markam S., Sato K., Wiers R. Emotions and emotion words. Everyday conceptions of emotion: An introduction to the psychology, anthropology and linguistics of emotion. In: Everyday Conceptions of Emotion. Behavioural and Social Sciences. Vol 81. New York, NY: Kluwer Academic/ Plenum Publishers: 1995 [Google Scholar]
  • 10.Frijda NH. The Emotions. New York, NY: Cambridge University Press. 1986 [Google Scholar]
  • 11.Darwin C. The Expression of The Emotions in Man and Animals. London, UK: John Murray; 1872 [Google Scholar]
  • 12.Sroufe LA. The coherence of individual development: Early care, attachment, and subsequent developmental issues. Am Psychol. 1979;34:834–841. [Google Scholar]
  • 13.Steiner JE. Human facial expressions in response to taste and smell stimulation. Adv Child Dev Behav. 1979;13:257–295. doi: 10.1016/s0065-2407(08)60349-3. [DOI] [PubMed] [Google Scholar]
  • 14.Izard CE., Huebner RR., Risser D., Dougherty L. The young infant's ability to produce discrete emotion expressions. Dev Psychol. 1980;16:132–140. [Google Scholar]
  • 15.Charlesworth WR., Kreutzer MA. Facial expressions in infants and children. In: Ekman P, ed. Darwin and Facial Expressions. New York, NY: Academic Press; 1973:91–168. [Google Scholar]
  • 16.Fridlund A. Human Facial Expression. An Evolutionary View. San Diego, CA: Academic Press; 1994 [Google Scholar]
  • 17.Izard CE. Human Emotions. New York, NY: Plenum Press; 1977 [Google Scholar]
  • 18.Wolf K., Mass R., Kiefer F., Naber D., Wiedemann K. The facial pattern of disgust, appetence, excited joy and relaxed joy - an improved facial EMG study. Scand J Psychol. 2005;46:403–409. doi: 10.1111/j.1467-9450.2005.00471.x. [DOI] [PubMed] [Google Scholar]
  • 19.Schumann NP., Bongers K., Guntinas-Lichius O., Scholle HC. Facial muscle activation patterns in healthy male humans: a multi-channel surface EMG study. J Neurosci Methods. 2010;187:120–128. doi: 10.1016/j.jneumeth.2009.12.019. [DOI] [PubMed] [Google Scholar]
  • 20.Heuberger A., Elst G., Kirsch KH., Hanke R. Microelectronic Systems: Circuits, Systems and Applications. Berlin, Heidelberg, Germany: Springer Science & Business Media; 2011 [Google Scholar]
  • 21.Heimann H., Spoerri T. Phenomenology of expression. Psychiatr Neurol. 1957;134:203–214. [PubMed] [Google Scholar]
  • 22.Heimann H., Spoerri T. The expressive syndrome of mimic disintegration in chronic schizophrenics. Schweiz Med Wochenschr. 1966;87:1126–1128. [PubMed] [Google Scholar]
  • 23.Wolf K., Koppel S., Maas R., Naber D. Identifizierung möglicher Kennparameter der “mimischen Desintegration” bei schizophrenen Patienten mittels einer Gesichts-EMG-Methode. Nervenarzt. 2005;76:1103–1108. doi: 10.1007/s00115-004-1782-8. [DOI] [PubMed] [Google Scholar]
  • 24.Okada T., Kubota Y., Sato W., Murai T., Pellion F., Gorog F. Common impairments of emotional facial expression recognition in schizophrenia across French and Japanese cultures. Front Psychol. 2015;23:1018–1026. doi: 10.3389/fpsyg.2015.01018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Gottheil E., Thornton CC., Exline RV. Appropriate and background affect in facial displays of emotion. Comparison of normal and schizophrenic males. Arch Gen Psychiatry. 1976;33:565–568. doi: 10.1001/archpsyc.1976.01770050033004. [DOI] [PubMed] [Google Scholar]
  • 26.Pulkkinen J., Nikkinen J., Kiviniemi V., et al Functional mapping of dynamic happy and fearful facial expressions in young adults with familial risk for psychosis - Oulu Brain and Mind Study. Schizophr Res. 2015;164:242–249. doi: 10.1016/j.schres.2015.01.039. [DOI] [PubMed] [Google Scholar]
  • 27.Borod JC., Welkowitz J., Alpert M., et al Parameters of emotional processing in neuropsychiatric disorders: conceptual issues and a battery of tests. J Commun Disord. 1990;23:247–271. doi: 10.1016/0021-9924(90)90003-h. [DOI] [PubMed] [Google Scholar]
  • 28.Martin CC., Borod JC., Alpert M., Brozgold A., Welkowitz J. Spontaneous expression of facial emotion in schizophrenic and right-brain-damaged patients. J Commun Disord. 1990;23:287–301. doi: 10.1016/0021-9924(90)90005-j. [DOI] [PubMed] [Google Scholar]
  • 29.Schneider F., Heimann H., Himer W., Huss D., Mattes R., Adam B. Computer-based analysis of facial action in schizophrenic and depressed patients. Eur Arch Psychiatry Clin Neurosci. 1990;240:67–76. doi: 10.1007/BF02189974. [DOI] [PubMed] [Google Scholar]
  • 30.Walker EF., Grimes KE., Davis DM., Smith AJ. Childhood precursors of schizophrenia: facial expressions of emotion. Am J Psychiatry. 1993;150:1654–1660. doi: 10.1176/ajp.150.11.1654. [DOI] [PubMed] [Google Scholar]
  • 31.Kring AM., Barrett LF., Card DE. On the broad applicability of the affective circumplex: representations of affective knowledge among schizophrenia patients. Psychol Sci. 2003;14:207–214. doi: 10.1111/1467-9280.02433. [DOI] [PubMed] [Google Scholar]
  • 32.Mandal MK., Pandey R., Prasad A. Facial expressions of emotion and schizophrenia - a review. Schizophr Bull. 1998;24:399–412. doi: 10.1093/oxfordjournals.schbul.a033335. [DOI] [PubMed] [Google Scholar]
  • 33.Trémeau F., Malaspina D., Duval F., et al Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatent comparison subjects. Am J Psychiatry. 2005;162:92–101. doi: 10.1176/appi.ajp.162.1.92. [DOI] [PubMed] [Google Scholar]
  • 34.Trémeau F. A review of emotion deficits in schizophrenia. Dialogues Clin Neurosci. 2006;8:59–70. doi: 10.31887/DCNS.2006.8.1/ftremeau. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Berenbaum H. Posed facial expressions of emotion in schizophrenia and depression. Psychol Med. 1992;22:929–937. doi: 10.1017/s0033291700038502. [DOI] [PubMed] [Google Scholar]
  • 36.Iwase M., Yamashita K., Takahashi K., et al Diminished facial expression despite the existence of pleasant emotional experience in schizophrenia. Methods Find Exp Clin Pharmacol. 1999;21:189–194. doi: 10.1358/mf.1999.21.3.534828. [DOI] [PubMed] [Google Scholar]
  • 37.Earnst KS., King AM. Emotional responding in deficit and non-deficit schizophrenia. Psychiatry Res. 1999;88:191–207. doi: 10.1016/s0165-1781(99)00083-9. [DOI] [PubMed] [Google Scholar]
  • 38.Yecker S., Borod JC., Brozgold A., Martin C., Alpert M., Welkowitz J. Lateralization of facial emotional expression in schizophrenic and depressed patients. J Neuropsychiatry Clin Neurosci. 1999;11:370–379. doi: 10.1176/jnp.11.3.370. [DOI] [PubMed] [Google Scholar]
  • 39.Mattes RM., Schneider F., Heimann H., Birbaumer N. Reduced emotional response of schizophrenic patients in remission during social interaction. Schizophr Res. 1995;17:249–255. doi: 10.1016/0920-9964(95)00014-3. [DOI] [PubMed] [Google Scholar]
  • 40.Lungwitz H. Die Entdeckung der Seele. Allgemeine Psychobiologie. Berlin, Germany: de Gruyter. 1947 [Google Scholar]
  • 41.Lungwitz H., Becker R. Psychobiology and Cognitive Therapy of Neuroses. Basel, Switzerland: Birkhäuser. 1993 [Google Scholar]

Articles from Dialogues in Clinical Neuroscience are provided here courtesy of Taylor & Francis

RESOURCES