Facial expression and emotion

Abstract

Human facial expressions are unique in their ability to express our emotions and communicate them to others. The mimic expression of basic emotions is very similar across different cultures and has also many features in common with other mammals. This suggests a common genetic origin of the association between facial expressions and emotion. However, recent studies also show cultural influences and differences. The recognition of emotions from facial expressions, as well as the process of expressing one’s emotions facially, occurs within an extremely complex cerebral network. Due to the complexity of the cerebral processing system, there are a variety of neurological and psychiatric disorders that can significantly disrupt the coupling of facial expressions and emotions. Wearing masks also limits our ability to convey and recognize emotions through facial expressions. Through facial expressions, however, not only “real” emotions can be expressed, but also acted ones. Thus, facial expressions open up the possibility of faking socially desired expressions and also of consciously faking emotions. However, these pretenses are mostly imperfect and can be accompanied by short-term facial movements that indicate the emotions that are actually present (microexpressions). These microexpressions are of very short duration and often barely perceptible by humans, but they are the ideal application area for computer-aided analysis. This automatic identification of microexpressions has not only received scientific attention in recent years, but its use is also being tested in security-related areas. This article summarizes the current state of knowledge of facial expressions and emotions.

1. Introduction

Humans are social creatures who can hardly survive on their own. The everyday life of most people consists of a multitude of interactions with our fellow human beings. These interactions do not only transfer factual information, but also reveal information about the sender (self-revelation level), the relationship of the interactors (relationship level), and information about what we are trying to achieve from our communication partner (appeal level). Except for the factual information level, sending and perceiving emotions is crucial for successful communication. Emotion perception is helpful for contextual understanding of information, interpretation of others’ actions and also their intentions. Emotions can be conveyed in a variety of ways in human communication. For example, speech or gestures can be used in addition to facial expressions. However, facial expressions are by far the most commonly used form for human communication. The correct interpretation of other people’s emotions from their facial expressions can therefore lead to considerable advantages in social interaction ^{1 2} .

Over the past 50 years, great progress has been made in our understanding of mimic emotion transmission. In contrast, our understanding of emotions themselves lags far behind, and they remain one of the most fascinating and mysterious products of brain function. This is evident in the difficulty of defining emotions.

In this article, we will first address the definition of facial expression, communication, and emotion, and then discuss the voluntary and involuntary aspects of our facial expression of emotion. We will discuss the societal benefits, evolutionary origins, and current possibilities of automated emotion recognition.

2. Facial expression, communication, and emotions

The term of facial expression describes visible movements of the facial surface, which is conditioned by a specific pattern of muscle activity. Facial expressions can be perceived and interpreted by people. It is a central element of human expressive behavior and is particularly suitable for expressing emotions and communicating them to the outside world. The interpretation of emotions from facial expressions requires higher cerebral processing capacities, which besides the evaluation of the current visual input also strongly depends on contextual information and the availability of memory content ² . In contrast to facial expressions and communication, the concept of emotion is not clearly defined and there are still controversial discussions about what an emotion is in the first place and whether consciousness is a prerequisite for emotions ^{3 4 5 6 7 8} . Some authors believe that emotions are conscious abstractions of one‘s internal state, which are assembled cognitively. According to this definition, an emotion requires awareness of one’s self, and thus emotion as a concept would no longer be applicable to many other species. Other authors argue that consciousness is merely a property of emotions and their processing, which is additive to the basal emotion properties ^{9 10 11} .

Emotions are often described by words (happy, sad, angry). In psychological research, “valence” and “arousal” have become accepted as a way of describing the concept numerically. Valence describes the affective content of an emotion as a continuum from pleasant to unpleasant or good to bad, while arousal describes the degree of internal activation by the emotion.

In this paper, the latter definition will be used and thus emotions considered as evolutionarily conserved functional states of the brain ^{7 12 13} . According to this view, emotions emerged from evolutionary processes and primitive forms may still be present in humans as conserved features. However, it is undisputed that emotions trigger certain reactions on biochemical, physiological, and behavioral levels ¹⁴ .

To interpret an emotion from a person’s facial expression, we do not always need the entire face; single parts are often sufficient (e.g., sad eyes or happy mouth) ^{15 16} . The integration of a wide variety of individual features across the entire face is a highly integrative performance that is significantly more complex than just a summation of individual properties ^{17 18} .

The significance of individual facial regions for decoding emotions differs between different emotions. For example, the eyes are particularly importance for the perception of sadness ¹⁹ , anger ^{20 21} , and fear ^{22 23} . Although other parts of the face also provide cues for these emotions, it has been shown, for example, for fear that emotion recognition form the upper part of the face is more accurate than from the lower one ²⁴ . In contrast, people find it easier to recognize happy facial expressions from the lower part of the face ²⁴ . Accordingly, observers of happy faces focus longer on the mouth region ^{22 23 25} . For the nose region, a particular importance in decoding facial expressions of disgust could be demonstrated ¹⁹ .

In addition to the interpretation of individual facial regions, facial expression is also processed and interpreted holistically, i.e., there is still a processing of facial expression that is not composed of the individual features and is thus more than the sum of its parts ²⁶ . The significance of this holistic facial interpretation for the recognition of emotions differs between emotions. However, it is true for all emotions that their interpretation is composed neither purely of the perception and processing of individual features of subregions of the face nor exclusively of this holistic face interpretation, but consists of a combination of these two parallel processes ^{16 27} .

3. Voluntary and involuntary facial expression, classification of facial expression – microexpressions

Humans can control their facial expressions arbitrarily. This voluntary control can be used to feign an emotional state. This deception mostly refers to a communication partner, but can also be directed to oneself. The voluntary control of facial expressions has a target state, which corresponds to an internal model (what we want to express). In addition to the voluntary facial movements, however, felt emotions are generally expressed involuntarily by facial expressions and the corresponding muscles are involuntarily controlled. They penetrate our conscious control and are reflected in our face. This emotional leakage does not always have to be accompanied by large movements of muscles, but can in part also be very discreet. These movements are summarized under the term of microexpressions. Specifically, we understand microexpressions as facial movements that are short-term, involuntary, and automatic. Certain muscle groups contract or relax in fixed patterns. It is further assumed that these patterns have a genetic basis and are closely coupled to the emotional state of a person. The genetic basis has further led to the assumption that there is a set of basal emotional states that are universal across cultures ²⁸ , although this assumption continues to be debated to this day ²⁹ . Because these movements are involuntary, it is assumed that they allow conclusions to be drawn about a person’s actual emotional state.

Of particular interest is the interaction of voluntary and involuntary facial movements. Voluntary facial movement is often used to conceal the actual emotion felt and thus to deceive the communication partner. The question is, whether these voluntary facial movements are fast enough and, more importantly, complete enough to suppress involuntary microexpressions that give a hint to genuine emotion.

In a classic experiment, Paul Ekman and Wallace Friesen filmed a psychiatric patient trying to convince the therapist that she was fine and could be discharged from the psychiatric ward. At the end of the interview, the patient broke down crying and confessed that she continued to be unwell. During the phase of pretending to be well, various short-term microexpressions could be detected, which clearly indicated deception ³⁰ . Subsequently, there were still a large number of studies on this topic that ultimately concluded that it is not always possible for the deceiver to maintain the deception perfectly and that it is often involuntary that emotions escape via facial expressions in the form of microexpressions. Due to the involuntary nature of these microexpressions, it is further assumed that they allow conclusions to be drawn about the actual emotional state of a person.

4. Cultural similarities and differences in facial expression and emotion

The dominance of the Western world in academia over the past century has resulted in the majority of studies on the perception of facial expressions and emotions being conducted on culturally westernized Caucasians. There have been few studies in the last century that have examined the relationship between facial expressions and emotions across cultures and ethnicities. The first significant studies were conducted by the American psychologist Paul Ekman in the 1960s and 1970s and he concluded that facial expression of emotions was universal across cultures ^{28 31} . This was thought to be true for at least six basic emotions ³² . This view remained unchallenged for a long time, but more recent studies now suggest cultural differences ²⁹ . This is especially true when emotional expression is specifically examined according to contextual and cultural influences. For example, people of Western and East Asian cultures show common perceptions of pain, but differences in the mimic expression of pleasure have also been found ²⁹ .

In our everyday life and communication with others, however, we assume as a matter of course that our model of emotions and emotion perception is universal. Seemingly contrary to this statement, however, there is also a racial bias that influences the perception of the mimically expressed emotions of people from other cultures ³³ .

5. Influence of diseases on facial expressions and emotions

Facial expressions – as well as the interpretation of facial expressions of others – can be altered by various diseases. The classic example for a disturbance of the mimic expression of emotions is Parkinson’s syndrome. This is a neurodegenerative disease that is associated with a reduction in facial movements ^{34 35} . In later stages, the ability to modulate facial expressions is severely restricted so that even the term of “mask face” is used ³⁵ . In this stage, the patients do no longer show any emotionally triggered facial expression. This diminished or, in some cases, nonexistent mimic expression has a significant impact on the patients’ ability to communicate and on how they are perceived by others ^{36 37 38} . Thus, observers perceive the corresponding patients as less empathetic, bored, anxious, and cranky ^{36 37 38} . This leads to less interest in relationships with these patients ³⁶ . The consequences are increasing deterioration of social relationships, even with close people, increasing social isolation, and a concomitant deterioration in perceived quality of life ^{34 35} . These significant effects highlight the importance of facial expressions for social interaction.

Reduced or absent facial expressions also have a direct impact on communication behavior. In the context of medical conversations, for example, there are often monologues of the doctors in which disease models are illustrated or further therapy steps are explained. Although in such situations often only one person speaks, the speaker receives a multitude of mimic messages, expressing e.g. the degree of understanding of what is said, but also giving information about the emotional state of the interlocutor. If the speaker becomes aware of these messages, he/she then modulates the content according to the feedback received. The complete absence of any mimic response in such a conversational satiation is an impressive experience, since it gives the feeling of conducting the conversation without orientation. The speaker has no indication of whether the interlocutor has understood what has been said, nor does it become clear whether the interlocutor reacts with annoyance, rejection, or acceptance. The condition leads to uncertainty and discomfort for the speaker. There seems to be an evolutionarily determined need for communicative feedback. The “still face” experiments conducted with babies and toddlers by Edward Tronick seem to point in this direction ³⁹ . In these experiments, the mothers suddenly stop any mimic expression. The children then start various actions in order to restore interaction. They increasingly show reactions of discomfort until they usually react with crying ³⁹ .

These experiments, but also the feelings as adults in similar situations do not only emphasize the importance of facial expressions for communication, but also force the interlocutors to change their communication behavior. Longer monologues are no longer possible, since one has to rely on verbal feedback to what is said. It also makes clear how strongly even “monologues” depend on the constant non-verbal exchange of information.

The effect of reduced or absent facial expressions described in the example of Parkinson’s syndrome is directly transferable to other diseases that are accompanied by a reduced ability to express emotions facially, e.g., muscular dystrophies, myasthenia gravis, or amyotrophic lateral sclerosis. The negative effects on communication and perception of the affected person can also be found in unilateral disorders of facial motor function such as peripheral facial nerve palsy ⁴⁰ .

However, in addition to the reduced ability to express emotions mimically, Parkinson patients also suffer from a reduced ability to perceive mimic expressions of others ^{41 42 43} and also have fundamental difficulties with processing the emotional content ^{44 45 46} . This is a characteristic of many neurodegenerative diseases. The accompanying cerebral changes of the disease are often considered as cause. However, the influence of one’s own restricted facial expressions has also been discussed for some years as a cause of the limited ability to interpret the emotional content of the facial expressions of others. Thus, limitations in the interpretation of the emotional content of others’ facial expressions is also found in myasthenia gravis ⁴⁷ , myotonic dystrophy ⁴⁸ , Duchenne muscle dystrophy ⁴⁹ , and amyotrophic lateral sclerosis ⁵⁰ . Although all these diseases primarily affect the patients’ motor function, they also have central effects. Interestingly, however, changes in the interpretation of emotional facial expressions of others can also be observed in purely peripheral disorders such as peripheral facial nerve palsy ^{51 52} , supporting the theory that limitations in one’s own purely motor ability to express emotions facially have a negative effect on the ability to interpret emotions in the facial expressions of others. This theory is supported by the findings that, for example, in amyotrophic lateral sclerosis, mimic emotion detection is reduced even when other cognitive abilities are still preserved ⁵⁰ . Also, Parkinson patients as well as patients with Huntington’s disease show a direct correlation between the ability to express themselves mimically and the ability to interpret the emotions of others ^{53 54 55} . In summary, the main common feature of peripheral facial nerve palsy with all these described neurodegenerative diseases is the reduction of the ability to control facial expressions motorically. Nevertheless, all diseases show common effects on emotion recognition and cerebral processing of emotions. This suggests that even in the presence of possible alternative causes for a reduced emotion recognition, the purely motor disorder may impair it. These findings also align well with the currently dominant theory of embodied recognition. This theory states that an important mechanism for perceiving facial emotions is their sensorimotor simulation in one’s own brain. This requires a mental replication of the perceived facial expression. However, since one’s own facial expression is disturbed, also the mental replicability is altered, which ultimately limits the ability to simulate and thus recognize facial emotions ^{56 57} .

However, in addition to these peripheral and/or central disorders described above, there are also many central disorders that interfere with emotion detection of others’ facial expressions. For example, deficits are found in patients with schizophrenia ⁵⁸ , chronic alcoholism ⁵⁹ , borderline personality disorder ⁶⁰ , and multiple sclerosis ^{61 62 63} .

6. Facial expression and emotions in the era of masks

Wearing face masks has become a commonplace in the era of the COVID pandemic. Face masks represent the most cost-effective protective measure against the SARS-CoV2 virus and allow continued personal social contact ⁶⁴ . Nevertheless, considerable resistance to the wearing of masks was shown in many countries. The main reason discussed is the feeling of restriction of personal freedom ^{65 66} . The fact that wearing a face mask is perceived as such a strong restriction of personal freedom despite the considerable advantages seems to be related to the perceived strong restriction of physical and psychological freedom by the face mask. Most probably, a piece of textile that would have to be worn as a bandage on the upper arm, for example, would not have been associated with the same feeling of restriction of personal freedom. Wearing a face mask is certainly an inconvenience and can also lead to an increase in headaches ⁶⁷ or acne ⁶⁸ in some individuals. However, these side effects are rare. It is much more likely that the feeling of restriction of freedom arises from the fact that the mask covers most of the very regions of the body that makes us communicative and social creatures. However, our acceptance of wearing the mask again affects how we are perceived by others. Thus negative feelings when wearing the mask affect also the social perception by other mask users ⁶⁹ . Wearing masks naturally also disturbs the recognition of emotions from the facial expressions of our fellow human beings and makes social interactions more difficult ⁷⁰ . Since the mimic emotion recognition is not composed of the sum of facial subareas, but also involves as strong holistic process, wearing masks worsens this ability more than would be expected from the lack of information from the covered facial areas ⁷¹ . It is further argued that the resulting deficits in nonverbal communication give people a feeling of insecurity and discouragement and can also promote the emergence of psychological disorders if there is a corresponding predisposition ⁷² .

The effects described above are currently being reduced by the increasing abolition of the obligation to wear face masks. However, it is still obligatory in the health care system and thus a particular challenge for physicians regarding empathy and conversation skills. It is crucial that both communication partners are aware of the resulting perception deficits and that these deficits are compensated by changing and, above all, increasing bidirectional communication behavior.

7. Facial expressions and emotions in other species

In many species, the expression of emotion takes place on biochemical and physiological levels similar to human responses. For example, responses to fear are very similar with accelerated respiration, change in blood distribution, hormone release, and pupil dilation. In facial expressions, there are significantly stronger differences between species, which are due to the different number of muscle groups in the face and the ability to control them.

Already Darwin stated that animals can develop specific facial expressions adapted to their species ⁷³ . The “Facial Action Coding System” developed by Ekman and Friesen for the anatomy-based classification of emotions in humans ⁷⁴ could also be successfully used in seven other species including chimpanzees ^{75 76 77} . Even in rodents, mimic expressions with at least an affective content could be detected ^{78 79} .

However, it is undisputed that the possibilities of facial expression are reduced with a lower level of development ⁷⁵ . In this context, the question of how much self-awareness is necessary to express emotions mimically and to recognize them in others remains of great interest.

8. Cerebral processing of mimic emotion perception

Functional imaging studies show that the processing of emotional faces is associated with increased activity in visual, limbic, temporoparietal, cerebellar, as well as prefrontal areas and in the putamen ⁸⁰ ( Fig. 1 ). Visual information first reaches the visual cortex. Here, processing of emotional faces occurs in several subareas (inferior occipital gyrus, lingual gyrus, fusiform gyrus). Processing in these regions appears to be deep in the processing hierarchy and independent of the emotion processed or its valence ^{81 82} . The cerebellum is also involved in emotion processing. Although the cerebellar areas have strong anatomical connections to the cortical association cortex and various limbic structures ⁸³ , they, like the visual areas, appear to be independent of the emotion itself being processed ^{84 85 86} .

Fig. 1.

Regions of increased neuronal activity in response to happy, sad, angry, fearful, and disgusted human faces compared with neutral faces (FG fusiform gyrus, ACC anterior cingulate cortex, LG lingual gyrus, IOG inferior occipital gyrus, MFG medial frontal gyrus) ⁸⁰ .

Fearful faces activate the medial frontal cortex, which plays a key role in conscious experience and the regulation of emotion experience.

The amygdala is activated when looking at sad, fearful, or even happy faces, whereas angry or disgusted faces do not cause a change in amygdala activity. For the latter two emotions, particularly strong activation was found in the insular region ⁸⁰ that seems to respond more strongly to disgusted faces than to angry ones. Limbic regions such as the amygdala and hippocampus are thought to be more involved in processing external positive emotional sensory information, while the insular region is activated primarily for interoceptive stimuli and distressing external stimuli ⁸⁷ . The cingulate cortex is particularly activated by the perception of happy faces ⁸⁷ .

This distribution makes clear that the processing of emotional contents of faces occurs in a complex network of brain regions. Different emotions involve different brain regions to different degrees, but many brain regions are involved in the processing of most emotions.

However, the cerebral activation pattern in the processing of emotional faces seems to depend not only on the emotion presented but also on the attention paid to it ^{88 89 90} . In general, there is consensus that face processing shows right-brain dominance. This is a long-standing finding for general processing of faces for identity recognition, emotion, and attention ^{88 89 90} . This is also congruent to the finding that in patients with parietal lesions, right-sided lesions lead to stronger neglect symptoms compared to left-sided ones ⁹¹ . This right-sided dominance for general processing of faces also seems to refer to the processing of emotional faces. Thus, a bilateral presentation of fearful faces found greater activation and connectivity of right visual cortex and right amygdala compared with neutral faces ⁹² . Even when little attention is paid to faces, right lateralized increased activity is found in the amygdala and temporal pole ⁹³ .

The processing of emotional faces described above takes place explicitly, i.e. with an awareness of the information. This is in contrast to the implicit, i.e. unconscious, processing of emotional information from faces. Experimentally, this unconscious perception can be generated, e.g., by temporally retroactive masking with other stimuli or by interocular suppression. This implicit processing is particularly interesting because it was shown early that patients with structural lesions of the primary visual cortex could discriminate specific characteristics of these stimuli ⁹⁴ . Subsequent studies revealed that this ability relates particularly to the perception of emotional faces ⁹⁵ – which is a phenomenon that has been called affective blindness. Affected patients show the ability to correctly identify emotions from faces with a disproportionately high frequency, although they cannot see them ^{96 97} . They show increased activity in the left amygdala, left striatum, and right middle temporal gyrus ⁹³ . Processing of unconsciously perceived faces is also dependent on expressed emotions. In the processing of unconsciously perceived faces with positive emotions, the temporal and parietal cortex are more involved than in the unconscious processing of neutral faces. In contrast, unconsciously perceived facial expressions with negative emotional content lead to stronger activities in subcortical regions ⁹³ .

Overall, the parallel existence of a subcortical processing pathway to the right amygdala via the midbrain and thalamus has been proposed for the processing of emotional faces, while a parallel cortical pathway exists for conscious face perception ^{93 98 99 100} . However, this view is controversial, as some authors argue that the subconscious processing of emotional faces is also cortical ^{101 102 103} . The existence of a subcortical parallel pathway would also explain the fact that basal emotions in faces can be perceived in prefrontal areas at the speed of up to 120 ms ¹⁰⁴ , which is surprisingly fast considering that motion-independent foveal information reaches the occipital cortex only after about 60 ms and the complex compound P100 component is measurable only after 100 ms. Despite this considerable speed, however, a fast cortical pathway cannot be ruled out here either. In addition to these fast components, however, significantly later components are also measurable in the prefrontal cortex. It appears that the interpretation of emotions occurs in a temporally staggered process. The prefrontal cortex is also of particular importance in this context because it modulates visual information processing in other brain regions ¹⁰⁴ . This iterative information processing seems particularly plausible because it shows strong similarities to the parallel organization in other sensory systems ^{105 106} . The existence of parallel very fast responses also indicates the considerable evolutionary importance of emotion recognition, since this additive processing pathway must have provided an evolutionary advantage due to its speed.

9. Interpretation of emotions from facial expressions in their meaning in society

It is generally agreed that the emotions in faces we look at have an effect on us. Although we would not cognitively classify the nice smile of the hotel employee at the counter as a sign of joy about our visit, we mostly perceive it as pleasant or at least the absence of a smile as unpleasant. In the legal systems of Western democracies, the interpretation of emotions by judges is also seen as influencing the sentence. This fact presupposes the self-conception that we believe we can correctly classify emotions. Thus, the display of grief, both in facial expressions and in words, often has the effect of reducing the sentence. In this regard, many people consider facial expressions to be more decisive under the assumption that they allow a stronger insight into emotions than words of regret read aloud. For example, a US Supreme Court Justice (Anthony Kennedy) wrote in 1992 that reading the defendant’s emotions is necessary to “know the heart and mind of the offender”.

Even if the interpretation of emotions and their assessment on sentencing is considered fair in our legal system, the same is not true for emotion interpretation in other areas of society. The American psychologist Paul Ekman developed and implemented a training program for the US Transportation Security Administration in 2007 that trained staff to observe passengers for suspicious signs of stress, deception, or anxiety. The program is based on Paul Ekman’s research from the late 1970s, during which Ekman published several articles on identifying emotions from minimal changes in facial expression ¹⁰⁷ . It is further based on the assumption that the attempt to hide or lie about emotions leads to the leakage of emotions in the form of microexpressions in facial expressions.

However, the considerable inaccuracy and racial bias led to important criticism by scientists, members of the US Congress, and various human rights organizations ¹⁰⁸ . Leaving aside ethical and moral aspects, this program is highly interesting from a purely scientific perspective, as it explores the hypothesis that it is possible to determine from the mere observation of people whether they are cheating or planning hostile acts. An idea of the size of the program comes from the number of people involved. In 2015, 2,800 people were employed in this so-called SPOT program and between 2007 and 2013, 900 million US dollars were invested. Of course, no academic-driven research can reach this scale. The enormous data size and the various technology deployments make this program particularly interesting.

The aforementioned hypothesis has found many supporters, and thus personnel have been hired at London Heathrow Airport for behavioral observation, and the US Department of Homeland Security is also pursuing a program for electronic assessment of nonverbal behavior. In this context, the study of facial expressions is a particularly important aspect. Critics argue that there is no scientific evidence to support the assumption that conclusions about intent or future behavior can be drawn from observations ¹⁰⁹ . This assessment was shared by a review commission of the program by the US Congress. In response to these concerns, the Transportation Security Administration commissioned an independent study, the results of which are currently pending.

Automated analysis and categorization of emotions from faces has been intensively researched over the last 2 decades. Current algorithms are able to identify emotions with more than 90% correctness from static pictures. In the last 10 years, the analysis of pictures has taken a back seat in research, while increasing research has been done on the analysis of emotions from video sequences. Here, too, significant success has been achieved in the automated analysis and categorization of emotions from faces due to the considerable increase in computer power and, above all, to advances in the research field of computer vision ^{110 111 112} .

The analysis of videos is much more complex than the analysis of static images. When analyzing videos, it is not a matter of analyzing each individual picture, but rather these emotion analyses are dependent on the characteristics of the face in the past and in the future. In particular, unlike static images, voluntary and involuntary facial movements can be distinguished in videos. It is precisely the “leakage” of genuine emotion through a feigned emotion that can be analyzed. And it is precisely this interplay of voluntary and involuntary movements and the associated detection of microexpressions that are the subject of very active research ¹¹³ .

The “leakage” of genuine emotions through an acted facial expression lasts for a very short time and is often difficult to detect by the human eye. The duration of microexpressions is reported to be in the range of 40–200 ms ^{30 114} . Another property of microexpressions that makes them difficult to detect by the human eye is that they are often fragmented. Only part of the muscle groups expressing a particular emotion are activated during these microexpressions ¹¹⁵ .

These characteristics of microexpressions make them an ideal application area for computational image analysis. There are many applications, e.g., in business negotiations, in psychiatric interviews, and in forensics ^{108 116} . A first approach was developed by Paul Ekman in 2002. He trained people with the help of a tool to detect 7 different categories of microexpressions ¹¹⁷ . The results were however sobering, since it could be shown that students, who worked with this tool in the recognition of microexpressions, were inferior to the staff of the US Coast Guard that worked without assistance ¹¹⁸ .

To date, the development of analysis methods has been significantly limited by the paucity of available datasets. To create datasets with the richest possible microexpressions, subjects are encouraged to watch videos with emotional content and are simultaneously offered rewards if they do not show emotions. However, labeling these data remains difficult and results in currently only a few existing freely accessible datasets of limited size ^{113 119} .

Additionally, analyses are complicated by head movements of the recorded person but also by facial movements without emotional reference (e.g. blinking). Besides the interpretation of the emotional content of a microexpression, especially the identification of the phases of occurrence of this microexpression represents a considerable problem ¹²⁰ . For these partial problems, there has been an increasing number of investigations and a significant improvement in the quality of results, especially in the last 12 years. If the number of emotions to be recognized is limited (e.g., to positive, negative, surprised, and others), results with an accuracy of around 70% can be obtained ^{121 122 123} .

On single databases (no performance across databases), current results show an accuracy of 65% to more than 70% in the recognition, if the characteristics that should be recognized are limited (e.g., positive, negative, surprised, and others). Current methods also focus on the application of so-called convolutional neural networks, which can achieve an accuracy of over 70% with different auxiliary methods ¹²⁴ and this partly even across different datasets ¹²⁵ . Despite this significant progress, there is still potential for improvement in emotion detection from videos and also in the detection and interpretation of microexpressions. The current quality of results clearly limits its use in practice.

10. Conclusion

The interplay of facial expressions and emotions is an important part of daily social interaction. Scientific research over the past 50 years has yielded a great increase in knowledge. In particular, deeper insights into evolutionary origins and behavioral properties of facial expression and emotion have been gained. Cerebral processing has also been intensively researched and is now largely anatomically assignable. However, functional models of the recognition and expression of emotion in facial expressions, which would significantly advance our understanding of cerebral processing mechanisms, are still lacking. In the coming years, the recognition of emotions in videos and in particular the recognition of microexpressions will be a major scientific and, increasingly, economic challenge.