Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion

Abstract

The emergence of artificial emotional intelligence technology is revolutionizing the fields of computers and robotics, allowing for a new level of communication and understanding of human behavior that was once thought impossible. While recent advancements in deep learning have transformed the field of computer vision, automated understanding of evoked or expressed emotions in visual media remains in its infancy. This foundering stems from the absence of a universally accepted definition of “emotion,” coupled with the inherently subjective nature of emotions and their intricate nuances. In this article, we provide a comprehensive, multidisciplinary overview of the field of emotion analysis in visual media, drawing on insights from psychology, engineering, and the arts. We begin by exploring the psychological foundations of emotion and the computational principles that underpin the understanding of emotions from images and videos. We then review the latest research and systems within the field, accentuating the most promising approaches. We also discuss the current technological challenges and limitations of emotion analysis, underscoring the necessity for continued investigation and innovation. We contend that this represents a “Holy Grail” research problem in computing and delineate pivotal directions for future inquiry. Finally, we examine the ethical ramifications of emotion-understanding technologies and contemplate their potential societal impacts. Overall, this article endeavors to equip readers with a deeper understanding of the domain of emotion analysis in visual media and to inspire further research and development in this captivating and rapidly evolving field.

I. INTRODUCTION

As artificial intelligence (AI) technology becomes more prevalent and capable of performing a wide range of tasks, the need for effective communication between humans and AI systems is becoming increasingly important. The adoption of smart home products and services is projected to reach 400 million worldwide, with smart devices, such as Alexa and Astro, becoming increasingly common in households [1]. However, these devices are currently limited to executing specific commands and do not possess the capability to understand or respond to human emotions [2]. This lack of emotional intelligence (EQ) limits their potential applications, and this constraint is particularly relevant for future robotic applications, such as personal assistant robots, social robots, service robots, factory/warehouse robots, and police robots, which require close collaboration and a comprehensive understanding of human behavior and emotions.

The ability to impart EQ to AI when dealing with visual information is a topic of growing interest. This article aims to address the fundamental question of how to “teach” AI to understand and respond to human emotions based on images and videos. The potential technical solutions to these questions have far-reaching implications for various application domains, including human–AI interaction, autonomous driving, social media, entertainment, information management and retrieval, design, industrial safety, and education.

To provide a comprehensive and well-balanced view of this complex subject, it is essential to draw on the expertise of various fields, including computer and information science and engineering, psychology, data science, movement analysis, and performing arts. The interdisciplinary nature of this topic highlights the need for collaboration and cooperation among researchers from different fields in order to achieve a deeper understanding of the subject.

In this article, we focus on the topic of affective visual information analysis as it represents a highly nuanced and complex area of study with strong connections to well-established scholarly fields, such as computer vision, multimedia, and image and video processing. However, it is important to note that the techniques presented here can be integrated with other data modalities, such as speech, sensor-generated streaming data, and text, in order to enhance the performance of real-world applications.

The primary objective of this article is to introduce the technical communities to the emerging field of affective visual information analysis. Recognizing the breadth and dynamic nature of this field, we do not aim to provide a comprehensive survey of all subareas. Instead, our discussion focuses on the fundamental psychological and computational principles (see Sections II and III), recent advancements and developments (see Section IV), core challenges and open issues (see Section V), connections to other areas of research and development (see Section VI), and ethical considerations related to this new technology (see Section VII). We apologize in advance for any important publications that may have been omitted in our discussion.

Recently, there have been some other surveys and reviews on artificial EQ (AEI), such as facial expression recognition (FER) [3], [4], [5], [6], microexpression recognition (MER) [7], [8], [9], textual sentiment classification [10], [11], [12], music and speech emotion recognition [13], [14], [15], affective image content analysis [16], emotional body gesture recognition [17], bodily expressed emotion recognition [18], emotion recognition from physiological signals [19], [20], multimodal emotion recognition [21], [22], and affective theory use [23]. These articles mainly focus on emotion and sentiment analysis for a specific modality from the perspective of machine learning and pattern recognition or focus on the psychological emotion theories. Cambria [24] summarized the common tasks of affective computing and sentiment analysis, and classified existing methods into three main categories: knowledge-based, statistical, and hybrid approaches. Poria et al. [25] and Wang et al. [26] reviewed both unimodal and multimodal emotion recognition before the year of 2017 and between the years of 2017 and 2020, respectively. As opposed to those reviews, this article aims to provide a comprehensive overview of emotion analysis from visual media (e.g., both images and videos) with insights drawn from multiple disciplines.

II. EMOTION: THE PSYCHOLOGICAL FOUNDATION

How we define emotion largely descends from the theoretical framework used to study it. In this section, we provide an overview of the most prominent emotion theories, beginning with Darwin, and underscore how contemporary dimensional approaches to understanding emotion align with both the processing of emotion by the human brain and current computer vision approaches for modeling emotion to make predictions about human perception (see Section II-A). In addition, we examine the intrinsic link between emotion and adaptive behavior, a contention that is largely shared across different emotion theories (see Section II-B).

A. Definitions and Models of Emotion

One of the first emotion theories put forth was Charles Darwin’s in his seminal book “On the Expression of the Emotions in Man and Animals” [32]. This book proposed that humans possess a finite set of biologically privileged emotions that evolved to confer upon us survival-related behavior. William James [33] later added to this arguing that the experience of emotion is ultimately our experience of distinct patterns of physiological arousal and physical behaviors associated with each emotion [34]. Building upon these assumptions, Ekman’s Neurocultural Theory of Emotion [35] further perpetuated the notion that there exists a “universal affect program” that underlies the experience and expression of several discrete emotions, such as anger, fear, sadness, and happiness. According to this theory, basic emotional experiences and emotional displays evolved as adaptive responses to specific environmental contingencies and, thus, felt, expressed, and recognized emotions are uniform across all people and cultures, and are marked by specific patterns of physiological and neural responsivity.

Considerable research has since questioned the utility of this approach. This includes findings that people: 1) are often ill-equipped to describe their own emotions in discrete emotion terms, both in research and clinical settings [36]; 2) show low consensus in their ability to categorize both facial and vocal expressions of emotions in discrete emotion terms [37]; and 3) show high intercorrelations across the emotional experiences they do report [38]. Such findings have prompted many researchers to explore alternative approaches to conceptualizing and measuring emotional experience that necessarily involve a cognitive component.

Magda Arnold’s cognitive appraisal theory of emotion was the first to introduce the necessity of cognition in emotion elicitation [39]. Although she did not disagree with Darwin and James that emotions are adaptive states spurring on survival-related behavior, she, nonetheless, took them to task for not considering vast individual variation in emotional experiences. Arnold rightly underscored the capacity for the same emotion-evoking events to lead to different emotional experiences in different people. This becomes readily apparent when considering one’s own emotional experiences. For example, diving off a cliff may generate an aversive fear state in one person but an enjoyable thrill state in another. The difference in how an event is evaluated, therefore, shapes the emotion that results. Central to her theory was the importance of cognitive appraisal in initially eliciting an emotion. Once elicited, she largely agreed with Darwin’s functional assumptions regarding the survival benefits of emotion-related behavior. Later, Schachter and Singer [40] drew on these insights to help resolve ongoing debates regarding James’ theory of embodied emotional experience. Their research demonstrated that the emotion we experience when adrenaline is released in our body depends on our cognitive framing and context. Those injected with adrenaline reported feeling happier when in a fun context and more irritated when in an angering context. The only difference was the cognitive appraisal that framed the experience of that arousal.

Building upon these ideas further, Mehrabian and Russell [41] proposed the Pleasure, Arousal, and Dominance (PAD) Emotional State Model, which suggests a dimensional account of emotion, one in which PAD constitutes the fundamental dimensions that represent all emotions. Later, Russell dropped dominance as a key dimension and focused on what he refers to as “core affect,” suggesting that all emotions can be reduced to the fundamental psychological and biological dimensions of pleasantness and arousal [42]. Dimensional approaches offer a way of conceptualizing and assessing emotion that closely approximates how the human brain processes emotion (see Fig. 1 for a comparison of different dimensional/circumplex models).

Fig. 1.

Five influential circumplex emotion models developed in the psychology literature [27], [28], [29], [30], [31]. (a) Russell, 1980. (b) Watson and Tellegen, 1985. (c) Thayer, 1989. (d) Larsen and Diener, 1992. (e) Knutson, 1996.

Notably, some early attempts to use computational methods to predict human emotions elicited by visual scenes employed the discrete emotion approach described at the outset of this section. For example, Mikels et al. [43] and Machajdik and Hanbury [44] used categorical approaches to assess the visual properties of stimuli taken from the International Affective Picture System (IAPS) [45], a widely used set of emotionally evocative photographs in the emotion literature. However, such approaches resulted in high levels of multicollinearity between emotions, making it difficult to disentangle emotions using traditional regression models. In contrast, adopting a dimensional approach not only aligns well with emerging theoretical accounts of emotion but has been validated by James Wang and his colleagues in the successful assessment of human aesthetics and emotions evoked by visual scenes, as well as bodily expressed emotion [46], [47], [48], [49], [50], [51], [52]. This offers a methodological approach that is consistent with dimensional theories of emotion.

B. Interplay Between Emotion and Behavior

Fridlund’s behavioral ecology perspective of emotion argues that emotional expression evolved primarily as a means of signaling behavioral intent [53]. The view that facial expression evolved specifically as a way to forecast behavioral intentions and consequences to others drew from Darwin’s seminal writings on expression [32] even though Darwin himself argued that expressions did not evolve for social communication per se. Fridlund’s argument is based on the idea that perceiving behavioral intentions is adaptive. From this perspective, anger may primarily convey to an observer a readiness to attack, whereas fear may primarily convey a readiness to submit or retreat (see [54]). From this perspective, behavioral intentions are considered “syndromes of correlated components” [53, p. 151]. Fridlund is not alone in these assumptions. Some researchers have gone so far as to suggest that feeling states associated with emotions are merely conscious perceptions of underlying behavioral intentions or action tendencies, which implies that emotional feeling is simply the experience of behavioral intention, similar to William James’s theory [55].

It is worth noting that empirical research has provided support for the idea that behavioral intention is conveyed through emotional expression. For example, one study demonstrated that action tendencies and emotion labels are attributed to faces at comparable levels of consistency [55]. Similarly, in forced-choice paradigms [54], cross-cultural evidence indicates that participants assign behavioral intention descriptors with about equal consistency as they do with emotion descriptors.

A focus on approach-avoidance tendencies has been highlighted in most of the research conducted to date. The ability to detect another’s intention to approach or avoid us is thought of as a principal factor governing social exchange. However, much of the work on approach-avoidance behavioral motivations has also tended to concentrate on the experience or response of an observer to a stimulus event [56]. One common method of operationalizing approach and avoidance then stems from traditional behavioral learning paradigms that link behavioral motivation and emotion through reward versus punishment contingencies [57]. Approach motivation is defined by appetitive, reward-related behavior, while avoidance motivation is defined by aversive, punishment-related behavior, where appetitive behavior is movement toward a reward, and aversive behavior is movement away from a punishment.

Much research has focused on the relationship between approach and avoidance tendencies and emotional experience [58]. However, there has been less attention paid to whether approach and avoidance tendencies are fundamentally signaled by the external expression of emotion. It stands to reason that, if the experience of emotion is associated with approach and avoidance tendencies, these tendencies should be signaled to others when expressed. This distinction is important as the approach-avoidance tendencies attributed to expressive faces may not always match the approach-avoidance reactions elicited by them. For example, the expression of joy arguably conveys a heightened likelihood of approach by the expressor and a reaction of approach from the observer [56]. In contrast, anger expressions signal approach by the expressor but tend to elicit avoidance by the observer.

Recent insights from the embodiment literature also provide evidence that emotional experiences are grounded in specific action tendencies [34]. This means that emotional experiences can be expressed through stored action tendencies in the body, rather than through semantic cues. For example, studies have examined the coherence between emotional experience (positive or negative) and approach (arm flexion, i.e., pulling toward) versus avoidance behavior (arm extension, i.e., pushing away) [59]. In one study, participants were randomly assigned to an arm flexion (approach behavior) or arm extension (avoidance behavior) condition, either during the reading of a story about a fictional character or during a positive versus negative semantic priming task before reading the story. Participants in the congruent conditions (happy prime and arm flexion, and sad prime and arm extension) were able to remember more items from the story.

Although important for explaining behavioral responding, these studies did not address whether basic tendencies to approach or avoid were also fundamentally signaled by emotional expressions. If they were, expressions coupled with approach and avoidant behaviors should impact the efficiency of emotion recognition. In one set of studies, anger expressions were found to facilitate judgments of approach versus withdrawing faces compared with fear expressions [60]. Similarly, perceived movement of a face toward or away from an observer likewise facilitated angry or fearful expression perception [61]. Thus, approach and avoidance movement are associated in a fundamental way with the recognition of anger and fear displays, respectively, supporting the conclusion that basic action tendencies are inherently associated with the perception of emotion.

In sum, despite widely debatable assumptions about the nature of emotion and emotional expression across various theories, most tend to agree that emotion expression conveys fundamental information regarding basic behavioral tendencies [60].

III. EMOTION : COMPUTATIONAL PRINCIPLES AND FOUNDATIONS

In this section, we aim to establish computational foundations for analyzing and recognizing emotions from visual media. Emotion recognition systems typically involve several fundamental data-related stages, including data collection (see Section III-A), data reliability assessment (see Section III-B), and data representation (see Sections III-D–III-F for general computer vision-based representation, movement coding, and context and function purpose detection, respectively). As we present specific examples at each stage, we will emphasize the underlying principles that they adhere to. We provide a list of representative datasets in Section III-C. We will also introduce the factors of acted portrayals (see Section III-G), cultural and gender dialects (see Section III-H), structure (see Section III-I), personality (see Section III-J), and affective style (see Section III-K) in inferring emotion, based on prior research.

A. Data Collection

Because the categories of emotions are not well-defined, it is not possible to program a computer to recognize all emotion categories based on a set of predefined logic rules, computational instructions, or procedures. Thus, researchers must take a data-driven approach in which computers learn from a large quantity of labeled, partially labeled, and/or unlabeled examples. To enable such research and subsequent real-world applications, it is essential to collect large-scale, high-quality, ecologically valid datasets. To highlight the complexity of the data collection problem and to introduce best practices, we describe a few data collection approaches that incorporate psychological principles in their design.

1). Evoked Emotion—Immediate Response:

In the field of modeling evoked emotion, earlier researchers utilized the IAPS dataset, which consisted of only 1082 images rated for evoked emotional response [46]. In 2017, Lu et al. [48] introduced one of the first large-scale datasets, the EmoSet, utilizing a human subject study. The EmoSet dataset is much larger, and all images are complex scenes that humans regularly encounter in daily life.

To create a diverse image collection, the researchers employed a data-crawling approach to gather nearly 44 000 images from social media and obtained emotion labels (both dimensional and categorical) using crowdsourcing via the Amazon Mechanical Turk (AMT) platform. They used the valence, arousal, and dominance (VAD) dimensional model [27], which is similar to the PAD model. The researchers followed strict psychological subject study procedures and validation approaches. The images were collected from more than 1000 users’ Web albums on Flickr using 558 emotional words as search terms. These words were summarized by Averill [62].

The researchers carefully designed their online crowdsourcing human subject study to ensure the quality of the data. For example, each image was presented to a subject for exactly 6 s. This design differed from conventional object recognition data annotation tasks, where the subject was often given no restrictions on the amount of time to view an image. This design followed psychological convention, as the intention was to collect the subject’s immediate affective response to the visual stimuli. If subjects were given varying amounts of time to view an image before rating it, the data would not be a reliable capture of their immediate affective response. To accommodate for this, subjects were given the option to click a “Reshow Image” button if they needed to refer back to the image. In addition, recognizing that categorical emotions may not cover all feelings, this method allowed the subject to enter other feelings that they may have had.

2). Evoked Emotion—Test–Retest Reliability:

The data collection method proposed by Lu et al. [48] aimed to understand immediate affective responses to visual content, but it did not ensure retest reliability of affective picture stimuli over time and across a population. Many psychological studies, from behavioral to neuroimaging studies, have used visual stimuli that consistently elicited specific emotions in human subject. While the IAPS and other pictorial datasets have validated their data, they have not examined the retest reliability or agreement over time of their picture stimuli.

To address this issue, Kim et al. [50] developed the Image Stimuli for Emotion Elicitation (ISEE) as the first set of stimuli for which there was an unbiased initial selection method and with images specifically selected for high retest correlation coefficients and high within-person agreement across time. The ISEE dataset used a subset of 10 696 images from the Flickr-crawled EmoSet. In the initial screening study, study participants rated stimuli twice for emotion elicitation across a one-week interval, resulting in the selection of 1620 images based on the number of ratings and retest reliability of each picture. Using this set of stimuli, a second phase of the study was conducted, again having participants rate images twice with a one-week interval, in which the researchers found a total of 158 unique images that elicited various levels of emotionality with both good reliability and good agreement over time. Fig. 2 shows 18 example images in the ISEE dataset.

Fig. 2.

Example images in the ISEE dataset [50]. Images were selected after a thorough test–retest reliability study.

3). Expressed Emotion—Body:

In the field of expressed emotion recognition, the collection of data on bodily expressed emotions has received less attention compared to the more widely studied areas of facial expression and microexpression data collection. In addition, whereas earlier studies often relied on data collected in controlled laboratory environments, recent advancements in technology have made it possible to collect data in more naturalistic, real-world settings. These “in-the-wild” datasets are more challenging to collect, but they offer the opportunity to capture a more diverse range of emotions and expressions. Whereas laboratory environments provide the advantage of advanced sensors, such as Motion Capture (MoCap), body temperature, and brain electroencephalogram (EEG) for collecting data, and it is possible to capture self-identified rather than perceived emotional expression, it is impossible to accurately replicate the vast array of diverse real-world scenarios within a controlled laboratory setting.

Using video clips from movies, TV shows, sporting, and wedding events as a source of data for emotion recognition has several advantages. These videos provide a wide range of scenarios, environments, and situations that can be used to train computer systems to understand human behavior, expression, and movement. For instance, these videos have recorded scenes during natural and man-made disasters, providing valuable information for understanding human emotions under extreme conditions. In addition, a large proportion of video shots in movies is of outdoor human activities, providing a diverse range of contexts for training.

However, it is important to note that using publicly available video clips as a source of data has its limitations. One such limitation is that this approach can only capture perceived emotions, as opposed to self-identified emotions. In many applications, perceived emotions are a sufficient proxy for actual emotions, particularly when the goal is for robots to “perceive” or “express” emotions in a way that is similar to humans for efficient communication with humans. A further constraint is that the videos mainly feature staged or user-selected scenes, rather than depicting natural everyday interactions. This topic will be further explored in Section III-G.

Luo et al. [51] developed the first dataset for bodily expressed emotion understanding (BEEU), named the Body Language Dataset (BoLD), using this approach. The data collection pipeline is illustrated in Fig. 3. The researchers collected hundreds of movies from the Internet and cut them into short clips. An identified character with landmark tracking in a single clip is called an instance. They used the AMT platform for crowdsourcing emotion annotations of a total of over 48 000 instances. The emotion annotation included the VAD dimensional model [27] and 26 emotion categories [63].

Fig. 3.

Data collection pipeline was developed by Luo et al. [51] to utilize crowdsourcing to annotate vast amounts of videos available on the Internet. The pipeline involved obtaining raw movies and videos from the Internet, segmenting them into short clips, processing the clips using computer vision techniques to extract posture information and track individual characters, and utilizing crowdsourcing and expert annotation to assign emotion and movement labels to the clips.

B. Data Quality Assurance

Quality control is a crucial aspect for crowdsourcing, particularly for affect annotations. Different individuals may have varying perceptions of affect, and their understanding can be influenced by factors such as cultural background, current mood, gender, and personal experiences. Even an honest participant may provide uninformative affect annotations, leading to poor-quality data. In this case, the variance in acquiring affect usually comes from two kinds of participants, i.e., dishonest ones, who give useless annotations for economic motivation, and exotic ones, who give inconsistent annotations compared with others. The existence of exotic participants is inherent in emotion studies. The annotations provided by an exotic participant could be valuable when aggregating the final ground truth or investigating cultural or gender effects of affect. However, we typically want to reduce the risk of high variance caused by dishonest and exotic participants in order to collect generalizable annotations.

In the case of the BoLD dataset [51], five complementary mechanisms were used, including three online approaches (i.e., analyzing while collecting the data) and two offline (i.e., postcollection analysis), based on a recent technological breakthrough for crowdsourced affective data collection [49]. These mechanisms were participant EQ screening [64], annotation sanity/consistency check [51], gold standard test based on control instances [51], and probabilistic multigraph modeling for reliability analysis [49].

Particularly critical is the probabilistic graphical model Ye et al. developed to jointly model subjective reliability, which is independent of supplied questions, and regularity [49]. For brevity of discussion, we focus on using the mode(s) of the posterior as point estimates. We assumed that each subject $i$ had a reliability parameter ${\tau }_i\in [0,1]$ and regularity parameters ${\alpha }_i$, ${\beta }_i>0$ characterizing their agreement behavior with the population, for $i=1,\dots ,m$. We also used the parameter $\gamma$ for the rate of agreement between subjects by pure chance. Let $\text{Θ}=\left({\left\{{\tau }_i,{\alpha }_i,{\beta }_i\right\}}_{i=1}^m,\gamma \right)$ be the set of parameters. Let ${\text{Ω}}_k$ be a random subsample from subjects $\{1,\dots ,m\}$ who labeled the stimulus $k$, where $k=1,\dots ,n$. We also assumed that sets ${\text{Ω}}_k$’s were created independently of each other. For each image $k$, every subject paired from ${\text{Ω}}_k^2$, i.e., ($i$, $j$) with $i\ne j$, had a binary indicator $I_{i,j}^{(k)}\in \{0,1\}$ coding whether their opinions agreed on the respective stimulus. We assumed that $I_{i,j}^{(k)}$ was generated from a probabilistic process involving two latent variables. The first latent variable $T_j^{(k)}$ indicated whether subject $O_j$ was reliable or not. Given that it was binary, a natural choice of model was the Bernoulli distribution. The second latent variable $J_i^{(k)}$, lying between 0 and 1, measured the extent to which subject $O_i$ agreed with other reliable responses. We used the beta distribution parameterized by ${\alpha }_i$ and ${\beta }_i$ to model $J_i^{(k)}$ because it was a widely used and flexible parametric distribution for quantities on the interval [0, 1].

In a nutshell, $T_j^{(k)}$ is a latent switch (a.k.a. gate) that controls whether $I_{i,j}^{(k)}$ can be used for the posterior inference of the latent variable $J_i^{(k)}$. Hence, the researchers referred to the model as the gated latent beta allocation (GLBA). A graphical illustration of the model is shown in Fig. 4. If an uninformative annotator was in the subject pool, their reliability parameter ${\tau }_i$ was zero though others could still agree with their answers by chance at a rate of $\gamma$. On the other hand, if an individual was very reliable yet often provided controversial answers, their reliability ${\tau }_i$ could be one, while they typically disagreed with others, as indicated by their high irregularity

$$\mathbb{E}\left[J_i^{(k)}\right]=\frac{{\alpha }_i}{{\alpha }_i+{\beta }_i}\approx 0.$$

We were interested in finding both types of participants. Most participants were between these two extremes. The quantitative characterization of participants by GLBA will assist in selecting subsets of the data collection for quality control or gaining a comprehensive understanding of subjectivity. For more details, please refer to [49], [65].

Fig. 4.

Probabilistic graphical model, GLBA, was developed by Ye et al. [49] to model the subjective reliability and regularity in a crowdsourced affective data collection. The work enables researchers in affective computing to effectively identify and exclude highly subjective annotation data provided by uninformative human participants, thereby improving the overall quality of the collected data.

A recent study [66] presented a Python-based software program called MuSe-Toolbox, which combines emotion annotations from multiple individuals. The software includes several existing annotation fusion methods, such as estimator weighted evaluator (EWE) [67] and generic-canonical time warping (GCTW) [68]. In addition, the authors have developed a new fusion method based on EWE, named rater-aligned annotation weighting (RAAW), which is also included in the software. Furthermore, MuSe-Toolbox includes the capability to convert continuous emotion annotations into categorical labels.

C. Existing Datasets

Several recent literature surveys have provided an overview of existing datasets for emotions in visual media. In order to avoid duplication of effort, readers are directed to these papers for further information, which includes surveys on evoked emotion [16], BEEU [17], FER [4], MER [7], and multimodal emotion [22], [69], [70]. Table 1 presents a comparison of the properties of some representative datasets. Researchers are advised to thoroughly review the data collection protocol used before utilizing a dataset to ensure that the data have been collected in accordance with appropriate psychological guidelines. In addition, when crowdsourcing is utilized, effective mechanisms are essential for filtering out uninformative annotations.

Table 1.

Recent Representative Datasets for Emotion Recognition

Dataset Name	Labeled Samples	Data Type	Categorical Emotions	Continuous Emotions	Lab Controlled	Year	Primary Application
IAPS [45]	1.2k	I	-	VAD		2005	Evoked
FI [71]	23.3k	I	8§	-		2016	Evoked
VideoEmotion-8 [72]	1.2k	V	8†	-		2014	Evoked
Ekman-6 [73]	1.6k	V	6†	-		2018	Evoked
E-Walk [74]	1k	V	4‡	-		2019	BEEU
BoLD [51]	13k	V	26†	VAD		2020	BEEU
iMiGUE [75]	0.4k	V	2	-		2021	BEEU★
CK+ [76]	0.6k	V	7‡	-	✓	2010	FER
Aff-Wild [77]	0.3k	V	-	VA		2017	FER
AffectNet [78]	450k	I	7‡	-		2017	FER
EMOTIC [79]	34k	I	26†	VAD		2017	FER*
AFEW 8.0 [80]	1.8k	V	7‡	-		2018	FER
CAER [81]	13k	V	7‡	-		2019	FER*
DFEW [82]	16k	V	7‡	-		2020	FER
FERV39k [83]	39k	V	7‡	-		2022	FER
SAMM [84]	0.2k	V	7‡	-	✓	2016	MER
CAS(ME)2 [85]	0.06k	V	4‡	-	✓	2017	MER
ICT-MMMO [86]	0.4k	V,A,T	-	Sentiment		2013	Multi-Modal
MOSEI [87]	23.5k	V.A.T	6†	Sentiment		2018	Multi-Modal

^†A superset of Ekman’s basic emotions

^‡Ekman’s basic emotions + neutral

^§Mikels’ emotions

^★Micro-gesture understanding and emotion analysis dataset Data Type Key: (I)mage, (V)ideo, (A)udio, (T)ext

^*Context-aware emotion dataset

Data Type Key: (I)mage, (V)ideo, (A)udio, (T)ext

D. Data Representations

After the data collection and quality assurance stages, a significant technological challenge is to represent the emotion-relevant information present in the raw data in a concise form. While current deep neural network (DNN) approaches often utilize raw data, such as matrices of pixels, as input in the modeling process, utilizing a compact data representation can potentially improve the efficiency of the learning process, allowing for larger-scale experiments to be conducted with limited computational resources. In addition, a semantically meaningful data representation can facilitate interpretability, which is crucial for certain applications. There are numerous methods for compactly representing raw visual data, and we discuss several intriguing or widely used data representations for emotion modeling in the following.

1). Roundness, Angularity, Simplicity, and Complexity:

Colors and textures are commonly used in image analysis tasks to represent the content of an image, but research has shown that shape can also be an effective representation when analyzing evoked emotions. In both visual art and psychology, the characteristics of shapes, such as roundness, angularity, simplicity, and complexity, have been linked to specific emotional responses in humans. For example, round and simple shapes tend to evoke positive emotions, while angular and complex shapes evoke negative emotions. Leveraging this understanding, Lu et al. [46] developed a system that predicted evoked emotion based on line segments, curves, and angles extracted from an image. They used ellipse fitting to implicitly estimate roundness and angularity, and used features from the visual elements to estimate complexity. Later, they developed algorithms to explicitly estimate these representations [48]. Fig. 5 shows some example images with different levels of roundness, angularity, and simplicity. The researchers found that these three physically interpretable visual constructs achieved comparable classification accuracy to the hundreds of shape, texture, composition, and facial feature characteristics previously examined. This result was thought-provoking because just a few numerical-value representations could effectively predict evoked emotions.

Fig. 5.

Researchers developed algorithms to compute the roundness, angularity, and simplicity/complexity of a scene as the representation for estimating the evoked emotion [48].

2). Facial Action Coding System (FACS) and Facial Landmarks:

People use particular facial muscles to express certain facial expressions. For instance, people can express anger by frowning and pursing their lips. Consequently, each facial expression can be viewed as a combination of some facial muscle movements. Ekman and Friesen [88] developed the FACS in 1976, which encodes all movements of facial muscles. FACS defines a total of 32 atomic facial muscle actions, called Action Units (AUs), including Lids Tight (AU7), Cheek Raise (AU6), and so on. By detecting all AUs of a person and linking them to specific expressions, we can identify the individual’s facial expressions.

The problem of AU detection can be approached as a multilabel binary classification problem for each AU. Early work on AU detection used facial landmarks to identify regions of corresponding muscles and then applied neural networks [89] or support vector machines (SVMs) [90] for classification. More recent work has developed end-to-end AU detection networks [91]. Survey papers provide detailed introductions to facial AU detection [92], [93] and face landmark detection [94], [95]. Some researchers also used facial landmarks directly as a representation of facial information in their recognition work.

3). Body Pose and Body Mesh:

People can express emotions through body posture and movement. By manipulating the positioning of body parts (e.g., the shoulders and arms), people produce various postures and movements. The coordinates of human joints can serve as a representation of body language, reflecting the individual’s bodily expression. In the field of computer vision, 2-D pose estimation is a well-studied task for detecting the 2-D position of human joints in an image. Leveraging large-scale 2-D pose datasets (e.g., COCO [96]), researchers have proposed several high-performing pose networks [97], [98]. Even with challenging scenes, such as crowded or occluded scenes, these networks are able to provide comprehensive joint detection and linking.

However, 2-D pose estimation does not fully capture the 3-D nature of human posture and movement. 3-D human pose estimation, on the other hand, aims to predict the 3-D coordinates of human joints in space. Single-person 3-D pose estimation methods determine the 3-D joint coordinates relative to the person’s root joint (i.e., the torso) [99], [100]. In addition, some multiperson 3-D pose estimation approaches comprehensively estimate the absolute distance between the camera and the individuals in the image [101].

3-D human mesh estimation, which provides the 3-D coordinates of each point on the human mesh, is a further extension of the 3-D pose estimation. Researchers often utilize SMPL [103], [104] or other human graph models to represent the mesh. Fig. 6 illustrates an example of 2-D pose, 3-D pose, and 3-D mesh automatically generated from a scene with multiple people [102].

Fig. 6.

Example of state-of-the-art deep learning-based human pose and mesh detection with individuals in the scene separated properly [102]. The input to the methods is a standard 2-D RGB photograph. The 3-D mesh method provides an estimation of the relative depths of the people in the scene. (a) Original photograph. (b) 2-D pose. (c) 2-D pose, superimposed. (d) 3-D pose. (e) 3-D mesh. (f) 3-D mesh, alternative view.

While human pose and human mesh representations provide a higher level of abstraction compared to low-level representations, such as raw video or optical flow, there is still a significant gap between these intermediate-level representations and high-level emotion representations. To bridge this gap, an intermediate-level representation that effectively describes human movements is proposed. Specifically, Laban movement analysis (LMA) is a movement coding system initially developed by the dance community, similar to sheet music for describing music. Fig. 7 illustrates the layers of data representation for BEEU, from low pixel-level representations to the ultimate high-level emotion representation. Elevating each layer higher in this information pyramid requires considerable advancements in abstraction technology.

Fig. 7.

For BEEU, LMA as a movement coding system can serve as an intermediate level of representation to bridge the significant gap between the pose and the emotion/mental levels [51].

E. Human Movement Coding and LMA

Expressive human movement is a complex phenomenon. The human body has 360 joints, many of which can move various distances at different velocities and accelerations, and in two (and depending on the joint, sometimes more) different directions, resulting in an astronomical number of possible combinations. These variables create an infinite number of movements and postures that can convey different emotions. Beyond this array of body parts moving in space, the expressive movement also involves multiple qualitative motor components that appear in unique patterns in different individuals and situations. The complexity of human movement, thus, raises the question: how do we determine which of the numerous components present in the expressive movement are significant to code for emotional expression in movement? Thus, when choosing a coding system, the early stages of each research project can benefit from deeply considering which aspects of movements are central to the expression being studied. A multistage methodology, such as first identifying what is potentially relevant and then using preliminary analyses to refine the selection of movements most promising to code, can be helpful before selecting a method to code or quantify the multitude of variables present in unscripted movement (e.g., [105] and [106]).

After deliberating about which movement variables are relevant and meaningful, we must then consider the three main types of coding systems that have been used in various fields, such as psychology, computer vision, animation, robotics, and AI.

1. Lists of specific motor behaviors that have been found in scientific studies to be typical to the expressions of specific emotions, such as head down and moving slowly as characterizing sadness; moving backward and bringing the arms in front of the body as characterizing fear; jumping, expanding, and upward movements as characterizing happiness; and so on (for review of these studies and lists of these behaviors, see [107] and [108]).

2. Kinematic description of the human body models, such as skeleton-based models, contour-based models, or volume-based models. Most work in the field of emotion recognition is based on skeleton-based models [109]. This type of model uses 3-D coordinates of markers that were placed on (using various MoCap systems) or were mapped (using pose estimation techniques) to the main joints of the body to create a moving “skeleton,” which enables researchers to quantitatively capture the movement kinematics (e.g., [110] and [111]).

3. LMA, a comprehensive movement analysis system that depicts qualitative aspects of movement and, theoretically [112], [113], as well as through scientific research [114], [115], relates the different LMA motor elements (movement qualities) to cognitive and emotional aspects of the moving individual.

The first coding system, based on lists of motor behaviors, was primarily used in earlier studies in the field of psychology, where the encoding and decoding of motor behaviors into emotions and vice versa were done manually by human coders in a labor-intensive process. Other limitations of this coding system include the following.

1. It is based on a limited number of behaviors that have been used in prior scientific studies. However, people can physically express emotions in many different ways, so the list of previously observed and studied whole-body expressions may not be exhaustive or inclusive of cultural variations.

2. Likewise, because each study used different lists of behaviors, this method makes it difficult to compare results or to review them additively to arrive at larger verification. Thus, this coding system may miss parts of the range of bodily emotional expression, such as those never observed and coded before. This limitation is especially pronounced because many of these previous studies examined emotional expressions performed by actors, whose movement tended to rely upon more stereotypical bodily emotional expressions that were widely recognized by the audience, rather than naturally occurring motor expressions.

When using the second coding system, kinematic description, in particular, the skeleton-based models, researchers usually employ a set of markers similar to, or smaller than that provided by the Kinect Software Development Kit, and transform a large amount of 3-D data into various low- and/or high-level kinematic and dynamic motion features, which are easier to interpret than the raw data. Researchers using this method have studied specific features, such as head and elbow angles [111], maximum acceleration of hand and elbow relative to spine [116], and distance between joints [117], among others. For a review of such studies, please refer to [17]. In recent years, instead of computing handcrafted features on the sequence of body joints, researchers have employed various deep learning methods to generate representations of the dynamics of the human body embedded within the joint sequence, such as spatiotemporal graph convolutional network (ST-GCN) [51], [118], [119]. Although the 3-D data from joint markers can provide a relatively detailed, objective description of whole-body movement, this coding system has two main limitations.

1. Movement is often captured by a camera from a single view (usually the frontal), which can result in long segments of missing data from markers that are hidden by other body parts, people, or objects in a video frame. Automatic imputations of such missing data are often impractical as they tend to create incorrect and unrealistic movement trajectories.

2. The SDK system has only three markers along the torso, which are insufficient for capturing subtle movements in the chest area, movements that are usually observed during authentic emotional expressions, as opposed to acted (and often exaggerated) bodily emotional expressions. Another disadvantage is that these systems have not yet been able to successfully and reliably detect many qualitative changes in movement that are significant for perceiving emotional expression.

In contrast to the quantitative data from joint markers, which enable the capture of detailed movement of every body part, the third coding system mentioned above, LMA, describes qualitative aspects of movement and can relate to a general impression from movements of the entire body or to the movement of specific body parts. By identifying the association between LMA motor components and emotions, and characterizing the typical kinematics of these components using high-level features (e.g., [51], [120], [121], [122], [123], and [124]), researchers can overcome the limitations of other coding systems. If people express their emotions with movements that have never been observed in previous studies, we can still decode their emotions based on the quality of their movement. Similarly, if parts of the body, which are usually used to express a certain emotion, are not visible, it is possible that the emotion could still be decoded by identifying the motor elements composing the visible part of the movement. Moreover, by slightly changing the kinematics of a movement of a robot or animation (i.e., adding to a gesture or a functional movement the kinematics of certain LMA motor elements associated with a specific emotion), we can “color” this functional movement with an expression of that emotion, even when the movement is not the typical movement for expressing that emotion (e.g., [125] and [126]). Similarly, identifying the quality of a movement can enable decoding the expressed emotion even from functional actions, such as walking [127] or reaching and grasping. These advantages and the fact that LMA features have been found to be positively correlated with emotional expressions are why LMA coding is becoming popular in studies that encode or decode bodily emotion expressions (e.g., [51] and [123]). In addition, LMA offers the option to link our coding systems to diverse ways in which humans talk about and describe expressive movement—it is a comprehensive movement-theory system that is and can be used across disciplines for application in acting [128], therapy [129], education [130], and animation [131], among others. The last advantage to consider is that LMA is a comprehensive theory of body movement, much like art theory or music theory, including theories of harmony, and, thus, has been used by artists to attune to aesthetics including movement-perception of visual art (such as that discussed in Section VI-A) and visual, auditory, and movement elements of film and theater (as discussed in Section III-G). Like music theory, LMA is capable of attending to rhythm and phrasing as elements shift and unfold over time, aspects that may be crucial to communicating and interpreting emotional expression.

LMA identifies four major categories of movement: Body, Effort, Shape, and Space. Each category encompasses several subsets of motor components (LMA terms are spelled with capital letters to differentiate them from regular usage of these words). Fig. 8 illustrates some basic components of LMA, which are often used in coding.

Fig. 8.

Some basic components of LMA, which are often used in coding.

The Body category describes what is moving, and it is composed of the elements of Body Segments (e.g., arms, legs, and head), their coordination, and basic Body Actions, such as locomotion, jump, rotation, change of support, and so on.

The Effort category describes the qualitative aspect of movement or how we move. It expresses a person’s inner attitude toward movement, and it has four main factors, each describing the continuum between two extremes: indulging in the motor quality of that factor and fighting against that quality. An effort has four factors.

1. Weight effort, meaning the amount of force or pressure exerted by the body. Activated Weight Effort can be Strong or Light. Alternatively, there may be a lack of weight activation when we give in to the pull of gravity, which we describe as Passive or Heavy Weight.

2. Space effort describes attention to space, denoting the focus or attitude toward a chosen pathway, i.e., is the movement Direct or flexibly Indirect.

3. Time effort, describing the mover’s degree of urgency or acceleration/deceleration involved in a movement, i.e., is the movement Sudden or Sustained.

4. Flow effort, reflecting the element of control or the degree to which a movement is Bound, i.e., restrained or controlled by muscle contraction (usually cocontraction of agonist and antagonist’s muscles), versus Free, i.e., being released and liberated.

The Shape category reflects why we move: Shape describes how the body adapts its shape as we respond to our needs or the environment: do I want to connect with or avoid something, dominate, or cower under? The way the body sculpts itself in space reflects a relationship to self, others, or to the environment. This component includes Shape Flow that describes how the body changes to relate to oneself; it includes Shape Change that describes changes in the form or shape of the body and includes the motor components of Expanding the body or Condensing it in all directions and Rising or Sinking in the vertical dimension, Spreading and Enclosing in the horizontal dimension, and Advancing or Retreating in the sagittal dimension. Another Shape component is Shaping and Carving, which describes how a person shapes their body to shape or affect the environment or other people. For example, when we hug somebody, we might shape and carve the shape of our body, adjusting it to the shape of the other person’s body, or we might shape our ideas by carving or manipulating them through posture and gesture.

The Space category describes where the movement goes in the environment. It describes many spatial factors, such as the Direction, where the movement goes in space, such as Up and Down in the vertical dimension, Side open and Side across in the horizontal dimension, and Forward and Backward in the sagittal dimension; the Level of the movement in space relative to the entire body or parts of the body, such as Low level (movement toward the ground), Middle level, (movement maintaining level, without lowering or elevating), or High level (moving upward in space); Paths or how we travel through space by locomoting; and Pathways through the Kinesphere (the personal bubble of reach-space around each mover that can be moved in and through without locomoting or traveling.) Movement in the Kinesphere might take Central pathways, crossing the space close to the mover’s body center, Peripheral pathways along the periphery of the mover’s reach space, or Transverse pathways cutting across the reach space.

In addition, another important aspect of LMA that is particularly helpful and meaningful to expression is the Phrasing of movements. Phrasing describes changes over time, such as changes in the intensity of the movement over time, similar to musical phrases, which can be Increasing, Decreasing, or Rhythmic, among others. It can also depict how movement components shift during the same action or a series of actions occurring over time, for example, beginning emphatically with strength and then ending by making a light direct point conclusion.

Previous research has highlighted the lack of a notation system that directly encodes the correspondence between bodily expression and body movements in a way similar to FACS for face [51], [105]. LMA, by its nature, has the potential to serve as such a coding system for emotional expressions through body movement. Shafir et al. [115] identified the LMA motor components (qualities), whose existence in a movement could evoke each of the four basic emotions: anger, fear, sadness, and happiness. Melzer et al. [114] identified the LMA components whose existence in a movement caused that movement to be identified as expressing one of those four basic emotions. In an additional experiment by the Shafir group, Gilor, for her Master’s thesis, has been studying the motor elements that are used for expressing sadness and happiness. In this series of studies, the LMA motor components found to evoke these emotions through movement were, for the most part, the same as those used to identify or recognize each emotion from movement, or to express each emotion through movement. For example, anger was associated with Strong Weight Effort, Sudden Time Effort, Direct Space Effort, and Advancing Shape during both emotion elicitation and emotion recognition. Fear was associated with Retreating Shape, moving backward in Space for both emotion elicitation and recognition. In addition, Enclosing and Condensing Shape, and Bound Flow Effort, were also found for emotion elicitation through movement. Sadness was associated with Passive Weight Effort, Sinking Shape, and Head drop for emotion elicitation, emotional expression, and emotion recognition, and arms touching the upper body were also significant indicators for emotion elicitation and expression. Sadness expression was also associated with using Near-Reach Space and Stillness. In contrast, Happiness was associated with jumping, rhythmic movement, Free and Light Efforts, Spreading and Rising Shape, and moving upward in Space for emotion elicitation, emotion expression, and emotion recognition. Happiness expression was also associated with Sudden Time Effort and Rotation. These findings for Happiness were also validated by van Geest et al. [132]. While these studies represent a promising start, further research is needed to create a comprehensive LMA coding system for bodily emotional expressions.

F. Context and Functional Purpose

While human emotions are shown through the face and body, they are closely connected to context and purpose. Thus, the image context surrounding the human can also be used to further identify human emotions. The context information includes the semantic information of the background and what the person is holding, which can assist in identifying activities of the human, thereby allowing for more accurate prediction of the human’s emotion. For instance, people are more likely to feel happy than sad during a birthday party. Context information also includes interactions among humans, which can help infer emotion. For example, people are more likely to be angry when they are engaged in a heated argument with others.

Since the early 2000s, researchers in the field of image retrieval have been developing context recognition systems using machine learning and statistical modeling [133], [134], [135]. With the advent of deep learning and the widespread use of modern graphics processing units (GPUs) or AI processors, accurate object annotation from images has become more feasible. Several deep learning-based approaches have been proposed to leverage context information to enhance basic emotion recognition networks [81], [136], [137]. Sections IV-B3 and IV-E provide more details.

In addition to contextual information, the functional purpose of a person’s movement can also provide valuable insights when inferring emotions. Movement is a combination of both functional and emotional expression, and thus, emotion recognition systems must be able to differentiate between movement that serves a functional purpose and movement that expresses emotions. Action recognition [138], an actively studied topic in computer vision, has the potential to provide information on the function of a person’s movement and assist in disentangling functional and emotional components of movement.

G. Acted Portrayals of Emotion

Research on emotions has often turned to acted portrayals of emotion to overcome some of the challenges inherent in accessing adequate datasets, particularly because evoking intense, authentic emotions in a laboratory can be problematic both ethically [139] and practically. This is because, as noted in Section II, emotional responses vary. Datasets relying upon actors have been useful in overcoming challenges of obtaining ample, adequately labeled emotion-expression data because, compared to unscripted behavior (“in the wild”), where emotion expression often appears in blended or complex ways, actors are typically serving a narrative, in which emotion is part of the story. In addition, sampling emotional expression in the wild encounters cultural distinctions for the expressions themselves, as well as social norms for emotion expression or its repression [140], which may also be culturally scripted for gender. Thus, researchers interested in emotional expressivity and nonverbal communication of emotion often turn to trained actors [139] both to generate new datasets of emotionally expressive movement (e.g., [141]) and for the sampling of emotion expression (e.g., [51]). Such datasets are useful because actors coordinate all three channels of emotions expression, namely, vocal, facial, and body, to produce natural or authentic-seeming expressions of emotions. Some researchers have validated the perception of actor-based and expert movement-based datasets in the lab by showing them to lay observers [114], [141]. This approach also entails problems, in which, while it may capture norms of emotion expression and its clear communication, it may miss distinctions related to demographics such as gender [142], ethnolinguistic heritage, individual, or generational norms [143], [144]. According to cultural dialect theory, we all have nonverbal emotion accents [144], meaning that emotion is expressed differently by different people in different regions, from different cultures. Only some of those cultural dialects appear when sampling films. Such films have often been edited so that viewers beyond that cultural dialect can “read” the emotional expression central to the narrative. Nuance in nonverbal dialects may be excluded in favor of general appeal.

Yet, an advantage of generating datasets from actors or other trained expressive movers is that ground truth can be better established. The intention of the emotion expressed, the emotion felt, and later the emotion perceived from viewing can all be assessed when generating the dataset. Likewise, because actors coordinate image, voice, and movement in the service of storytelling, the context and purpose are clarified, and thus, many of the multiple expressive modes can be organized into individual emotion expression [145], [146]. Moreover, because performing arts productions, such as movies and films made of theater, music, and dance performances, integrate multiple communication modes, the creative team collaborates frequently about the emotional tone or intent of each work, coordinating lighting, scenery/background, objects, camerawork, and sound, with the performers (actors, dancers, and musicians). While the team articulates their intentions during the creative process, the resulting produced art often resonates with wider variation to different audiences, according to their perceptions and tastes.

For researchers relying upon acted emotions, it may be helpful to understand some ways in which actor training considers the role of emotions in theater and film [147]. The role of emotion in narrative arts may reflect some of the theories about the role of emotion itself [148]—to approximately inform the character (organism) about their needs, in order to drive them to take action to meet their needs, or to provide feedback on how recent actions meet or do not meet their needs. As actors prepare, they identify a character’s needs (called their objective) and are moved by emotion to drive the character’s action to overcome obstacles as they pursue the character’s objective. Thus, when collecting data from acted examples, emotion expression can often be found preceding action in the narrative or in response to it. Actors are also trained to listen with their whole being to their scene partner to “hold space” during dialog for emotion expression to fully unfold and complete its purpose of moving either the speaker or the listener to response or action. This expectation is particularly true in opera or other musical theater genres, which often extends the time for emotion expression.

In terms of 3-D modeling, actors trained for theater not only highly develop the specific emotion-related action tendencies of the body but also consider viewer perception from 3-D, for example, when performing on a thrust stage or theater-in-the-round. Thus, their emotion expression may be more easily picked up by systems marking parts of the body during movement. Because bodily emotion expression is so crucial to a narrative, an important application of this field might be to automate audio description of emotion expression in film for the visually impaired or audio description of movement features salient to emotion expression.

While we often do not recognize subtle emotion expression in strangers, and we can in those we know, when actors portray characters within the circumstances of a play, the audience, gradually over the arc of the play, comes to perceive the emotional expression of each character as revealed over time. Understanding how this works in art can help us develop systems that take the time to learn and better understand emotion expression in diverse contexts and individuals, similar to how current voice recognition learns individual accents.

H. Cultural and Gender Dialects

In a meta-analysis examining cultural dialects in the nonverbal expression of emotion, measurable variations were found in response to facial expression across cultures [149]. Moreover, it has long since been acknowledged that learned display rules [150] and decoding rules [151] vary across different cultures. For example, in Japanese culture, overt displays of negative emotional expression are discouraged. Furthermore, smiling behavior in this context is often considered an attempt to conceal negative emotions. In contrast, in American culture, negative expressions are considered more appropriate to display [151].

Previous research has similarly demonstrated a powerful influence of gender-related emotion stereotypes in driving differential expectations for the type of emotions that men and women are likely to experience and express. Men are expected to experience and express more power-related emotions, such as anger and contempt, whereas women are expected to experience and express more powerless emotions, such as sadness and fear [152]. These findings match self-reported emotional experiences and expressions with strong cultural consistency across 37 different countries worldwide [153]. Furthermore, there are even differences in the extent to which neutral facial appearance resembles different emotions, with male faces physically resembling anger expressions more (low, hooded brows, thin lips, and angular features) and female faces resembling fear expressions more [154].

While cultural norms affect how and whether emotions are displayed, such norms also influence how displays of emotion are perceived. For example, when a culture’s norm is to express emotion, less intensity is read into such displays, whereas, in cultures where the norm is not to express emotion intensely, the same displays are read as more intense [155]. In this way, visual percepts derived from objective stimulus characteristics can generate different subjective experiences based on culture and other individual factors. For instance, there is notable cultural variation in the extent to which basic visual information, such as background versus foreground, is integrated into observers’ perceptual experiences [156].

Culture and gender add complexity to human emotional expression, yet little research to date has examined individual variation in responses to visual scenes, either in terms of basic aesthetics or the emotional responses that people have. Future work assessing simple demographic details (e.g., gender and age) will begin to explore this important source of variation.

I. Structure

Some basic visual properties have been found that characterize positive versus negative experiences and preferences. Most notably, round features—whether represented in faces or objects—elicit feelings of positivity and warmth, and tend to be preferred over angular visual features. This preference has been used to explain the roundness of smiling faces and the angularity of anger displays [157]. Such visual curvature has also been found to influence attitudes toward and preference for even meaningless patterns represented in abstract visual designs [158]. The connection between affective responses and these basic visual forms has helped computer vision predict emotions evoked from pictorial scenes, as mentioned earlier [46].

Importantly, the dimensional approach to assessing visual properties underlying emotional experience can be used to examine both visual scenes and faces found within those scenes. Indeed, the dimensional approach adequately captures both “pure” and mixed emotional facial expressions [159], as well as affective responses to visual scenes, as demonstrated by the IAPS. Critically, even neutral displays have been found to elicit strong spontaneous impressions [160], ones that are effortless, nonreflective, and highly consensual. Recent research utilizing computer-based models suggests that these inferences are largely driven by the same physical properties found in emotional expressions. For instance, Said et al. [161] employed a Bayesian network trained to detect expressions in faces and then applied this analysis to images of neutral faces that had been rated on a number of common personality traits. The results showed that the trait ratings of faces were meaningfully associated with the perceptual resemblances that these “neutral” faces had with emotional expressions. Thus, these results speak to a mechanism of perceptual overlap, whereby expression and identity cues can both trigger similar emotional responses.

A reverse engineering model of emotional inferences has suggested that perceptions of stable personality traits can be derived from emotional expressions as well [162]. This work implicates appraisal theory as a primary mechanism by which observing facial expressions can inform stable personality inferences made of others. This account suggests that people use appraisals that are associated with specific emotions to reconstruct inferences of others’ underlying motives, intents, and personal dispositions, which they then use to derive stable impressions. It has likewise been shown that emotion-resembling features in an otherwise neutral face can drive person perception [152]. Finally, research has also suggested that facial expressions actually take on the form that they do to resemble static facial appearance cues associated with certain functional affordance, such as facial maturity [163] and gender-related appearance [152].

J. Personality

Personality describes an individual’s relatively stable patterns of thinking, feeling, and behaving. These are characterized by certain personality traits that represent a person’s disposition toward the world. Emotions, on the other hand, are the consequence of the individual’s response to external or internal stimuli. They may change when the stimuli change and, therefore, are considered as states (as opposed to traits). Just as a full-blown emotion represents an integration of feeling, action, appraisal, and wants at a particular time and location, personality represents the integration of these components over time and space. Researchers have tried to characterize the relationships between personality and emotions. Several studies found correlations between certain personality traits and specific emotions. For instance, the trait neuroticism was found to be correlated with negative emotions, while extroverted people were found to experience higher levels of positive emotions than introverted people [164]. These correlations were explained by relating different personality traits to specific emotion regulation strategies e.g., [165], or by demonstrating that evaluation mediates between certain personality traits and negative or positive affect [166]. Whatever the reason for these correlations is, the fact that they are correlated might help to create state-to-trait inferences. Therefore, if emotional states can be mapped to personality, the ability to automatically recognize emotions could provide tools for the automatic detection of personality.

Advances in computationally, data-driven methods offer promising strides toward personality traits being predicted based on a dataset of individuals’ behavior, self-reported personality traits, or physiological measures. It is also possible to use factor analyses to identify underlying dimensions of personality, such as the Big Five personality traits, or openness, conscientiousness, extraversion, agreeableness, and neuroticism (OCEAN) [167], [168] (see Fig. 9). Personality can likewise be inferred from facial expressions of emotion [162] and even from emotion-resembling facial appearance [169]. Machine learning algorithms have recently been successfully employed to predict human perceptions of personality traits based on such facial emotion cues [170]. Furthermore, network analysis, such as social network analysis, can also be incorporated to identify patterns of connectivity between different personality traits or behavioral measures. Finally, interpreting a person’s emotional expression might also depend on the perception of their personality when additional data are available for inferring personality [171]. For further information, readers can refer to recent research in this area, e.g., [172] and [173].

Fig. 9.

Big Five personality traits or OCEAN.

K. Affective Style

Affective styles driven by the tendencies to approach reward versus avoid punishments found their way into early conflict theories [174] and remain a mainstay of contemporary psychology in theories such as Carver and Scheier’s Control Theory [175] and Higgins’s Self-discrepancy Theory [176].

Evidence for the specific association between emotion and approach-avoidance motivation has largely involved the examination of differential hemispheric asymmetries in cortical activation. Greater right frontal activation has been associated with avoidance motivation, as well as with flattened positive affect and increased negative affect. Greater left frontal activation has been associated with approach motivation and positive affect [177]. Supporting the meaningfulness of these findings, Davidson [177] argued that projections from the mesolimbic reward system, including basal ganglia and ventral striatum, which are associated with dopamine release, give rise to greater left frontal activation. Projections from the amygdala associated with the release of the primary vigilance-related transmitter norepinephrine give rise to greater right frontal activation.

Further evidence supporting the emotion/behavior orientation link stems from evidence accumulated in studies using measures of behavioral motivation based on Gray’s [178] proposed emotion systems. The most widely studied of these are the behavioral activation system (BAS) and the behavioral inhibition system (BIS). The BAS is argued to be highly related to appetitive or approach-oriented behavior in response to reward, whereas the BIS is argued to be related to inhibited or avoidance-oriented behavior in response to punishment. Carver and White [179] developed a BIS/BAS self-report rating measure that is thought to tap into these fundamental behavioral dispositions. They found that extreme scores on BIS/BAS scales were linked to behavioral sensitivity toward reward versus punishment contingencies, respectively [179]. BIS/BAS measures have been shown to be related to emotional predisposition, with positive emotionality being related to the dominance of BAS over BIS, and depressiveness and fearful anxiety being related to the dominance of BIS over BAS [180].

Notably, for many years, there existed a valence (positive/negative) versus motivational (approach/avoidance) confound in all work conducted in the emotion/behavior domain. Negative emotions were associated with avoidance-oriented and positive emotions with approach-oriented behavior, a contention supported by much of the work reviewed above. The valence/motivation confound led researchers Harmon-Jones and Allen [181] to test for hemispheric asymmetries in activation associated with anger, a negative emotion with an approach orientation (aggression). They argued that, if left hemispheric lateralization was associated with anger, this would indicate that the hemispheric lateralization in activation previously found was in fact due to behavioral motivation. However, right hemispheric lateralization would indicate that they were due to valence. In these studies, they found that dispositional anger [181] was associated with left lateralized EEG activation, consistent with the first interpretation and with that previously reported only for positive emotion. They supported this conclusion by showing that the dominance of BAS over BIS was associated with anger [58].

IV. EMOTION RECOGNITION: KEY IDEAS AND SYSTEMS

The field of computer-based emotion recognition from visual media is nascent but has seen encouraging developments in the past two decades. Interest in this area has also grown sharply (see Section IV-A). We will highlight some existing research ideas and systems related to modeling evoked emotion (see Section IV-B), facial expression (see Section IV-C), and bodily expression (see Section IV-D). In addition, we will discuss integrated approaches to emotion recognition (see Sections IV-E and IV-F). Because of the breadth of the field, it is not possible to cover all exciting developments, so we will focus our review on the evolution of the field and some of the most current, cutting-edge results.

A. Exponential Growth of the Field

To gain insight into the growing interest of the IEEE and computing communities in emotion recognition research, we conducted a survey of publications in the IEEE Xplore and ACM Digital Library (DL) databases. Results revealed an exponential increase in the number of publications related to emotion or sentiment in images and videos over the last two decades (see Fig. 10).

Fig. 10.

Exponential growth in the number of IEEE and ACM publications related to emotion in images and videos in the recent two decades was observed. Statistics are based on querying the IEEE and ACM publication databases. The lower rate of expansion observed in the year 2020 can be attributed to the disruptions caused by the COVID-19 pandemic and the limitations that it imposed on human subject studies.

As of February 2023, a total of 48 154 publications were found in IEEE Xplore, with the majority (35 247 or 73.2%) being in conference proceedings, followed by journals (8214 or 17.1%), books (2400 or 5.0%), magazines (1413 or 2.9%), and early access journal articles (831 or 1.7%). The field has experienced substantial growth, with a 25-fold increase during the period from the early 2000s to 2022, rising from an average of 275 publications per year to about 7000 per year in 2022.

In the ACM DL, a total of 30 185 publications were found, with conference proceedings making up the majority (24 817 or 82.2%), followed by journals (3851 or 12.8%), magazines (780 or 2.6%), newsletters (505 or 1.7%), and books (264 or 0.9%). In the ACM community, the field has seen a 22-fold growth during the same period, with an average of 170 publications per year in the early 2000s rising to about 3800 per year in 2022.

A baseline search indicated that the field related to images and videos had a roughly linear, sixfold growth during the same period. Emotion-related research accounted for 14.2% of image- and video-related publications. The growth in emotion-related research outpaced the baseline growth by a significant margin, suggesting that it has a higher future potential. The annual growth rate for the field of emotion in images and videos is 15%–16%. If this growth continues, the annual number of publications in this field is expected to double every five years.

B. Modeling Evoked Emotion

Evoked emotions can be inferred when viewers see an image or a video, either based on changes in their physical body or the stimuli being viewed, i.e., explicit affective cues or implicit affective stimuli [22], [69], [114]. In this section, we will focus on implicit affective stimuli, particularly images and videos. Similar to other machine learning and computer vision tasks [16], [182], an evoked emotion recognition system from images or videos typically consists of three components: emotion feature extraction, feature selection and fusion, and feature-to-evoked-emotion mapping. The generalized frameworks for this process are illustrated in Figs. 11 and 12. The first step is to extract emotion features from the original images and videos, typically after preprocessing and converting them to numerical representations that are easier to process. Feature selection and fusion aim to select discriminative features that are more relevant to emotions, reduce the dimensionality of features, and combine different types of features into a unified representation. Finally, each unified representation is classified. Feature-to-evoked-emotion mapping learns a classifier to project the feature representation to specific evoked emotions.

Fig. 11.

Generalized framework for evoked emotion prediction from images.

Fig. 12.

Generalized framework for evoked emotion prediction from videos.

1). Visual Emotion Feature Representation:

Prior to the emergence of deep learning, visual emotion features were primarily developed manually, drawing inspiration from various interrelated disciplines, including computer vision, art theory, psychology, and aesthetics.

One straightforward approach employs existing low-level vision features as representations of emotion. Yanulevskaya et al. [183] extracted holistic Wiccest and Gabor features. Other researchers have employed scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), self-similarities, GIST, and local binary patterns (LBPs) to extract low-level vision representations of emotion for video keyframes [72], [184]. Yang et al. [185] compared their method with multiple low-level vision features, including SIFT, HOG, and Gabor. Rao et al. [186] first employed bag of visual words of SIFT features for each image block from multiple scales and then represented each block by extracting latent topics based on probabilistic latent semantic analysis. While such low-level vision representations are still used, such as LBP and optical flows [187], they do not effectively bridge the semantic gap and cannot bridge the more challenging affective gap [16].

More robust approaches to exploring visual emotion features employ theories on art and aesthetics to design and develop features. Art is often created with the intention of evoking emotional reactions from the audience. As Pablo Picasso claims, “Art washes away from the soul the dust of everyday life,” and as Paul Cézanne asserts, “A work of art which did not begin in emotion is not art,” [44], [188], [189]. Generally, art theory includes elements of art and principles of art. The elements of art, such as color, line, texture, shape, and form, serve as building blocks in creating artwork. Meanwhile, the principles of art correspond to the rules, tools, or guidelines for arranging and orchestrating the elements of art, such as balance, emphasis, harmony, variety, and gradation. Among these elements, color is the most commonly used artistic feature [44], [187], [190], [191], [192], [193], [194], [195], followed by texture. Lu et al. [46] investigated the relationship between shape and emotions through an in-depth statistical analysis. Zhao et al. [188] systematically formulated and implemented six artistic principles except for rhythm and proportion and combined them into a unified representation.

Aesthetics is widely acknowledged to have a strong correlation with emotion [182], [196]. Artwork that is aesthetically designed tends to attract viewers’ attention and create a sense of immersion. As early as 2006, Datta et al. [52] developed computer algorithms to predict the aesthetic quality of images. Among the various aesthetic features, composition, such as the rule of thirds, has been the most popular [44], [52], [190], [191], [196], [197]. The figure-ground relationship, which refers to cognitive feasibility in distinguishing the foreground and the background, and several other aesthetic features were designed [196]. Sartori et al. [198] analyze the influence of different color combinations on evoking binary sentiment from abstract paintings. Artistic principles and aesthetics were used to organize artistic elements from different but correlated perspectives, which were sometimes not clearly differentiated and extracted together [196], [197]. By considering relationships among different elements, artistic principles and aesthetics have been demonstrated to be more interpretable, robust, and accurate than artistic elements and low-level vision representations for recognizing evoked emotions [188], [196].

To bridge the gap between low-level visual features and high-level semantics, an intermediate level of attributes and characteristics was designed. These intermediate attributes and characteristics were then applied to the prediction of evoked emotions [192], [193], [194], [199]. For example, in addition to generic scene attributes, eigenface-based facial expressions were also considered [199]. These attributes performed better than low-level vision representations, but the interpretability was still limited.

High-level content and concept play an essential role in evoking emotions for images with obvious semantics, such as natural images. The number and size of faces and skins contained in an image have been used as an early and simple content representation [44]. Facial expressions of images containing faces are a direct cue for viewers to produce emotional reactions and are, therefore, often employed as content representation [192], [193]. Jiang et al. [72] developed a method that involved detecting objects and scenes from keyframes. Based on the observation that general nouns such as “baby” were detectable but had a weak link to emotion, whereas adjectives such as “crying” could provide strong emotional information but were difficult to detect, Borth et al. [200] introduced adjective–noun pairs (ANPs) by adding an adjective before a noun, such as “crying baby.” The combination enabled strong ability to map concepts to emotions while remaining detectable. A large visual sentiment ontology named SentiBank was proposed to detect the probability of 1200 ANPs. Thus, as a milestone, SentiBank cannot be ignored as a baseline in performance evaluation, even in the current deep learning era. A multilingual visual sentiment ontology (MVSO) [201] was later extended to deal with different cultures and languages. About 16k sentiment-biased visual concepts across 12 languages are constructed and detected. ANP representations are widely used as high-level semantic representations [72], [192], [193], [194], [202]. These content and concept representations achieve the best performance for the images containing such semantics but fail for abstract paintings.

Fortunately, the rise of deep and very deep convolutional neural networks (CNNs) in image classification has led to deep learning becoming the primary learning strategy in various fields, including computer vision and natural language processing (NLP). This is also the case for evoked emotion modeling [16]. Given sufficient annotated data, deep learning models can be trained to achieve superior performance on various types of images, including natural and social photographs, as well as abstract and artistic paintings. In many cases, features are automatically learned without the need for manual crafting.

Recently, global representation at the image level has been demonstrated to hold promise for evoked emotion analysis. One approach to extract deep features is to directly apply pretrained CNNs, such as AlexNet [203], VGGNet [204], and ResNet [205], to the given images and obtain responses of the last (few) fully connected (FC) layers [71], [190], [194], [197], [202], [206], [207], [208], [209], [210], [211]. Other methods have begun to use the output of a transformer’s encoder as visual reorientation [212], [213]. Xu et al. [206] demonstrated that the deep features from the last but one FC layer outperformed those from the last FC layer. The extracted CNN features were transformed to another kernelized space via discrete Fourier transform and sparse representation to suppress the impact of noise in videos [207]. Chen et al. [214] first obtained event, object, and scene scores with state-of-the-art pretrained detectors based on DNNs. They then integrated these into a context fusion network to generate a unified representation. More recently, Song et al. [195] used the pretrained MS Azure Cognitive Service API to extract objects contained in an image and corresponding confidence scores as visual hints, which were then transformed to TF-IDF representation. This pretrained deep feature can be essentially viewed as another handcrafted feature since the classifier used for the final emotion prediction needs to be separately trained. To enable endto-end training, fine-tuning is widely employed to adjust the deep features to be more correlated with evoked emotions [71], [190], [215], [216]. Other inspiring improvements include multitask learning [185], [217] and emotion correlation mining [218], [219], [220] to better learn emotion-specific deep features. The image-level global representation extracts deep features from the global aspect taking the original image as input. To deal with the temporal correlations among successive frames in videos, 3-D CNN (C3D) [221] is adopted taking a series of frames as input [222]. Although contextual information is considered, the global representation treats local regions equally without considering their importance.

Strategies such as attention [223], [224], [225], [226] and sentiment maps [227], [228] have been widely used to extract region-level representations in order to emphasize the importance of local regions in evoked emotion prediction. Spatial attention is used to determine the correspondence between local image regions and detected textual visual attributes [223]. Besides operating on the global representation to obtain attended local representation, the spatial attention map is enforced to be consistent with prior knowledge contained in the detected saliency map [224]. Both spatial and channelwise attentions are considered to reflect the importance of the spatial local regions and the interdependency between different channels [226]. In contrast, Fan et al. [225] investigated how image emotion could be used to predict human attention and found that emotional content could strongly and briefly attract human visual attention. Yang et al. [227] designed a weakly supervised coupled CNN to explore local information by detecting a sentiment-specific map using a cross-spatial pooling strategy, which only required image-level labels. The holistic and local representations were combined by coupling the sentiment map. To address the issue of overemphasis on local information and neglect of some discriminative sentiment regions, a discriminate enhancement map was recently constructed by spatial weighting and channel weighting [228]. Both the discriminate enhancement map and the sentiment map were coupled with the global representation.

In recent years, a growing body of research has focused on developing effective multilevel representations [219], [229], [230], [231], [232], [233], [234], [235], [236], [237], [238]. One strategy has been to view different CNN layers as different levels [229], [231], [234]. Multiple levels of features were extracted at different branches (four [231] and four plus one main [229] branches). The features from different levels were then integrated by a bidirectional gated recurrent unit (GRU) [229] or a fusion layer with basic operations, such as the mean. These two methods both claim that features from global to local levels can be extracted at different layers, but the correspondence between them is still unclear. This issue is partially addressed in the multilevel dependent attention network (MDAN) [234]. Based on the assumption that different semantic levels and affective levels are correlated, affective semantic mapping disentangles the affective gap by one-to-one mapping between semantic and affective levels. Besides the global-level learning at the highest affective level with emotion hierarchy preserved, local learning is incorporated at each semantic level to differentiate among emotions at the corresponding affective level [234]. Furthermore, multihead cross channel attention and level-dependent class activation maps are designed to model levelwise channel dependencies and spatial attention within each semantic level, respectively. Further research is needed to explore more effective ways to map semantic and affective levels, particularly when the number of levels varies significantly.

Another strategy for extracting multilevel representations involves utilizing object, saliency, and affective region detection [219], [230], [232], [235], [236], [237], [238]. Inspired by the Stimuli-Organism-Response (S-O-R) emotion model in psychology, Yang et al. [219] selected specific emotional stimuli, including image-level color and region-level objects and faces, using off-the-shelf detection models. Corresponding to the selected stimuli, three specific networks were designed to extract the features, including CNN-based color and other global representations, long-short term memory (LSTM)-based semantic correlations between different objects, and CNN-based facial expressions. Besides the correlations between different objects based on GCN, Yang et al. [232] also mined the correlations between scenes and objects using a scene-based attention mechanism, motivated by the assumption that scenes guide objects to evoke distinct emotions. A similar approach was applied by Cheng et al. [238] but with the input to the object detector being the extracted temporal–spatial features by C3D. These methods treat objects, typically detected by Faster R-CNN, as regions, and reflect the importance of different regions through attention [219] or graph reasoning [232], [238]. Objects and faces were also, respectively, detected in [236] and [237], and corresponding CNN features were extracted. Rao et al. [230] employed an emotional region proposal method to select emotional regions and remove nonemotional ones. The region’s emotion score was a combination of the probability of the region containing an object and that of the region evoking emotions. A similar approach for selecting emotionally charged regions was used by Zhang et al. [235], but the emotion score was a weighted combination of two probabilities.

Efforts have been made to combine handcrafted and deep representations in order to take advantage of their complementary information. For example, Liu et al. [197] combined them after feature dimension reduction, whereas Chen et al. [239] used AlexNet-style deep CNNs [203] to train ANP detectors and achieved improved ANP classification accuracy. The DeepSentiBank has also been used to extract keyframe features in videos [210], [240]. More recently, the correlations of content features based on pretrained CNN and color features from color moments have been taken into consideration [241]. Specifically, the proposed cross correlation model consists of two modules: an attention module to enhance color representation using content vectors and a convolution module to enrich content representation using pooled color embedding matrices. Further research is needed to investigate the potential of this approach to interact between handcrafted and deep representations.

2). Audio Emotion Feature Representation:

The audio features used in video-based evoked emotion analysis are often sourced from the fields of acoustics, speech, and music signal processing [69]. Similar to visual features, audio features for emotion analysis can be divided into two categories: handcrafted and deep learning-based.

Among handcrafted feature representations, the Mel-frequency cepstral coefficients (MFCCs) are commonly adopted features [72], [184], [191]. Energy entropy, signal energy, zero crossing rate, spectral rolloff, spectral centroid, and spectral flux are also employed [72], [184]. The openSMILE toolkit is a popular option for extracting handcrafted audio features [242]. El Ayadi et al. [243] classified speech features into four groups: continuous, qualitative, spectral, and Teager energy operator-based. Panda et al. [14] summarized music features into eight dimensions, including melody, harmony, rhythm, dynamics, tone color, expressivity, texture, and form. For more information on handcrafted audio feature representations, please refer to these papers.

Learning-based deep audio feature representations mirror the learning-based strategy used in the visual stream in videos, and some studies have explored the use of CNN-based deep features for audio streams. One approach is to send the raw audio signal directly to a 1-D CNN. However, a more popular method involves transforming the raw audio signal into MFCC, which can be viewed as an image, before feeding it into a 2-D CNN [202], [208], [211], [213], [222], [237], [238]. When simultaneously considering the MFCC from multiple video segments, C3D or LSTM can be utilized. For further information on modeling the temporal information in videos, see Section IV-B4.

3). Contextual Feature Representation:

The context information that is important in evoked emotion prediction can be divided into two categories: within visual media and across modalities. For context within visual media, one common approach is to extract features from different levels that correspond to semantics at different scales, such as the global and local levels. For more details, see Section IV-B1. When considering context across modalities, the social context of users, including their common interest groups, contact lists, and similarity of comments to the same images, is taken into account in personalized image emotion recognition [193]. In addition to the visual and audio representations, motion representation is also considered [213].

4). Feature Selection and Fusion:

As previously mentioned, features can be extracted within a single modality (e.g., image) or across multiple modalities. On the one hand, different types of features can have varying discriminative abilities. High-dimensional features may suppress the power of low-dimensional ones when fusing them together. High-dimensional features may cause a “curse of dimensionality” and corresponding overfitting. On the other hand, fusing different features together can improve the performance of emotion analysis by jointly exploiting their representation ability and complementary information. Therefore, feature selection and fusion techniques are often used, particularly for datasets with a small number of training samples.

Before being processed by the feature extractor, an image or video often undergoes preprocessing. Resizing an image to a fixed spatial resolution is straightforward, but videos pose a greater challenge due to differences in both spatial and temporal structures. It is important to determine how to combine frame- or segment-level features into a unified video-level representation. In the following, we first explore commonly used temporal information modeling in videos, followed by a summary of feature selection and fusion techniques.

In the area of temporal information modeling, there is a need to divide the input video into segments and extract keyframes to be used for feature extraction. A straightforward method is to use all frames of the entire video [207], [236], which ensures the least information loss but also results in high computational complexity. Common methods for segment sampling include fixed time interval [72], [184], [202], [211], [237], fixed number of segments [208], [222], and video shot detection [210], [238]. Sampling keyframes within a segment can be done through fixed frame sampling (uniform sampling) [72], [184], [211], [237], random sampling [208], middle frame selection [202], and mean histogram sampling (i.e., the frame with the closest histogram to the mean histogram) [191]. Recently, researchers have developed dedicated keyframe selection algorithms [210], [240]. One such method uses low-rank and sparse representation with Laplacian matrix regularization in an unsupervised manner, considering both the global and local structures [240]. Another method trains an image emotion recognition model on an additional image dataset to estimate the affective saliency of video frames [210]. Keyframes are selected based on the largest interframe difference of color histograms, after first sorting the segments (shots) based on affective saliency. Despite progress, effective and efficient selection of emotion-aware keyframes to enhance accuracy and speed remains an open challenge.

Another challenge is combining frame- or segment-level features into a unified video-level representation. Some straightforward methods include average pooling [72], [184], [207], [236], average pooling with temporal attention [208], max pooling [210], bag-of-words quantization [72], [184], LSTM [222], [238], and GRUs [237]. Another approach is to perform emotion prediction at the segment level and then combine the results using methods such as majority voting [202], [211].

Feature selection, especially for handcrafted features, is often employed when extracted features have high dimensions that contain redundant, noisy, or irrelevant information, the dimensions of different kinds of features to be fused vary significantly, or the size of the training samples is small. Some commonly used feature selection methods in evoked emotion modeling include cross-validation on the training set [183], wrapper-based selection [44], forward selection [46], and principal component analysis (PCA)-based selection [44], [46], [71], [188]. Feature selection can be combined with feature fusion, particularly in the case of early fusion [188], [197]. If the dimensions of the features being fused are similar, feature selection is carried out after fusion [188]. However, if dimensions are significantly different, feature selection is usually performed first [197] to prevent an overemphasis on high-dimensional features.

The integration of multiple features within a single modality or different kinds of representations across different modalities through feature fusion plays a crucial role in emotion recognition [22]. Model-free fusion, which operates independently of learning algorithms, is widely used and encompasses early fusion [44], [188], [190], [195], [196], [197], [198], [208], [219], [225], [229], [230], [232], [233], [237], late fusion [194], [199], [202], [211], [241], and hybrid fusion [210]. A hierarchical fusion that incorporates various feature sets at different levels of the hierarchy is also employed [187].

Model-based fusion, however, is performed explicitly during the learning process. Kernel-based fusion [72], [236] and graph-based fusion are often used for shallow learning models, whereas attention- [213], [222], [235], neural network- [184], [238], and tensor-based fusion strategies have been recently employed for deep learning models.

A transformer encoder with multihead self-attention layers is used in sentiment region correlation analysis to exploit dependencies between regions [235]. The self-attention transforms original regional features into a higher order representation between implied regions. Ou et al. [222] used a local–global attention mechanism to explore the intrinsic relationship among different modalities and for different segments. Local attention evaluates the importance of different modalities in each segment, whereas global attention captures the weight distribution of different segments. In contrast, Pang et al. [184] employed the deep Boltzmann machine (DBM) to infer nonlinear relationships among different features within and across modalities by learning a jointly shared embedding space. Cheng et al. [238] proposed an adaptive gated multimodal fusion model, which first mapped features from different modalities to the same dimension and then employed a gated multimodal unit (GMU) to find an intermediate representation. For a comprehensive survey on feature fusion strategies, please refer to [22].

5). Feature-to-Evoked-Emotion Mapping:

Based on the emotion representation models discussed in Section II-A, various evoked emotion analysis tasks can be undertaken, including classification, regression, retrieval, detection, and label distribution learning (LDL) [22]. The former three can also be classified into visual media-centric dominant emotion analyses and viewer-centric personalized emotion analyses, while the latter two are typically visual media-centric. After obtaining a unified representation, shallow or deep learning-based methods can be employed to map the features to evoke emotions.

A variety of machine learning algorithms have been utilized to learn the mapping between the unified representation after feature fusion and evoked emotions. These algorithms include naïve Bayes [44], [196], [200], logistic regression [184], [195], [199], [200], SVM and support vector regression with linear and radial basis function (RBF) kernels [46], [72], [183], [187], [188], [191], [195], [197], [199], [200], [207], [210], [211], [236], [240], linear discriminant analysis (LDA) [211], multiple instance learning [186], random forest [195], [210], decision tree [195], ensemble learning [202], a mixture of experts [237], sparse learning [192], [194], and graph/hypergraph learning [193]. There is still room for innovation in designing emotion-sensitive mapping algorithms.

The most common deep learning-based mapping is a multilayer perceptron based on one or more FC layers. The difference mainly lies in objective loss functions, such as cross-entropy loss for classification [71], [185], [210], [212], [215], [216], [217], [222], [227], [228], [229], [230], [231], [232], [233], [234], [235], [238], [240], [241], Euclidean and mean square error loss for regression [190], [195], contrastive loss and triplet loss for retrieval, and Kullback–Leibler (KL) divergence loss for LDL [189], [217], [218]. Unlike shallow learning methods, these deep learning techniques can typically be trained in an end-to-end manner. Some improvements have also been made to better explore the interactions between different tasks, and the characteristics and relationships of different emotion categories, such as through the use of polarity-consistent cross-entropy loss and regression loss [208], [226] and hierarchical cross-entropy loss [219]. More details are given in the following.

The aforementioned evoked emotion analysis tasks are interconnected. For instance, the detection of emotional regions can inform the emotion classification task, and the emotion category with the largest probability in LDL often aligns with the emotion classification results. However, a notable difference between evoked emotion modeling and traditional computer vision or machine learning lies in the existence of specific correlations among emotions. We summarize recent advancements in multitask learning and the exploration of emotion correlations in evoked emotion modeling.

Multitask learning has been shown to significantly improve performance compared to single-task learning by leveraging appropriate shared information and imposing reasonable constraints across multiple related tasks [244]. It has become popular in evoked emotion modeling, especially when training data are limited for each task. Based on the relations explored, multitask learning can be categorized into task relation learning-oriented [185], [217], [227], [228], [230], [233], testing data relation learning-oriented [192], feature relation learning-oriented [194], and viewer relation learning-oriented approaches [193]. Different evoked emotion analysis tasks are often jointly performed, such as classification and distribution learning [217], [230], classification and retrieval [185], and classification and detection [227], [228], [233]. Joint optimization of different objective losses allows models to extract more discriminative feature representations. For example, multitask shared sparse regression was proposed to predict continuous emotion distributions of multiple testing images with sparsity constraints, which takes advantage of feature group structures [192]. Different constraints across features are considered [194] to reflect their importance by a weighting strategy, which can be viewed as a special late fusion category. Rolling multitask hypergraph learning is proposed to simultaneously predict personalized emotion perceptions of different viewers where social connections among viewers are considered [193].

Exploring correlations among different emotion categories or continuous emotion values can improve evoked emotion analysis. Commonly considered emotion correlations include the following.

1. Emotion hierarchy: As emotion categories researchers attempt to model become more diverse and nuanced, the level of granularity increases [234]. As shown in Fig. 13(a), emotions can be organized into a hierarchy, which has been exploited in evoked emotion analysis. By considering the polarity-emotion hierarchy, i.e., whether two emotion categories belong to the same polarity, polarity-consistent cross-entropy loss [208] and regression loss [226] are designed to increase the penalty of predictions that have the opposite polarity to the ground truth. Hierarchical cross-entropy loss has been proposed to jointly consider both emotion and polarity loss [219]. For each level in the emotion hierarchy, one specific semantic level is mapped with local learning to acquire corresponding discrimination [234].

2. Emotion similarity: Similarities or distances between emotions can vary, with some being closer than others. For example, sadness is closer to disgust than it is to contentment. To account for these similarities, Mikels’ emotion wheel was introduced [193] [see Fig. 13(b) (left)]. Pairwise emotion similarity is defined as the reciprocal of “1 plus the number of steps required to discriminate one emotion from another.” Using Mikels’ wheel, the emotion distribution is transformed from a single emotion category [217], [230]. Chain center loss is derived from the triplet loss, with anchor-related-negative triplets selected based on emotion similarity [220]. A more accurate method of measuring emotion similarity, based on a well-grounded circular-structured representation called the emotion circle, has also been designed [218] [see Fig. 13(b) (right)]. Each emotion category can be represented as an emotion vector with three attributes (i.e., polarity, type, and intensity) and two properties (i.e., similarity and additivity), allowing for vector addition operations.

3. Emotion boundedness: Not every combination of VAD values makes sense in an emotion space. The transformation between discrete emotion states and their rough continuous values is often possible [16], [46] [see Fig. 13(c)]. For example, positive valence is linked to happiness, whereas negative valence is linked to sadness or anger even though exact boundaries may not be clear. When performing multitask learning involving both classification and regression, it is helpful to consider the constraints on these values. For example, it would not be valid to predict happiness with a negative valence. The BoLD dataset (see Section III-A3) leverages this concept to check the validity of crowdsourced annotations [51].

Fig. 13.

Illustration of different emotion correlations. (a) Emotion hierarchy: a three-level hierarchy from sentiment to primary emotion and secondary emotion [234]. (b) Emotion similarity: Mikels’ emotion wheel on the left [193] and emotion circle on the right [218]. (c) Emotion boundedness: rough transformation between discrete emotion categories and continuous emotion values in the VAD space [46].

6). Preliminary Benchmark Analysis:

We provide a summary of the image- and video-based evoked emotion classification accuracy of various representative methods, reported on the FI [71] and VideoEmotion-8 [72] datasets, respectively. The compared methods for image emotion prediction include SentiBank [200], artistic principles [188], DeepSentiBank [239], fine-tuned AlexNet [203], fine-tuned VGG-16 [204], fine-tuned ResNet-101 [205], progressive CNN [215], multilevel deep representation network (MldrNet) [231], weakly supervised coupled network (WSCNet) [227], polarity-consistent deep attention network (PDANet) [226], multilevel region-based CNN (MlrCNN) [230], stimuliaware visual emotion analysis network (SAVEAN) [219], scene-object interrelated visual emotion reasoning network (SOLVER) [232], MDAN [234], and SimEmotion [212]. Results in Table 2 indicate that: 1) fine-tuning DNNs, particularly those with more layers, outperforms handcrafted features (e.g., fine-tuned ResNet-101 [205] versus SentiBank [200]); 2) exploring local feature representations and combining representations from different levels can increase accuracy to around 75% (e.g., MDAN [234]); and 3) SimEmotion, which uses large-scale language-supervised pretraining, achieves an overall accuracy of around 80%, which remains lower than traditional computer vision tasks. The results highlight the challenges in evoked emotion prediction due to large intraclass variance and the need for further progress toward human-level emotion understanding.

Table 2.

Performance Comparison of Representative Methods for Eight-Class Evoked Emotion Classification From Images on the FI Dataset, Measured by Average Accuracy (%). *The Results Are Mainly Obtained From MDAN [234] and SimEmotion [212]

Method	Venue	Accuracy (%)
SentiBank [200]	ACM MM 2013	44.5
Artistic principles [188]	ACM MM 2014	46.1
DeepSentiBank [239]	arXiv 2014	53.2
Fine-tuned AlexNet [203]	NeurIPS 2012	58.3
Fine-tuned VGG-16 [204]	ICLR 2015	65.5
Fine-tuned ResNet-101 [205]	CVPR 2016	66.2
Porgressive CNN [215]	AAAI 2015	56.2
MldrNet [231]	NPL 2020	65.2
WSCNet [227]	CVPR 2018	70.1
PDANet [226]	ACM MM 2019	72.1
MlrCNN [230]	NEUCOM 2019	75.5
SAVEAN [219]	TIP 2021	72.4
SOLVER [232]	TIP 2021	72.3
MDAN [234]	CVPR 2022	76.4
SimEmotion [212]	TAFFC 2022	80.3

The compared methods for video emotion prediction include SentiBank [200], enhanced multimodal deep Bolzmann machine (E-MDBM) [184], image transfer encoding (ITE) [73], visual + audio + attribute (V. + Au. + At.) [72], CFN [214], V. + Au. + At. + E-MDBM [184], Kernelized features and Kernelized + SentiBank [207], visual–audio attention network (VAANet) [208], KeyFrame [210], frame-level adaptation and emotion intensity learning (FAEIL) [207], and temporal-aware multimodal (TAM) methods [213]. Results in Table 3 indicate that: 1) fusing information from multiple modalities is more effective than using a single modality (e.g., VAANet [208] versus SentiBank [200]); 2) the current best accuracy is less than 60%; and 3) effectively fusing information from different modalities and selecting key segments or frames are two challenges. Evoked emotion prediction from videos is even more challenging than image-based prediction and requires further research efforts.

Table 3.

Performance Comparison of Representative Methods for Eight-Class Evoked Emotion Classification From Videos on the VideoEmotion-8 Dataset, Measured by Average Accuracy (%). *The Results Are Mainly Obtained From VAANet [208] and TAM [213]

Method	Venue	Accuracy (%)
SentiBauk [200]	ACM MM 2013	35.5
E-MDBM [184]	TMM 2015	40.4
ITE [73]	TAFFC 2018	44.7
V.+Au.+At. [72]	AAAI 2014	46.1
CFN [214]	ACM MM 2016	50.4
V.+Au.+At.+E-MDBM [184]	TMM 2015	51.1
Kernelized [207]	TMM 2018	49.7
Kernelized+SentiBank [207]	TMM 2018	52.5
VAANet [208]	AAAI 2020	54.5
KeyFrame [210]	MTAP 2021	52.9
FAEIL [207]	TMM 2021	57.6
TAM [213]	ACM MM 2022	57.5

C. Facial Expression and Microexpression Recognition

Facial expressions play a crucial role in natural human communication and emotional perception. FER involves the automatic identification of a person’s emotional state through the analysis of images or video clips and has been a long-standing research topic in computer vision and affective computing.

1). Earlier Approaches:

Before 2012, traditional handcrafted features and pipelines were commonly used. The process generally included the following steps: detecting facial regions, extracting handcrafted features (e.g., LBP [245], nonnegative matrix factorization (NMF) [246], HOG [247]) from facial regions, and employ a statistical classifier (e.g., SVM [248]) to recognize emotions.

Readers interested in traditional methodologies are advised to refer to survey articles [249], [250], [251], [252].

Since the creation of the ImageNet dataset and deep CNN AlexNet in 2012, DNNs have demonstrated remarkable image representation capabilities. FER researchers have established large-scale datasets (e.g., EmotoNet [253] and AffectNet [78]) that provide ample training data for DNNs. Consequently, deep learning approaches have become the dominant approach in FER. A survey by Li and Deng [4] provides a comprehensive summary of deep learning-based FER methods from 2012 to 2019. During this period, researchers proposed several DNN techniques to improve FER performance, which the authors categorize as follows.

1. Adding auxiliary blocks to the typical backbone network (e.g., ResNet [205] and VGG [204]). The scoring ensemble (SSE) [254] proposed three auxiliary blocks to extract features from the shallow, intermediate, and deep layers, respectively.

2. Ensembling different models to achieve outstanding performance. For example, Bargal et al. [255] concatenated features from three different networks— VGG13 [204], VGG16 [204], and ResNet [205].

3. Designing specialized loss functions (such as center loss [256], island loss [257], and (N + M)-tuple cluster loss [258]) to learn facial features.

4. Leveraging multitask learning to learn various features from facial images. For instance, Zhang et al. [259] proposed MSCNN to jointly learn face verification and FER tasks.

Moreover, some studies concentrated on FER from video clips using spatiotemporal networks to capture temporal correlations among video frames. Some researchers [260] also used recurrent neural networks (RNNs) (e.g., LSTM), whereas others [261], [262] used 3-D convolutional networks, such as C3D [221].

2). Recent Approaches:

Although deep learning-based methods have achieved remarkable success, FER remains a difficult task due to these three challenges.

1. Annotation: Annotations of FER datasets contain much ambiguity. Each annotator subjectively evaluates facial expressions in images, leading to different annotations of the same image. Some images also have inherent ambiguity, making it difficult to assign a clear emotion label. Low image quality (such as occlusion or poor lighting) and ambiguous facial expressions exacerbate this problem. For instance, the images in Fig. 14(a) are labeled as neutral, but this label is uncertain due to annotator subjectivity and/or image quality.

2. Robustness: FER contains several sources of disturbance. Datasets are made up of individuals of varying ages, genders, and cultures. In addition, visual variations (such as human facial pose, illumination, and occlusions) commonly exist in facial images, which cause distinct appearance differences. These identity and visual variations can make it difficult for FER models to extract useful features. As illustrated in Fig. 14(b), image appearances can vary greatly due to differences in age, gender, race, facial pose, and lighting. Moreover, individuals with different identities and cultures might display their facial expressions differently, adding another challenge for FER models.

3. Subtlety: Some facial expressions are delicate, and some emotions can also be conveyed through subtle facial actions. Such fine distinctions can make it challenging to distinguish between emotions. For instance, the difference between “fear” and “disgust” images in Fig. 14(c) is nuanced. Thus, FER models must efficiently extract discriminative features to differentiate emotions.

Fig. 14.

Three challenges in FER. Images in (a) and (b) are obtained from the AffectNet dataset [78], and images in (c) are obtained from the CK+ dataset [76]. (a) Annotation. (b) Robustness. (c) Subtlety.

Most deep learning methods developed after 2019 have attempted to address these problems. Regarding the annotation problem, Zeng et al. [263] pointed out the inconsistency of emotion annotation across different FER datasets due to annotators’ subjective evaluations of emotion. To address this issue, they proposed a framework called the Inconsistent Pseudo Annotations to Latent Truth (IPA2LT), which trains multiple independent models on different datasets separately. These models may assign inconsistent pseudolabels to the same image because each model reflects the subjectivity of the corresponding dataset annotators. By comparing image’s inconsistent labels, IPA2LT estimates the latent true label. Another factor that can contribute to annotation ambiguity is uncertain facial images, such as blurry images or those with ambiguous emotions. To mitigate the effect of uncertain images, researchers have proposed various approaches. Wang et al. [264] developed a self-cure network (SCN) to identify and then suppress uncertain images. SCN uses a self-attention mechanism to estimate the uncertainty of facial images and a relabeling mechanism to adjust the labels of those images. Chen et al. [265] argued that annotating uncertain images with multilabel and intensity, rather than the one-hot label commonly used in current FER datasets, is more suitable. LDL allows models to learn label distribution (i.e., multilabel with intensity). Chen et al. [265] introduced an approach called LDL on auxiliary label space graphs (LDLALSG). Given one image, LDL-ALSG first leverages models of related tasks (such as AU detection) to find its neighbor images. Then, LDL-ALSG employs a task guide loss to let images learn a similar label distribution (i.e., multilabel with intensity) with the neighbors. She et al. [266] combined LDL and uncertain image estimation. The proposed DMUE network mines the label distribution and estimates the uncertainty of images together.

To address robustness challenges in FER, researchers have explored mitigating the impact of identity variations on recognition [267], [268]. Chen and Joo [267] investigated the influence of gender and revealed that models tend to recognize women’s faces as happy more often than men’s, even when smile intensities are the same. To overcome this issue, the authors proposed a method that first detected facial AUs and then applied a triplet loss to ensure that people with similar AUs exhibited similar expressions, regardless of gender [267]. Zeng et al. [268] demonstrated that emotion categories can introduce bias into a dataset. In some datasets, emotions (such as disgust) occur less frequently than more prevalent emotions, such as happiness and sadness, leading to poor performance of networks trained on these datasets on minority emotion classes. The authors utilized a millionimage-level facial recognition dataset (much larger than the FER dataset) and a metalearning framework to address the issue [268]. Li et al. [269] recognized the category imbalance challenge and addressed it by proposing AdaReg loss to dynamically adjust the importance of each category.

Other researchers have focused on visual disturbance variations, which lead to the robustness issue. Wang et al. [270] designed a region attention network to capture important facial regions, thus obtaining occlusion-robust and pose-invariant image features. Zhang et al. [271] used a deviation learning network (DLN) to learn identityinvariant features. Wang et al. [272] considered both identity and pose variations together, where an encoder was followed by two discriminators that classified pose and identity independently, while the encoder extracted features that were invariant to both. A classifier for expressions was then used to produce expression predictions.

To address the subtlety problem in FER, Ruan et al. [273] decomposed facial expression features into shared features that represented expression similarities and unique features that represented expression-specific variations using a feature decomposition network (FDN) and a feature reconstruction network (FRN), respectively. The authors further addressed robustness and subtlety problems by proposing a disturbance feature extraction model (DFEM) to identify disturbance features (such as pose and identify) and an adaptive disturbance-disentangled model (ADDM) to remove disturbance features extracted by DFEM and extract the discriminative features across different facial expressions. Farzaneh and Qi [274] enhanced center loss [256]. Although center loss can learn discriminative features, it can also include some irrelevant features. The proposed deep attentive center loss adopts an attention mechanism to adaptively select important discriminative features. Xue et al. [275] leveraged transformers to detect discriminative features and showed that the original vision transformer (ViT) [276] can only capture most discriminative features but neglects other features. Thus, this work proposes the multiattention dropping (MAD) technique to randomly drop some attention maps such that the network can characterize all comprehensive features except the most discriminative ones. Furthermore, Savchenko et al. [277] achieved remarkable performance on FER using the well-performing network EffientNet [278] to extract discriminative features.

3). Preliminary Benchmark Analysis:

Table 4 provides a concise overview of the recent FER method performance on the AffectNet dataset, one of the most comprehensive FER benchmarks. The table shows the emotion classification accuracy of selected representative methods that were previously discussed in the text. As seen in the table, TranFER, ADDL, and EfficientNet-B2 are among the top-performing methods on the AffectNet benchmark. The utilization of advanced network structures from general image recognition tasks in TranFER and EfficientNet-B2 highlights the importance of drawing knowledge and expertise from the image recognition field in FER. However, it is important to note that the highest reported accuracy remains below 70%. Given the over 90% top-1 accuracy achieved by state-of-the-art image recognition methods on the challenging ImageNet benchmark, this deficit suggests significant potential for improvement in FER.

Table 4.

Performance Comparison of Representative Methods for Seven-Class FER on the AffectNet Dataset, Measured by Average Accuracy (%)

Method	Venue	Accuracy (%)
IPA2LT [263]	ECCV 2018	57.31
IPFR [272]	ACM MM 2019	57.40
RAN [270]	TIP 2020	52.97
SCN [264]	CVPR 2020	60.23
DACL [274]	WACV 2021	65.20
DMUE [266]	CVPR 2021	63.11
KTN [269]	TIP 2021	63.97
TranFER [275]	CVPR 2021	66.23
Face2Exp [268]	CVPR 2022	64.23
ADDL [279]	IJCV 2022	66.20
EfficientNet-B2 [277]	TAFFC 2022	66.34

4). Microexpression Recognition:

The methods described above are for recognizing facial expressions. However, individuals may consciously exhibit certain facial expressions to conceal their authentic emotions. In contrast to conventional facial expressions, which can be deliberately controlled, microexpressions are fleeting and spontaneous and can uncover an individual’s genuine emotions. A microexpression is brief in duration and can be imperceptible with the naked eye. MER often requires high frame-rate videos as input and the development of spotting algorithms to isolate temporally the microexpression within videos. A recent survey gives a more in-depth introduction to developments in MER [7].

D. Bodily Expressed Emotion Understanding

In everyday life, people express their emotions through various means, including via their facial expressions and body movements. Recognizing emotions from body movements has some distinct advantages over recognizing emotions from facial images for many computer and robotic applications.

1. In crowded environments where facial images may be obscured or lack sufficient resolution, body movements and postures can still be reasonably estimated. This context is particularly important in robotic applications where the robot may not be close to all individuals in its environment.

2. Due to privacy concerns, facial information may be inaccessible. For example, in some medical applications, sharing of facial images or videos is restricted to protect sensitive patient identity information.

3. Incorporating body expressions as an additional modality can result in more accurate emotion recognition compared to using facial images alone. For example, when the person is not facing the camera, the camera cannot obtain a frontal view.

Psychologists have conducted extensive studies to examine the relationship between body movements and emotions. Research suggests that body movements and postures are crucial for understanding emotion, encoding rich information about an individual’s status, including awareness, intention, and emotional state [114], [280], [281], [282], [283]. Several studies, including one published in Science, found that the human body may be more diagnostic than the face for emotion recognition [283], [284], [285].

However, the field of BEEU in visual media has progressed relatively slowly. Unlike FER, which has seen significant progress with deep learning methods since 2013, most BEEU studies relied on traditional, handcrafted features until 2018. The bottleneck for BEEU is the scarcity of large-scale, high-quality datasets. As mentioned in Section III-A3, collecting and annotating a dataset of bodily expressions with high-quality labels are extremely challenging and costly. Understanding and perception of emotions from concrete observations are heavily influenced by context, interpretation, ethnicity, and culture. There is often no gold-standard label for emotions, especially bodily expressions. Prior to 2018, research on bodily expression was limited to small, acted, and constrained lab-setting video data [286], [287], [288], [289]. These datasets were insufficient for deep learning-based models that required a large amount of data. The recent BoLD dataset by Luo et al. [51] introduced in Section III-A3 is the largest BEEU dataset to date. It contains over 10 000 video clips of body movements with high-quality emotion labels. In addition, Randhavane et al. [74] proposed the E-Walk data, including 1136 3-D pose sequences extracted from raw video clips. As a result, computer vision research is increasingly focused on emotion recognition through body movement recordings.

Because static body images alone can hardly inform a person’s emotions, BEEU often considers human video clips. According to input modality of the models, BEEU methods can be classified as pixel-based and skeleton-based. Pixel-based methods use entire video clips, whereas skeleton-based methods first extract 2-D/3-D pose information and then feed it into the models. Some BEEU works, which focus on movement when walking, are known as gait-based. Herein, we refer to them as skeleton-based as well because gait is also a 2-D/3-D pose. Figs. 15 and 16 illustrate the two kinds of methods.

Fig. 15.

Skeleton-based BEEU takes the 2-D/3-D pose sequence of the target human instance as input.

Fig. 16.

Pixel-based BEEU uses raw videos as input. The target human is cropped, and the resulting frames are input into a video analysis neural network.

1). Skeleton-Based Methods:

Our review of past publications shows a greater adoption of skeleton-based approaches compared to pixel-based. This is due to two reasons. First, skeleton data, consisting of sequences of 2-D/3-D joint coordinates, require less engineering effort to process than video clips. A straightforward method is to feed coordinates into a machine learning classifier (e.g., SVM) for direct emotion prediction results. Second, improved MoCap systems enable researchers to easily collect accurate 3-D poses from individuals walking in laboratory settings.

In early stages of skeleton-based BEEU research, a conventional approach was followed, which entailed the extraction of low-level features from 2-D/3-D pose sequences and subsequent utilization of a machine learning classifier to predict emotion. These features were categorized as follows.

1. Frequent domain features, which transform temporal information into a frequency domain, are obtained through Fourier transformation. For instance, Li et al. [290] used Fourier transformation to convert 3-D pose sequences into frequency domain features, which were then classified using LDA, naïve Bayes, decision tree, and SVM algorithms.

2. Motion features, characterize the movement of body joints, include joint velocity and acceleration [51].

3. Geometry features, which describe self-transformation of the body, encompass angles of specific skeletons and distance between certain joints, among others. In particular, Crenn et al. [291] combined motion features, geometry features, and frequent domain features into an SVM classifier.

Luo et al. [51] compared traditional and deep learning-based methods using the BoLD dataset and developed the Automated Recognition of Bodily Expression of Emotion (ARBEE) system. A traditional machine learning approach was designed, where motion and geometry features were extracted from 2-D pose sequences, and a random forest classifier was employed to categorize emotions. In addition, an ST-GCN was utilized to train and evaluate the BoLD dataset. The result indicated that carefully designed traditional machine learning methods outperformed the ST-GCN model from scratch.

Since the development of ARBEE, various deep learning-based methods have emerged. These methods can be broadly categorized into three groups based on the type of neural network used: RNN, graph neural network (GNN), and CNN.

Randhavane et al. [74] used an LSTM network, which is a type of RNN, to extract temporal features from a 3-D pose sequence, which were then concatenated with handcrafted features for classification. Bhattacharya et al. [292] leveraged a semisupervised technique to improve the performance of an RNN model. The work consisted of a GRU (a kind of RNN model) for feature extraction from a 3-D pose sequence, followed by an autoencoder with both encoder and decoder components. During training, when the input data were labeled with emotions, the classifier after the encoder produced the emotion prediction, and the decoder reconstructed the 3-D pose. If the input lacked emotion labels, only the decoder was used for reconstruction.

Bhattacharya et al. [293] adopted ST-GCN [294] to classify emotion categories from 3-D pose sequences. To increase the size of the training set, the authors used a conditional variational autoencoder (CVAE) to generate some synthetic data. Banerjee et al. [295] combined GCN and NLP techniques to achieve zero-shot emotion recognition, which entailed recognizing novel emotion categories not seen during training. The authors used ST-GCN to extract visual features from the 3-D pose sequences and used the word2vec method to obtain word embeddings from emotion labels. An adversarial autoencoder was used to align visual features with word embeddings. During inference, the system searched for the emotion label that best matched the output visual feature.

Inspired by the success of image recognition, some studies have attempted to convert skeleton sequences into images. Narayanan et al. [296] embedded 3-D pose sequences into images and then utilized a CNN for classification. Hu et al. [297] employed a two-stream (TS) CNN, where one stream directly embedded the 3-D pose into an image, and the other stream converted handcrafted features from the 3-D pose into another image. The two CNNs were integrated using transformer layers.

2). Pixel-Based Methods:

Pixel-based networks for BEEU require video clips as input. Due to an increased redundancy in video clips compared to 2-D/3-D pose sequences, these networks necessitate larger training datasets to extract distinctive features. With the availability of the large-scale BEEU dataset, researchers have focused on pixel-based methods. Because the well-established field of action recognition, which also uses videos to analyze human behavior, has many similarities to BEEU, current BEEU research often incorporates networks from action recognition.

The ARBEE study benchmarked the performance of various action recognition networks on the BoLD dataset [51]. They cropped human body regions from raw videos as the input and then fed them into different methods, including a traditional handcrafted feature method and three deep learning methods (TS [298], I3D [138], and TSN [299]). Both TS and TSN consisted of a TS network, where one stream processed the RGB image sequence and the other processed the optical flow sequence. TS and TSN used 2-D convolutional networks, whereas I3D employed a 3-D convolutional network. Results indicated that all deep learning methods significantly outperformed the traditional handcrafted feature method. Among the three deep learning methods, I3D performed worse than TS and TSN, potentially due to its requirement for more training data to reach a comparable level of performance. The BoLD dataset is not as extensive as action recognition datasets, such as Kinetics-400 [300], and thus, I3D trained on BoLD could not fully exhibit its capability as it could on Kinetics-400. At the same time, TS and TSN produced similar results.

The ARBEE study also evaluated the impact of a person’s face on BEEU model performance [51]. They designed an ablation study with three different video inputs: the whole body (crop the whole human body from the raw video clips), just the face (only crop the face part), and the body without the face (crop the whole human body but mask the facial region) [51]. Results showed that using either the face or body alone was comparable to using the whole body. This demonstrated that both the face and body contributed significantly to the final prediction. Although the whole body setting of the TSN model outperformed the separate models, it did so by combining facial and body emotions.

Most BEEU research has followed ARBEE to continue pixel-based approaches. Because ARBEE cropped human body regions as input, a direct idea to improve upon ARBEE is to explicitly utilize facial images and background images as input as well. Recent studies [301], [302] used an extra network to extract context information from whole images and then fused the context features with features extracted from body images. Some research adopts an additional network with facial images as input [303]. Moreover, inspired by the cutting-edge vision-language research of CLIP [304], Zhang et al. [305] developed EmotionCLIP, a contrastive vision-language pretraining paradigm to extract comprehensive visual emotion representations from whole images, encompassing both context information and human body information simultaneously. Because EmotionCLIP uses only uncurated data, it addresses the challenge of data scarcity in emotion understanding. EmotionCLIP outperforms state-of-the-art supervised visual emotion recognition methods and competes with many multimodal approaches across various benchmarks, demonstrating its effectiveness and transferability.

Certain studies delve deeply into the analysis of body gestures and movement in the context of BEEU. ARBEE currently provides only a generalized emotion label for entire video clips, lacking specific descriptions for human body gestures or movements. In contrast, Liu et al. [75] and Chen et al. [306] introduced datasets for microgesture understanding and emotion analysis, featuring detailed body gesture labels, such as crossed fingers, for each human movement clip. These enriched datasets have the potential to improve machines’ understanding of emotions conveyed through gestures. As mentioned in Section III-E, LMA is a comprehensive method for describing human movement. Wu et al. further advanced BEEU by presenting an LMA dataset that provides accurate LMA labels for human movements [307]. Wu et al. [307] incorporated a novel dual-task model structure that simultaneously predicts emotions and LMA labels, achieving remarkable performance on the BoLD dataset.

3). Preliminary Benchmark Analysis and Current Directions:

Table 5 presents the results of various BEEU methods using the BoLD dataset, with performance measured by mean Average Precision (mAP) across 26 emotional categories. Results indicated that pixel-based methods outperformed skeleton-based ones, which was not surprising given RGB images contained more information. Despite significant progress in BEEU, performance remains relatively low, with mAP scores below 25%.

Table 5.

Performance Comparison of Different Methods for Bodily Expressive Emotion Understanding on the BoLD Dataset

Method	Venue	mAP (%)
Skeleton-based:
ST-GCN [263]	AAAI 2018	12.63
Random Forest [51]	IJCV 2020	13.59
Pixel-based:
TS [298]	NeurIPS 2014	17.04
TSN [299]	ECCV 2016	17.02
I3D [138]	CVPR 2017	15.34
Filntisis et al. [302]	ECCVW 2020	17.96
Pikoulis et al. [303]	FG 2021	21.87
EmotionCLIP [305]	CVPR 2023	22.51
Wu et al. [307]	arXiv 2023	23.09

As BEEU is a relatively new area in computer vision, we explore its potential future directions inspired by the trajectory of related areas, such as action recognition and FER. First, existing bodily expression datasets are not sufficiently large. BoLD contains only thousands of instances, which is much less than contained in action recognition datasets, such as Kinetics-400. The BoLD team is developing larger and more comprehensive datasets to satisfy data requirements of deep learning methods, but those expansions are not yet complete. Second, existing approaches are largely based on action recognition methods without leveraging deep affective features. Previous work only applied low-level features, neglecting characteristics of concurrent body movement. Third, annotation ambiguity, similar to FER, is also challenging. Fourth, any method that segments body regions must account for changing relationships among body sections (i.e., Shape Change in LMA), which is crucial for bodily expressed emotion, and may be particularly relevant to the dimensional emotion model because Shape Change reveals approach and retreat. Section V discusses technological barriers more broadly and in greater depth.

E. Integrating Multiple Visual Input to Model Expression

We discussed earlier how facial and body images have been used as separate inputs to identify human emotions. As was already established, the context of people in a scene also contributes to inferring their emotions. Humans synthesize all visual information to produce emotional determinations. Naturally, computers are also capable of making identification by combining all visual inputs, including facial images, body images, and context information.

As illustrated in Fig. 17, recent work used multistream networks to fuse different visual inputs. Kosti et al. [136] and Lee et al. [81] developed initial approaches by adopting a TS network. Specifically, Kosti et al. used body images and entire images as input to extract context information and body features separately. Lee et al. adopted faceless images (i.e., cropped the human face out of the entire image) and facial images as input, and a feature fusing network that dynamically fused context and facial features based on their significance.

Fig. 17.

Multistream network extracts features from different inputs, which can be various visual inputs (such as entire and body images) or inputs from other modalities (such as audio). The fusion module combines those features.

Subsequent research has taken two paths. Some researchers have explored more kinds of visual inputs. For example, Mittal et al. [137] demonstrated that a depth map can indicate how people interact with each other. The estimated depth map from the raw image served as one input. In addition, this work used three other inputs–the facial image, the 2-D pose, and the bodiless (i.e., cropping the human body out of the entire image) images—to extract facial information, body posture information, and context information separately. This was a four-stream network overall. Studies [301], [302], [303] have also used a three-stream network with the entire, facial, and body images as input.

Other researchers focused on more effective fusion of visual features. Existing multistream approaches in typical emotion recognition adopt a simple fusion strategy, i.e., ensemble the predictions from each stream [301], [302], [303]. To improve upon this, Le et al. [308] proposed the global–local attention (GLA) module to enhance interaction between facial and context features.

F. Multimodal Modeling of Expressions

Emotions can be conveyed and perceived not just through visual signals but also through text and audio. To effectively process multimodal signals, however, is nontrivial. With advancements in deep learning, new technical approaches to this problem have emerged, particularly the vision-and-language model [304], [309]. Similar to methods discussed in Section IV-E, multimodal approaches also follow a multistream pipeline, as illustrated in Fig. 17. These techniques use independent networks to extract features from different inputs of multiple modalities and fuse these features with a fusion network.

Multimodal approaches typically utilize backbone networks, such as BERT [310] and ResNet, to extract text and audio features. The visual feature extraction process, however, differs and is typically performed using one of the following three methods.

1. Region features: A detection network is employed to identify regions of interest (ROIs), from which features are extracted.

2. Grid features: A backbone network, such as ResNet 101, is used to extract features for the entire image.

3. Patch projection: The image is split into patches, and a linear layer is used to generate a linear embedding, as described in ViT [276].

For the fusion process, a simple approach is to directly ensemble the final prediction of the different networks. Another method involves concatenating feature maps from different modality networks and using a single network to fuse the features. Yet, another approach is to use multiple networks to process different features, with interactions between the different networks.

Some research in emotion recognition focuses on integrating audio and visual inputs. For example, Tzirakis et al. [311] attempted to fuse audio signals with facial images. They incorporated an audio stream network to extract audio features, followed by an LSTM network to fuse facial and audio features. Antoniadis et al. [312] combined audio signals with multiple visual modalities, including facial image, body image, and context image. In contrast to using CNN, Shirian et al. [313] utilized a GCN structure to process audio, facial, and body inputs, and then used a pooling layer to process the fused features.

With the rapid advancement of NLP, there has been a surge in research on multimodal sentiment analysis, which incorporates text, audio, and visual input. Survey papers by Zhao et al. [22] and Gandhi et al. [70] provided comprehensive overviews of advancements in multimodal sentiment analysis techniques. Noteworthy methods from recent years include the development of a multitask network by Akhtar et al. [314], which used text, acoustic, and visual frames of videos as input with intermodal attention modules to adjust the contribution of each modality. Most existing methods adopt multimodel emotion labels as the supervision, ignoring unimodal labels. Yu et al. [315] proposed a label generation module that generated unimodal labels for each modality in a self-supervised manner, enabling a multitask network to train all unimodal and multimodal tasks simultaneously. Jiang et al. [316] adopted several baseline models for each modality input and used PCA to find the optimal feature for the modality. Furthermore, an early fusion strategy combined all features. Yang et al. [317] considered the consistency and differences among various modalities in a unified manner, using common and private encoders to learn modality-invariant and modality-unique features, respectively, across all modalities. With a similar motivation, Zhang et al. [318] proposed a cascade and specific scoring model to represent the interrelationship and intrarelationship across different modalities. Zhang et al. [319] used reinforcement learning and domain knowledge to process fused features from multiple modalities in conversational videos. Through reinforcement learning, dueling DQN predicted the sentiment of the target sentence based on features of previous sentences. Moreover, information in the first several sentences was used as domain knowledge for subsequent predictions. Mittal et al. [320] incorporated multiple visual inputs, resulting in the inclusion of five distinct types of input: face, body, context, audio, and text.

To show recent advancements in multimodal sentiment analysis, Table 6 presents the performance of selected representative methods on the widely used benchmark, the MOSEI dataset. Performance is evaluated through mean absolute error (MAE) and binary accuracy (Acc-2) metrics. The top-performing methods were Self-MM [315] and FDMER [317]. Self-MM utilizes a multitask network to train all modalities simultaneously, whereas FDMER employs modules to capture shared and individual features across modalities. The essence of multimodal sentiment analysis remains the optimization of relationships between different modalities.

Table 6.

Performance Comparison of Representative Methods for Multimodal Sentiment Analysis on the MOSEI Dataset. *The Results Are Mainly Obtained From FDMER [317]

Method	Venue	MAE	Acc-2 (%)
TFN [322]	EMNLP 2017	0.593	82.5
LMF [323]	ACL 2018	0.623	82.0
MFM [324]	ICLR 2019	0.568	84.4
ICCN [325]	AAAI 2020	0.565	84.2
MISA [326]	ACM MM 2020	0.555	85.5
Self-MM [315]	AAAI 2021	0.530	85.2
FDMER [317]	ACM MM 2022	0.536	86.1

In addition, some research centers on fusing multimodal features. For example, Cambria et al. [321] presented sentic blending to fuse scalable multimodal input. In a multidimensional space, it constructed a continuous stream for each modality, which depicted the semantic and cognitive development of humans. The streams of different modalities were then fused over time.

V. MODELING EMOTION: SIGNIFICANT TECHNOLOGICAL BARRIERS

Despite advancements propelled by deep learning and big data, solving the problem of emotion modeling is nowhere in sight. In this section, we share insights on some of the most significant technological barriers in computer vision (see Section V-A), statistical modeling and machine learning (see Section V-B), AI (see Section V-C), and emotion modeling (see Sections V-D–V-G) that hinder progress in the field. At present, there are no straightforward solutions to these issues. Some fundamental technologies in multiple related fields must be further developed before substantial progress can be made in emotion understanding. EQ plays a significant role in our cognitive functions, such as decision-making, information retrievability, and attention allocation. AEI will likely become an integral part of future-generation AI and robotics. To put it succinctly, AEI based on visual information is a “Holy Grail” research problem in computing.

A. Fundamental Computer Vision Methods

1). Pretraining Techniques:

The task of annotating emotion is a time-consuming process. Acquiring a large-scale dataset for FER or BEEU on the level of ImageNet is difficult. To completely overcome the challenge of emotion recognition, DNNs must acquire sufficient representation capability from extensive datasets. This gap can be bridged by using pretraining techniques, in which a model is initially trained on a massive dataset for an upstream task and then fine-tuned for a downstream task. During pretraining, the model is expected to learn emotion recognition capabilities from the upstream task. The choice of upstream task is critical because it determines the amount of emotion-related capabilities that the model can acquire.

The widely adopted pretraining strategy is to train the model on ImageNet for image classification tasks. However, the ARBEE team has demonstrated that this approach does not significantly enhance the performance of the BEEU task [51].

Our hypothesis is that image classification and emotion recognition require different types of discriminative features to be extracted. This observation reflects the fact that, while most individuals can recognize common objects, a portion of the population is unable to discern subtle emotions. In light of the current state of computer vision, two upstream tasks have been promising for pretraining: self-supervised learning (SSL) [327], [328] and image–text retrieval [304]. Regarding SSL, our concern is that, despite pretraining with state-of-theart SSL techniques, the model still required large downstream datasets in the fine-tuning phase. For instance, even after pretraining with SSL, action recognition tasks required fine-tuning on large-scale Kinetics-400 for several epochs to achieve remarkable performance [329]. As for image–text retrieval, it is crucial to carefully design the text prompts to effectively bridge an emotion recognition model with a pretrained image–text model. Further research is needed to determine how pretraining techniques can substantially enhance emotion recognition.

2). Comprehensive and Robust Body Landmarks:

The effectiveness of current 2-D pose estimation methods has led to their widespread usage in downstream tasks, such as action recognition [97], [294]. However, for emotion recognition, these methods face significant limitations. First, body landmarks used in pose estimation are insufficient for the analysis of subtle emotions, as they do not account for all relevant movements. For instance, many methods only provide one landmark on the chest, and this limitation prevents skeleton-based emotion recognition algorithms from leveraging information regarding chest expansion or contraction, which is a common indicator of a person’s confidence. Similarly, a person can express emotions through finger movements, which are not captured by most pose-estimation algorithms due to being too fine-grained. Developing an emotion-specific pose estimation method would require constructing a new, large-scale annotated dataset with additional landmarks, which can be a time-consuming and costly process. Second, 2-D pose estimation results can be noisy due to the jitter errors [330]. Although these errors may have a minimal effect on metrics of pose estimation benchmarks, they can significantly impact the understanding of subtle bodily expressions, which demands substantially higher precision of landmark locations [51]. Given that pose estimation serves as the starting point for skeleton-based BEEU methods, any errors in human pose can have a ripple effect on a final emotion prediction.

3). Accurate 3-D Pose or Mesh in-the-Wild:

The integration of 3-D pose information has the potential to substantially enhance BEEU algorithms because the extra dimension allows for a better understanding of movements in 3-D space. In other words, BEEU algorithms could depend on precise 3-D pose or mesh as input, which should enhance its overall accuracy. While accurate 3-D poses can be obtained through MoCap systems for laboratorycollected data, obtaining accurate 3-D poses for in-thewild data is challenging. It is impossible to set up MoCap to capture the 3-D pose of a person because the area of movement is often too large for the placement of MoCap cameras. Furthermore, existing 3-D pose or mesh estimation approaches have poor performance when applied to in-the-wild images. The difficulty of collecting 3-D annotations for in-the-wild images has resulted in a lack of large-scale, high-quality 3-D pose datasets. Current 3-D pose models are heavily reliant on lab-collected 3-D datasets for training, which are subject to a distinct domain shift from in-the-wild images. In a laboratory setting, the lighting and environment are fixed, and the appearance and posture of individuals are monotonous. Conversely, in-the-wild images exhibit significant variations in these factors, making it difficult for current models to generalize well to in-the-wild images.

B. Fundamental Statistical Modeling and Learning Methods

The field of emotion modeling has seen a shift from conventional computer vision and machine learning techniques (e.g., [46]) to deep learning-based methods (e.g., [51] and [331]). For many other computer vision problems, deep learning has shown its power in substantially advancing the state of the art compared to other machine learning methods. However, some intrinsic limitations of deep learning continue to limit progress in the field of emotion modeling. In this context, we aim to highlight some fundamental data-driven AI capabilities that, if developed, could drive the field forward.

1). Modeling a Complex Space With Scarce Data:

A person’s emotions and behaviors are influenced by various factors, such as personality, gender, age, race/ethnicity, cultural background, and situational context. Modeling this multidimensional, complex space requires a massive amount of data to sample it sufficiently and properly. Despite the potential to collect data from the Internet, manually annotating such a large dataset for emotion can be prohibitively expensive. This conundrum challenges machine learning researchers to find ways to learn meaningful information with limited data collection.

A relatively more specific challenge facing researchers developing datasets for emotion research is determining the appropriate amount of data needed to obtain meaningful results through machine learning. Emotion data, when collected from public sources, tend to be naturally imbalanced, with some emotion categories (e.g., happiness and sadness) having a much higher number of samples than others (e.g., yearning and pain) [51]. Unlike in typical object recognition where metadata (e.g., keywords and image file names) can aid in crawling a reasonably balanced dataset, we usually cannot determine or estimate the emotion label for a piece of visual data from just the available metadata. Making a balanced, representative dataset depends on crawling or collecting a very large dataset, in the hope of obtaining a sufficient number of samples for the less prevalent categories. Such a laborious process greatly increases the cost of data collection. A potential solution to this challenge is to use AI to guide a more efficient data collection process so that limited annotation resources can be used to achieve maximum benefit for AI training.

2). Explainability, Interpretability, and Trust:

The ability of AI systems to provide clear and interpretable explanations of their reasoning processes is particularly crucial in critical applications, such as healthcare, because endusers, who may not have a data science background, need to have confidence in the AI system’s quantitative findings. There is ongoing research in this area within the broader field of AI, with some progress being made [332], [333], [334]. However, it is widely considered an open challenge.

The task of emotional understanding presents unique challenges in terms of explainability, interpretability, and trust. First, emotions are highly subjective, and AI models typically learn from a general population’s responses based on data collected from numerous human subjects. The AI’s ability to explain or interpret results, thus, is limited to a general perspective, which may naturally be considered unreasonable to a certain extent by virtually any particular individual, especially if their opinions on emotions frequently deviate from the norm.

Second, causal relationships are often more significant in emotion-related applications. For example, in determining why a sunset scene evokes positive emotions in many viewers, it is important to understand whether it is the sunset scenario, the orange hues, the gradient of colors, the horizontal composition, or some other properties that cause viewers to feel positive emotions. Without this knowledge, algorithms developed may behave unpredictably in many situations, acting as black boxes. Currently, methods for causal discovery are often not scalable to high-dimensional or complex data and are sensitive to sparse data. Therefore, there is a need to develop appropriate frameworks and methods for high-dimensional and complex causal discovery specifically tailored to the understanding of emotions.

Third, overlapping semantics among emotion labels add complexity to interpreting results. The BoLD dataset has shown, using video clip annotations, that several pairs of emotion labels are highly correlated [51]. Examples are pleasure and happiness (correlation = 0.57), happiness and excitement (0.40), sadness and suffering (0.39), annoyance and disapproval (0.37), sensitivity and sadness (0.37), and affection and happiness (0.35). Even in the dimensional VAD model, researchers have detected correlations between valence and dominance (0.359) and between arousal and dominance (0.356). Current data-driven approaches can often provide a probability score for each emotion label in the classification system. Although it remains common practice to sort these scores to determine the most likely emotion, it is not always clear what a mixture of scores represents. For example, what does it mean to have 80% happiness, 60% excitement, and 50% sadness? Is it reasonable to just assume that the data point should be classified as happiness? Or, should it be considered a mixture of happiness, excitement, and sadness all at the same time, perhaps as representative of humans often complex emotional states? Or, could it be some partial combination of these categories? More fundamental statistical learning methods will likely be needed to address this issue in a principled way.

One potential strategy in automated emotion recognition is to uncover useful patterns from a large amount of data, with the aim of gaining a deeper understanding of emotions beyond simple classification. Such findings may either support or challenge existing psychological theories or inspire new hypotheses. However, black-box machine learning models can provide little insight into why a decision is made, which restricts our ability to gain knowledge through automated learning. As a result, there has been a recent and growing focus on developing interpretable machine learning.

There are two primary approaches to interpretable machine learning. The first is known as model-agnostic approaches [335], [336] and involves using relatively simple models, such as decision trees and linear models, to approximate locally the output of a black-box model, such as DNN. A fundamental issue with this approach is that the explanation is local, usually a neighborhood around every input point. This outcome raises the question of whether such limited explanations are truly useful. Because the power of explaining a phenomenon implies the capability to reveal underlying mechanisms coherently for a wide range of cases, a severe lack of generality in interpretation undermines this goal.

In the second approach, the emphasis is on developing models that are inherently interpretable. Classic statistical models, due to their simple structure, are often inherently interpretable. However, their accuracy is often significantly lower compared to top-performing black-box models. This drawback of classic models has motivated researchers to develop models with enhanced accuracy without losing interpretability. For example, Seo et al. [337] proposed the concept of cosupervision by DNN to train a mixture of linear models (MLM), aimed at filling the gap between transparent and black-box models. The idea is to treat a DNN model as an approximation to the optimal prediction function based on which augmented data are generated. Clustering methods are used to partition the feature space based on the augmented data, and a linear regression model (or logistic regression for classification) is fit in each region. Although MLMs have existed in various contexts, they have not been widely adopted for high-dimensional data because of the difficulties in generating a good partition. The authors overcame this challenge by exploiting DNN. They also developed methods to help interpret models either by visualization or simple description. Advances in the direction of developing accurate models that are directly interpretable are valuable for emotion recognition.

3). Modeling Under Uncertainty:

Data available for modeling human emotions and behaviors are often suboptimal, leading to various challenges in accurate modeling. For instance, in the case of BEEU, the available data are often limited to partial body movements (e.g., only the upper body is visible in the video) and may include occlusions (i.e., certain body parts are blocked from view). In addition, the automated detection of human body landmarks in video frames is not always precise. An interesting research direction will be to establish accurate models based on incomplete and inaccurate data. Furthermore, it is important to quantify uncertainty throughout the machine learning process, given that such uncertainties are present at each step. This is necessary in order to effectively communicate results to users and help them make informed interpretations.

4). Learning Paradigms for Ambiguous Concepts:

The utilization of traditional learning frameworks, such as DNN, SVM, and classification and regression trees (CARTs), may prove to be limited in tackling complex problems, such as modeling not-so-well-defined concepts (e.g., emotions and movements). Unlike more concrete objects, such as cars and apples, there may not always be a clear ground truth for the expression of emotions in a video clip. The ambiguity of establishing a ground truth can be influenced by subjective interpretations of annotators, leading to different viewpoints and no correct interpretation.

Although these traditional learning frameworks are optimized for well-defined concepts, their straightforward application to emotion recognition may lead to incorrect assumptions, such as assuming a majority annotator holds the ground truth. In the case of BEEU, the creation of a mid-layer of movement concepts between pixel-level information and high-level emotions brings forth another challenge. Many movement classes are qualitative in nature (e.g., dropping weight, rhythmicity, strong, light, or smooth movement, and space harmony), further complicating the development of accurate models. The need for advanced learning paradigms that can effectively tackle the challenges of modeling complex, ambiguous concepts, such as emotions and movements, has become apparent.

5). AI Fairness and Imbalanced Datasets:

In the collection of data regarding human emotions and behaviors, it is not uncommon for certain demographic or emotion/behavior groups to have smaller sample sizes compared to others. For instance, data about white people are often more abundant than data about black people, and data about happiness and sadness are typically more abundant than data about esteem, fatigue, and annoyance. Similarly, data across cultural groups vary significantly. Despite being a current area of research in AI [338], tackling such inadequacies is particularly complex in emotion understanding due to its unique characteristics.

6). Efficiency of AI Models:

For some applications, emotion recognition algorithms must be fit onto robots or mobile devices that are limited by their onboard computing hardware and battery power. Complex problems that require multiple AI algorithms and models to work in concert can be difficult to address in real time without high-performance GPU/CPU computing hardware. Therefore, it is crucial to simplify the mathematical models while maintaining their accuracy.

Machine learning researchers have derived techniques to simplify the model, including channel pruning techniques. In an earlier attempt, Ye et al. [339] proposed a method for accelerating the computations of deep CNNs, which directly simplifies the channel-to-channel computation graph without the need to perform the computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structure sparse. There have been numerous recent studies on pruning neural networks [340], [341].

C. Fundamental AI Methods

These are some fundamental AI components for creating an effective human–AI interaction and collaboration system that can have a significant impact on relevant domains. Simply increasing the amount of data alone may not be sufficient to counterbalance the deficiencies in these fundamental areas of AI. In the following, we will explore some of these challenges in greater detail.

1). Decision-Making Based on Complex Information:

In real-world applications, such as mental healthcare, we often need to incorporate multiple sources of information (e.g., nonverbal cues, verbal cues, health record information, and observations over time), and different people may have different sets of information inputs (e.g., some patients may not have observations over time, whereas others may not have detailed health records). For example, when a patient with depression visits a clinic, a combination of behaviors, speech, and health record information can be used to make informed decisions. To effectively connect all relevant information, research is required to develop a comprehensive framework. This fundamental area of AI research has the potential to impact many AI applications.

2). Integrative Computational Models and Simulations for Understanding Human Communication and Collaboration:

Current research primarily concentrates on analyzing individuals, such as deducing an individual’s emotional state from their behavior. However, emotional expressions play a significant role in interpersonal communication and collaboration. The emotional expression of one individual can have a significant impact on the emotions and behavior of those around them. Thus, there is a need for the development of integrative computational models and simulations to investigate the emotional interactions among individuals. As the problem of understanding individual emotions remains unresolved, a comprehensive approach to study interpersonal emotional interactions must consider the uncertainties present in individual-level emotion recognition.

In addition, when robots are integrated into the interaction, the distinction between human–human and human–robot interactions (HRIs) must be weighed. The external design and behavior of robots can vary greatly and offer a much wider range of possibilities than humans, making it challenging to sample comprehensively the potential space of robot interactions. For example, a robot can take on various forms, such as a human, animal, or a unique entity with its own distinct personality. Similarly, robots can move in ways that go beyond human or animal-like motions. While research can be conducted with limitations on specific types of robots, the results may not be applicable to other forms of robots.

3). Incorporation of Knowledge and Understanding of Cognitive Processes:

When humans interpret emotions through visual media, they rely on their accumulated knowledge and experience of the world around them. However, the same behavior can be interpreted differently depending on the context or situation. Data-driven emotion recognition approaches require vast amounts of training data to be effective, but the countless possible scenarios and contexts can make obtaining such data challenging. To address this issue, AI researchers have been exploring the development of common-sense knowledge representations [342]. Integrating these advances into an emotion recognition framework is an important area of research. In addition, cognitive scientists have gained valuable insights into human cognitive processes through experiments, and incorporating these findings into the design of a next-generation emotion recognition system could be a key to its success.

The expression and interpretation of emotions by humans involve various levels and scales, ranging from basic physiological processes impacting a person’s behavior to sociocultural structures that shape their knowledge and actions. Currently, multilevel and multiscale analyses of emotions are a rare occurrence in AI due to the complexity it entails.

4). Prediction of Actions:

Most of the current research in the field focuses on emotion recognition using visual information. However, for certain human–AI interaction applications, it is necessary to not only recognize emotions based on past data but also proactively gather information in real time and make predictions about future events. For example, in the event of a heated argument between two human workers, a robot may need to move closer to better understand the situation, and based on changes in the behavior of the workers, it may need to predict any potential danger and take action to resolve the issue. This might entail alerting others or attempting to redirect the attention of involved parties. Research is required to map emotion recognition with appropriate actions, even while acknowledging the inherent uncertainty involved in the process of emotion understanding. Such a process is more nuanced than typical scene understanding.

D. Demographics

Unlike many computer vision problems, such as object detection, when it comes to emotion, we simply cannot ignore the effect of demographics. Emotional responses can vary greatly among different demographic groups, including gender, age, race, ethnicity, education, socioeconomic status, religion, marital status, sexual orientation, health and disability status, and psychiatric diagnosis. Existing machine learning-based recognition technologies are not equipped to effectively handle such a vast array of demographic factors. In the absence of sound methods for evaluation, our brains tend to resort to shortcuts that may not be dependable, in order to conserve energy and navigate difficult situations where solid judgment is not present. To fill the gap, we often rely on stereotypes, heuristics, experience, and limited understanding to gauge emotions in other demographic groups, which may be unreliable. However, when we design AI systems, such shortcuts are not acceptable as mistakes made by machines can have disproportionately negative consequences for individuals and society as a whole. Addressing the issue of demographics in automated emotion understanding will likely remain a persistent challenge in the field.

E. Disentangling Personality, Function, Emotion, and Style

A person’s behavior, captured by imaging or movement sensors, is a combination of several elements, including personality, function, emotion, and style. Even if we can find solutions to the problems that we have mentioned earlier, separating these elements so that emotional expression can be properly analyzed will remain challenging. For example, the same punching motion would convey different emotions in a volleyball game versus during an argument between two individuals. This single example highlights the need for AI to first understand the purpose of the movement. For the same function, with the exact sequence of movements, two persons with very different personalities and contexts would likely be expressing different emotions or at least different levels of the same emotion. Without knowing people’s personality traits, it will be impossible to pinpoint their emotional state. There is a need to advance technology to differentiate between these factors in movements.

While fine-tuning the learned model to a specific person is possible, it usually requires collecting a substantial amount of annotated data from that person, which may not be feasible in practical applications requiring personalization. Further research is necessary to develop methods for personalizing emotion-related models with minimal additional data collection.

F. Partitioning the Space of Emotion

Thus far, technology developers have mostly relied on psychological theories of emotion, including the various models that we discussed. However, these models have limitations that make them not ideal for AI applications. For example, if a model used in an AI program has too many components, the program may struggle to differentiate among them. At the same time, if the model is too simple with too few components, the AI may not be able to fully grasp the human emotion for the intended application. The VAD model offers a solution to this issue, but it is not suitable for AI applications for which specific emotions need to be identified. A deeper understanding of the emotional spectrum in AI will lead to more effective applications.

In a recent study, Wortman and Wang [343] articulated that the strongest models needed robust coverage, which meant defining the minimal core set of emotions from which all others could be derived. Using techniques from NLP and statistical clustering, these researchers showed that a set of 15 discrete emotion categories could achieve maximum coverage. This finding applies to six major languages—Arabic, Chinese, English, French, Spanish, and Russian—they have tested. Categories were identified as affable, affection, afraid, anger, apathetic, confused, happiness, honest, playful, rejected, sadness, spiteful, strange, surprised, and unhealthy. A more refined model with 25 categories was also proposed, which included the addition of accepted, despondent, enthusiasm, exuberant, fearless, frustration, loathed, reluctant, sarcastic, terrific, and yearning, and the removal of rejected. Through the analysis of two large-scale emotion recognition datasets, including BoLD, the researchers confirmed the superiority of their models compared to existing models [343].

G. Benchmarking Emotion Recognition

Effective benchmarking has been instrumental in driving advancements in various AI research areas. However, benchmarking for emotion recognition is a challenging task due to its unique nature and the obstacles discussed earlier. In the following, we offer insights on how to establish meaningful benchmarks for the field of emotion recognition, with a specific emphasis on the relatively new area of BEEU, where benchmarking is currently lacking.

1). Benchmark Task Types:

A suite of tasks should be devised, including basic tasks, such as single-data-type recognition of emotion (based on video only, images only, audio only, skeleton only, and human mesh only), as well as multimodal recognition (a combination of video, audio, and text). Emotion localization, which involves determining the range of frames in a video that depicts a targeted label, as well as movement recognition or LMA recognition using video, skeleton, or human mesh, should also be considered. Furthermore, tasks related to predicting emotion from movement coding and video or based on interaction can be developed. With the rich data across various contexts, natural environments, or situations (e.g., celebration, disaster, and learning), data mining tasks focused on social interactions, including the comparison between age groups or the impact of assistive animals on mood, can be explored. In addition, real-world use-case challenges targeting specific applications can be utilized to assess algorithms’ broad applicability and robustness.

2). Testing and Evaluation:

In a benchmarking competition, the performance of participating teams’ algorithms or systems can be evaluated using various criteria. Along with standard prediction accuracy based on shared training and testing datasets and the extent of emotion coverage, a system’s performance with limited training data and equitable accuracy across different demographic subgroups, such as gender or ethnicity, can also be considered. The competition host can supply training data of varying sizes and required metadata, allowing participating teams to focus on a specific evaluation criterion and compete against others using the same standard. This competition format promotes diverse scientific exploration among participating teams and collectively teams focusing on different standards broaden the scope of models being investigated, effectively fostering a form of free-style community collaboration.

3). Verification and Validation:

To validate software packages developed by participating teams in the competition, winning teams should be required to deposit their packages on repositories such as GitHub. The competition host should provide guidelines for verification, including compatibility with common computing environments, comprehensive documentation, and clear feedback on the execution status and reasons for any unexpected termination. To maintain fairness, true labels for test cases should not be disclosed prior to the completion of a competition.

Software packages should undergo thorough verification and validation throughout the entire training and testing pipeline. The competition host should replicate the training and testing process provided by each winning team, and the results should be compared to the claimed results. To streamline the validation process, subsampling of test cases may be employed.

4). Risk Management:

To gauge the robustness of winning teams’ software packages, they should be asked to provide results from a set of robustness tests although, during the competition, the comparison standard should be based on a single, focused criterion within a relatively straightforward test framework. Specifically, the impact of variations in factors such as batch randomization, bootstrapped sampling of training images, and training data size should be numerically evaluated.

5). Evaluation Metric:

As a person’s emotional state does not fall exclusively into a single type, to provide a fair evaluation of algorithms developed by the competition participants, emotions can be characterized by a distribution over a given set of types, allowing each dataset to be described by different (and multiple) types and users to select the set that works best for their methods. Both the ground truth and the output of emotion recognition algorithms are formatted as distributions over these types. Suppose that there are a total of $K$ emotion types denoted by $e_1,e_2,\dots ,e_K$. Different from a typical classification problem, there is a more subtle relationship between these types. Each pair of emotion types has a specified distance (or similarity) instead of just being different. For example, the emotions “happy” and “sad” are farther apart than “happy” and “excited.” As a result, when we compare two emotion distributions over these types, we want to account for the underlying distances between emotion types. These pairwise distances can be estimated from the data based on how two emotion types co-occur. We can then use Wasserstein distance [344] to compute the overall distance between the ground truth and the computer prediction. Other commonly used distances between distributions such as $L_p$ norm or KL divergence cannot factor in the underlying distances between emotion types. Let the distance between $e_i$ and $e_j$ be $c_{i,j}$, $i$, $j=1,\dots ,K$. Consider two probability mass functions over $\left\{e_1,\dots ,e_K\right\}:p=\left(p_1,\dots ,p_K\right)$ and $q=\left(q_1,\dots ,q_K\right)$. The Wasserstein distance is defined by an optimal transport problem [344]. Let $\mathbf{W}={\left(w_{i,j}\right)}_{i,j=1,\dots ,K}$ be a nonnegative matching matrix between $p$ and $q$. The Wasserstein distance is

$$D(p,q)=\underset{\mathbf{W}}{\min }\sum _{i=1}^K\sum _{j=1}^K{w}_{i,j}{c}_{i,j}$$

$$w_{i,j}\ge 0,i,j=1,\dots ,K$$

$$\sum _{i=1}^K{w}_{i,j}=q_j,j=1,\dots ,K$$

$$\sum _{j=1}^K{w}_{i,j}=p_i,i=1,\dots ,K.$$

VI. BEYOND EMOTION: INTERACTION WITH OTHER DOMAINS

Emotion, as one of the core components of human-to-human communication, can play an essential role in an array of future technological advancements impacting different parts of society. In this section, we provide an overview of how visual emotional understanding can be connected with other research problems, domains, or application areas, including art and design (see Sections VI-A and VI-B), mental health (see Section VI-C), robotics, AI agents, autonomous vehicles, animation, and gaming (see Section VI-D), information systems (see Section VI-E), industrial safety (see Section VI-F), and education (see Section VI-G). Instead of attempting to provide exhaustive coverage, we aim to highlight key intersections. Because some areas are in their early stages of development, we provide only a brief discussion of their potential.

A. Emotion and Visual Art

Art often depicts human emotional expressions, conveys the artist’s feelings, or evokes emotional responses in viewers. Except for certain genres in visual art, e.g., realism, achieving lifelikeness is usually not the primary goal. Dutch post-impressionist painter Vincent van Gogh wrote, “I want to paint what I feel, and feel what I paint.” Similarly, fine-art photographer Ansel Adams stated, “A great photograph is one that fully expresses what one feels, in the deepest sense, about what is being photographed.” It is evident that artists intentionally link visual elements in their works with emotions. However, the relationship between visual elements in art and the emotion they evoke is still largely an enigma.

Because visual artworks are almost always handcrafted by an artist and artists often develop unique styles, artworks are often abstract and difficult to analyze. In 2016, Lu et al. extended this research on evoked emotion in photographs [46] to paintings [47]. They acknowledged that using models developed for photographs on paintings would not be accurate due to the different visual characteristics of the two types of images. To address this, they created an adaptive learning algorithm that leveraged labeled photographs and unlabeled paintings to infer the visual appeal of paintings.

To convey emotion effectively, artists often create and incorporate certain visual elements that are not commonly seen in real-world objects or scenes. An example is van Gogh’s highly rhythmic brushstroke style, which has been shown by computer vision researchers Li et al. to be one of the key characteristic differences between him and his contemporaries [345] (Fig. 18). In fact, van Gogh took piano lessons in the period 1883–1885 and in the middle of his painting career. He wrote to his younger brother Theo toward the latter part of his career in 1888, “...this bloody mistral is a real nuisance for doing brushstrokes that hold together and intertwine well, with feeling, like a piece of music played with emotion.” The study by Li et al. [345] highlights the importance of designing algorithms specifically to answer the art-historical question at hand rather than using existing computer vision algorithms meant for analyzing real-world scenes.

Fig. 18.

Automatic brushstroke extraction for Red Cabbages and Onions (Paris, October-November 1887, oil on canvas, 50.2 cm × 64.3 cm) by Vincent van Gogh [345]. Painting image courtesy of the Van Gogh Museum, Amsterdam (Vincent van Gogh Foundation). The brushstroke map was provided by the James Z. Wang Research Group (The Pennsylvania State University).

Because artwork is the crystallization of the creativity and imagination of artists, studying artwork using computers and modern AI has the potential to reveal new perspectives on the connection between visual characteristics and emotion. Artists often incorporate exaggerated visual expressions, such as carefully designed color palettes, tonal contrasts, brushstroke texture, and elegant curves. These features have inspired computer scientists to create new algorithms for analyzing visual content. For instance, Yao et al. [346] developed a color triplet analysis algorithm to predict the aesthetic quality of photographs, drawing inspiration from artists’ use of limited color palettes. Li et al. [347] created an algorithm for tonal adjustments based on the visual art concept of “Notan” that captures the dark and light arrangement of masses. Motivated by the use of explicit and implicit triangles in artworks, He et al. [348] and Zhou et al. [349] developed algorithms to identify triangles in images, which can assist portrait and landscape photographers with composition.

Current techniques for understanding emotion are not yet capable of analyzing certain aspects of emotion expressed in artwork, particularly at the level of composition or abstraction. For example, when emotions are conveyed through subtle interactions between people, the correspondence between low-level features that we can extract and the emotions that they represent cannot be easily established. American Impressionist painter Mary Cassatt’s work, for example, depicts the love bond between a mother and child through composition, pose, and brushstrokes, rather than through clear facial expressions. Similarly, American Modernist artist Georgia O’Keeffe used dramatic colors, elegant curves, and creative, abstract composition in her paintings of enlarged flowers and landscapes to convey feelings. She stated, “I had to create an equivalent for what I felt about what I was looking at—-not copy it.” There is still much to be discovered by technology researchers in terms of the systematic connection between visual elements in abstract artwork and the emotions they convey.

B. Emotion and Design

Emotion plays a key role in product design, whether it is for physical or virtual products. Cognitive scientist Norman [350] was a pioneer in the study of emotional design. A successful design should evoke positive emotions in users/customers, such as excitement and a sense of pride and identity. In physical products, from a bottle of water to a house, designers carefully select visual elements, such as round corners, simple shapes, and elegant curves, to evoke positive emotions in customers. Similarly, designers of websites, mobile apps, and other digital products and services use harmonious color schemes, simple and clean layouts, and emotion-provoking photographs to create a positive emotional impact on viewers or users.

By advancing evoked emotion prediction, future designers can be assisted by computers in multiple ways. First, computers can assess the evoked emotion of a draft design, based on models learned from large, annotated datasets. For example, a website designer can ask the computer to rate a sample screenshot, identify areas for improvement, and provide advice on how to improve it. To develop this capability, however, researchers need to gain a better understanding of how demographics affect emotion. Certain design elements, e.g., color, may evoke different feelings in different cultures. A system trained with a general population may perform poorly for certain demographic groups. Second, computers can provide designers with design options that not only meet customer needs but also evoke a specific emotion. For example, deep learning and generative adversarial networks (GANs) can already do the former task. If an additional emotion understanding component can be used to assess the design options generated and provide feedback to the system, the resulting designs can then evoke a desired emotion.

C. Emotion and Mental Health

Many mental health disorders can be considered disorders of emotion and emotion regulation [351]. This is because mental health disorders often entail extremes of chronic self-reported distress, sadness, anxiety, or lack of emotions, such as flat affect and numbness, as well as extremes in fluctuation of emotions [352]. For example, anxiety disorders are instances of being in an ongoing state of the fight or flight response, often viewing danger where none exists and, therefore, reacting as though the danger is constantly present and/or overreacting to ambiguous information. Emotion-related symptoms of anxiety disorders include panic attacks, fear, feelings of impending danger, agitation, excessive anxiety, feeling on edge, irritability, and restlessness [352]. Major depressive disorder has been conceptualized as a disorder of sustained negative affect, particularly sadness, and low levels of positive affect [353], [354]. Emotion-related symptoms of major depression include feeling sad or down most of the day nearly every day for at least two weeks and can also include an abundance of guilt, agitation, excessive crying, irritability, anxiety, apathy, hopelessness, loss of interest or pleasure in activities, mood swings, and feelings of restlessness. Similarly, bipolar disorder can include mood swings, elevated sadness, anger, anxiety, apathy, apprehension, euphoria, general discontent, guilt, hopelessness, loss of interest or pleasure, irritability, aggression, and agitation [352]. Extreme mood swings are also a prominent feature for some personality disorders, such as borderline personality disorder, which also entails intense depressed mood, irritability or anxiety lasting a few hours to a few days, chronic feelings of emptiness, intense or uncontrollable anger, shame, and guilt [352]. Schizophrenia is associated with symptoms that are associated with mood. Positive symptoms can include delusions or paranoia, feeling anxious, agitated, or tense, and being jumpy or catatonic. Negative symptoms can include lack of interest or enthusiasm, lack of drive, and being emotionally flat [352]. Because extreme emotions are associated with these disorders, researchers have examined ways to identify important distinctive features of them from videos.

Such studies have examined ways to use machine learning to code videos for nonverbal behaviors from facial expressions, body posture, gestures, voice analysis, and motoric functioning to diagnose mental health problems. In terms of facial expressions, studies have found that people with major depression, bipolar disorder, and schizophrenia demonstrated less facial expressivity compared to individuals without these disorders [355], [356], [357]. Depressed compared to nondepressed individuals also evidenced shorter durations and lower frequency of smiling behavior, less looking at an interviewer, and less eyebrow movement [358], [359], [360]. Such differences have been used to discriminate depressed from nondepressed individuals [361], [362], [363], [364], [365], [366]. Studies have similarly diagnosed differential facial movement features of people with disorders such as autism spectrum disorder [367], posttraumatic stress disorder, and generalized anxiety disorder from healthy controls [368]. Similar to facial emotion, studies have used linguistic and voice emotion analysis to detect disorders such as depression, schizophrenia, bipolar disorder, posttraumatic stress disorder, and anxiety disorders [369], [370], [371].

As with facial actions, gestures and body movements have also been examined. In terms of gestures, those with depression showed more self-touching than those without depression [358], [360], [372]. Compared to schizophrenic individuals, those with depression tended to make fewer hand gestures [358]. Bipolar and depressed people also showed less gross motor activity than those without these disorders [360]. However, depressed and bipolar individuals showed more gross motor activity than people with schizophrenia [360]. At the same time, patients with schizophrenia demonstrated fewer hand gestures [373], fewer small and large head movements, and shorter duration of eye contact compared to those with depression [358], [360], [374]. Additional studies have detected attention deficit hyperactivity disorder from gestures and body movements [367]. Such differences have been used to diagnose mental health problems [375], [376]. See Table 7 for more details about differentiating clinical disorders from healthy controls.

Table 7.

Distinct Emotion-Related Nonverbal Behaviors in Mental Health Disorders Compared to Healthy Controls

Major Depressive Disorder
• Reduced facial expressivity [355, 356]	• Reduced variability of head movements [377, 378]
• Less eyebrow movements [379, 358, 380, 381]	• More nonspecific gaze patterns [382]
• Looking-down behaviors [382]	• Less eye contact with another person [358]
• Reduced hand gestures [379, 381]	• Less smiling [379, 358, 380, 383, 381, 360, 359]
• More self-touching [358, 383, 381, 360]	• Slower voice [384]
• Reduced rate of speech [384]	• More monotonic voice [385, 386, 369, 387]
• Reduced speech [384]	• Reduced pitch range [384]
• Slower movements or abnormal postures [388, 372]	• Reduced gross motor activity [379, 389, 360]
• Reduced stride length and upward lifting motion of legs [390, 391]	• Slower Gait speed [392, 390, 393]
• Arm swing and vertical head movements while walking [393]	• Lateral upper body sway while walking [393]
• Slumped posture [394, 393, 395]	• Forward inclination of head and shoulders [396, 397]
• Balance difficulties during motor and cognitive tasks [398, 399, 393, 400, 401]
• Impaired balance and lower gait velocity [390, 394, 393, 402]	• Difficulty recognizing emotions [403, 404]
Bipolar Disorder
• Reduced levels of facial expressivity [405]	• Greater speech tonality [406, 407, 408]
• Less gross motor activity [360]
• More frequent and longer speech pauses when in depressive states [406, 409]
• More postural sway [410]	• Difficulty recognizing emotions [411, 412]
Schizophrenia
• Reduced facial expressivity [413, 355, 356, 414]	• Less upper facial movement expressing positive emotion [356, 415, 416]
• Less smiling [358, 417, 418]
• Reduced smiling eye gaze and head tilting associated with negative symptoms [419, 420]
• Fewer hand gestures when speaking [373, 421, 419, 422, 420]	• Fewer gestures and poses [421]
• Less head nodding [373, 419]	• Less head movement [374, 358]
• Lower total time talking [423, 424, 371]	• Slower rate of speech [425, 426]
• Longer speech pauses [423, 424, 371, 427]	• More pauses [371]
• Flat affect [371]	• Forward head posture and lower spine curvature [428]
• Balance difficulties and increased postural sway, [429, 430, 431, 432]	• Difficulty walking in a straight fine [433, 434]
• Slower velocity of walking and shorter strides [435]	• Difficulty recognizing emotions [411, 412, 436]
Anxiety Disorders
• Less eye contact [437, 438, 439]	• Instability of gaze direction [440]
• Grimacing [437]	• Nonsymmetrical lip deformations [441]
• Strained face [442]	• Eyelid twitching [442]
• Smiled less, [438]	• More frequent and faster head movements [443, 444, 445]
• More and faster blinking [443, 446, 447, 448]	• Nodded less [438]
• Small rapid head movements [446]	• Made fewer gestures [438]
• More physical movements indicative of nervousness (e.g., bouncing their knees, fidgety, reposuring [449, 450, 451, 452, 453, 438]
• Self touching [449]	• Speech errors [454]
• Speech dysfluency [437]	• More jittery voice [455, 456]
• Slow gait velocity associated with fear of falling [457, 458, 459]	• Balance dysfunction [460, 461, 462, 463, 464]
• Slower speed walking [463]	• Shorter steps [463]
• Enhanced recognition of anxiety [465, 466]
postTraumatic Stress Disorder
• Monotonous slower flatter speech [369, 467, 468, 469]	• Reduced facial emotion [470]
• More anger, aggression, hostility, less joy [471, 470]
Autism Spectrum Disorder
• Distinctions in gait [472]	• Difficulty recognizing emotions [473]

Gait, balance, and posture have also been used to identify mental health problems. For example, one meta-analysis summarized 33 studies of gait and balance in depression [474]. Depressed individuals had worse and more slumped posture [393], [395], [397], [475] and more postural instability and control [399]. In terms of gait, compared to healthy controls, those with depression took shorter strides, lifted their legs upward as opposed to a forward motion [390], [391], had more body sway, and walked more slowly, possibly to maintain their balance [390], [392], [393], [401]. These results are consistent with psychomotor retardation, a common symptom of depression. In terms of anxiety disorders, a study showed that these individuals walked more slowly, took shorter steps, and demonstrated problems with balance and mobility [463], [476]. Studies have also used gait to identify bipolar disorder [477], autism spectrum disorders [367], and attention deficit hyperactivity disorder [367].

In addition to mental health problems being disorders of emotional expression and experience, mental health problems can also be considered to be disorders of emotion recognition and understanding leading to social deficits. Understanding one’s own and others’ emotions has been termed the theory of mind. Tests for the theory of mind can include either identifying emotions by looking at images of faces with various emotional expressions (sometimes with parts of the faces obstructed) or watching a video of interpersonal interaction and answering questions about various people’s emotions and intentions. Having difficulties of theory of mind has been associated with depression [403], [404], social anxiety disorder [478], obsessive-compulsive disorder [479], schizophrenia [404], [480], bipolar disorder [404], [481], and autism spectrum disorders [473]. For example, both schizophrenic and bipolar patients showed emotional reactivity that was discordant to emotional videos [357]. Studies have also examined videotapes of facial emotional reactions to emotionally evocative videos as a means of diagnosing mental health problems. For example, using this technique, those with autism spectrum disorders demonstrated impairment in their ability to recognize emotions from body gestures [482], [483]. Thus, emotion regulation, emotional understanding, and emotional reactivity can be impaired in those with mental health problems. Such impairment, however, can be used to create systems to automatically detect the presence of these emotional disorders. See Table 7 for more details.

D. Emotion and Robotics, AI Agents, Autonomous Vehicles, Animation, and Gaming

A natural application domain for emotion understanding is robotics and AI. In science fiction films, robots and AI agents are often depicted as having a high level of EQ, such as R2-D2, T-800, and Wall-E. They are able to understand human emotion, effectively communicate their own emotional feelings, engage in emotional exchanges with other robots or humans, take appropriate actions in challenging situations or conflicts, and so on. The idea of empowering robots and AI with this level of EQ is widely seen as a desirable and ultimate goal or a “Holy Grail.”

Some recent surveys studied the field of robotics and emotion [484], [485], [486], covering topics such as advanced sensors, the latest modeling methods, and techniques for performing emotional actions. Research in the fields of HRI, human–machine interaction (HMI), and human–AI interaction is highly relevant. However, because BEEU is in its infancy and is considered a bottleneck technology, we have yet to see its applications in robotics.

If we can effectively model human emotions through both facial and bodily expressions, robots can work more effectively with human counterparts. Humans would be able to communicate with robots in a way similar to how emotions are used in human-to-human communication. For example, when human workers in a warehouse want to stop a fast-moving robot, they could wave their hands swiftly to signal distress. Similarly, pedestrians could wave their hands to signal to a self-driving vehicle on a highway that there is an accident ahead, and cars should slow down to avoid a collision. In such situations, traditional forms of communication such as speech and facial expressions may not effectively convey a sense of urgency.

Effective emotional communication can help us understand the intention of robots or AI. For instance, emotionless robots can be perceived as unfriendly or unsympathetic. In certain robotic or AI applications, such as companion robots or assistive robots, it is desirable to project a compassionate and supportive image to establish trust and cooperation between the device and humans interacting with it. Researchers have begun to investigate the relationship among robotics, personality, and motion [487].

In animated films, robots can display emotional behaviors, but these are often created by recording the movements of human actors through MoCap. That is, the animated characters mimic the movements of the actors behind the scenes. However, the capacity for computers to comprehend emotions akin to human perception could enable animated characters to use an emotion synthesis engine to autonomously generate authentic emotional behaviors. Advancements in computer graphics, virtual reality, and deep learning techniques, including GANs, transformers, diffusion models, and contrastive learning, have facilitated the creation of increasingly realistic and dynamic visual content. These technologies potentially enable the synthesis of complex and nuanced emotional displays.

Emotion understanding can substantially enhance the gaming experience by making games more emotionally responsive and immersive, as well as by providing personalized feedback to players. Game designers can make design decisions that enhance a player’s experience based on the player’s frustration level. If players are feeling sad, the game could offer them a story-based scenario that is more emotionally uplifting. By providing meaningful feedback, players are more likely to stay engaged with the game, improving their overall experience.

E. Emotion and Information Systems

Emotion understanding can play a pivotal role in advancing information systems. Currently, when searching for visual content in online collections, we primarily rely on keyword-based metadata. Whereas recent developments in deep learning have enabled information systems to search using machine-generated annotations, these annotations are typically limited to identifying objects and relationships (e.g., a boy in a yellow shirt playing soccer).

IBM scientists demonstrated that computers with the ability to understand emotions could aid in sorting through a large amount of visual content and composing emotion-stimulating visual summaries [488]. In 1996, they created the first computer system for generating movie trailers. The trailer it produced for the 20th Century Fox sci-fi thriller “Morgan” was released as the official trailer. The system identified the top ten moments for inclusion in the trailer (see Fig. 19). This work represents a significant milestone in understanding evoked emotions.

Fig. 19.

Two scenes included in the computer-generated trailer for the sci-fi thriller “Morgan” [488]. The computer program considered these scenes most suited to give viewers a sense of horror or thrill. The copyright of these movie frames belongs to 20th Century Fox. They are used here for illustrating a scientific concept only.

IBM’s program is likely to be the starting point for a surge of emotion-based information systems. We can expect to see new applications, such as evoked emotion assessment systems, emotion-based recommender systems, emotion-driven photo/video editing software, and emotion-based media summarization/synopsizing services.

F. Emotion and Industrial Safety

Emotion understanding can be useful in promoting safety in workplaces such as factories and warehouses. It can provide early warnings of potential safety risks, such as worker fatigue or stress, allowing managers to take proactive measures to address the situation. Such capabilities can also provide personalized support and resources to workers who are experiencing emotional distress, improving overall emotional well-being and contributing to a safer work environment. The National Safety Council estimates that fatigue costs employers over $130 billion annually in lost productivity, and over 70 million Americans have sleep disorders [489]. Existing research on fatigue detection typically involves specialized sensors or vision systems that monitor the face or eyes [490], [491], [492], [493], [494], [495], [496]. However, sensor-based approaches have limitations such as the need to wear them, size, cost, and reliability. Thus, there is a need to develop recognition systems that use body movement to enhance such systems [490], [497].

G. Emotion and Education

Emotion recognition technology can help create a more engaging and effective learning experience for online education. Many universities have been offering online courses for years, but the COVID-19 pandemic led to the widespread adoption of online teaching using video conferencing platforms in 2020 and 2021. Even as in-person instruction has resumed, many educational institutions continue to conduct some of their teaching activities online. For example, instructors may be allowed to teach a portion of their classes online for pedagogical or emergency reasons or to hold office hours online. In a traditional classroom setting, instructors can gauge students’ attentiveness and emotional states by observing their facial and bodily expressions. Such feedback can help instructors better understand and respond to students’ needs, e.g., by adjusting the pace of instruction or covering alternative materials. However, in an online teaching environment, instructors often can only see the faces of a small number of students, which does not provide real-time feedback on the instruction. Potentially, if an online teaching platform could dynamically monitor the students and provide aggregated feedback (e.g., the percentage of students with high attentiveness and the overall mood of the class) to the instructor, the quality of online learning could be improved. To protect students’ privacy, the monitoring process should only produce overall statistics.

VII. EMOTION AND SOCIETY: ETHICS

New technologies often bring new ethical concerns. In the field of emotion understanding, we have begun to witness the potential misuse of these technologies. In this section, we will discuss some of the general ethical issues surrounding the development and deployment of these technologies.

1). Generalizability:

Because the emotion space is complex, it is important to recognize that there will always be outliers or unusual situations (e.g., an otherworldly scene or an eccentric behavior) that may not be captured by our models. Without proper consideration of demographic differences and individual variations, these technologies may only provide a broad overview of the general population. To be truly beneficial, the system must be carefully tailored to the specific needs of an individual. Likewise, diversity of representation in sample datasets is critical to ensure that algorithms emerging from them are inclusive.

2). Verification of Accuracy or Performance:

It is important for researchers to keep in mind that there is almost always a lack of ground truth in emotion understanding. We have discussed the impact on data collection, modeling, and evaluation/benchmarking earlier (see Sections III-B, V-B, and V-G). For AI models, it is imperative that the output, design, and training processes are transparent and auditable. Black-box models can become uncontrollable if not properly monitored.

3). Privacy—Data Collection:

The collection of human behavior data, including facial, body, and vocal information, raises privacy concerns. Research involving sensitive populations, such as patients in psychological clinics, must be conducted with utmost care. Furthermore, almost all emotion-related annotations must be collected from humans. To protect human subjects, research protocols must be carefully designed to collect only necessary information, deidentify before distribution, and protect the data with proper access control and encryption. All protocols must be reviewed by an Institutional Review Board.

4). Privacy—Use of Technology:

In today’s automated world, people are losing their privacy to whoever controls data: governments and companies are collecting data about where we are at any given moment; our financial transactions are followed and verified; companies are collecting data about our purchases, preferences, and social networks; most public places are constantly videotaped; and so on. People must sacrifice privacy to live a normal life because everything is computerized. We are being followed, and “Big brother” knows all about us. The only thing we can still keep to ourselves is our thoughts and emotions.

Once AI reaches high-accuracy automatic emotion recognition, our emotions will not be private anymore, and videos of our movements could be used against us by authorities or whoever will have videos of our movements. This situation could become very frightening. Moreover, if we want to hide our emotions, we will have to move in a way that will not reveal them, like using a “poker face” to hide facial expressions. However, because specific movements not only express associated emotions but also enhance those emotions [115], moving in ways that flatten emotional expressions can also flatten the felt emotions, and living in such a way can lead to depression or other mental health problems.

5). Synthesized Affective Behavior:

As much as its potential use in entertainment, success in emotion modeling could inevitably lead to even more lifelike deepfakes and similar abuses. As a society, instead of being fearful of the negative impact of new, beneficial technologies, we need to take on the challenge of detecting fakes, much as we recognize scammers, and mitigating the harm.

6). Lower the Risks:

To mitigate the risks of misuse, proactive measures must be taken. It is essential that laws and regulations are established to keep pace with the rapid development of AEI technologies. As researchers, we have a responsibility to involve affected communities, particularly those that are traditionally marginalized, such as minority groups, elderly individuals, and mental health patients, in the design, development, and deployment processes to ensure that these individuals’ perspectives and needs are recognized and valued.

7). Performance Criteria:

To promote a responsible and ethical expansion of the field, it is crucial to establish reliable mechanisms for comparing the predictions of different algorithms and learning procedures. As discussed in Section V-G, a thorough evaluation of algorithms should consider not only accuracy and speed but also factors such as interpretability, demographic representation, context coverage, emotion space coverage, and personalization capabilities.

VIII. CONCLUSION

We provided an overview of the stimulating and exponentially growing field of visual emotion understanding. Adopting a multidisciplinary approach, we discussed the foundational principles guiding technological progress, reviewed recent innovations and system development, identified open challenges, and highlighted potential intersections with other fields. Our objective was to provide a comprehensive introduction to this vast field sufficient to intrigue researchers across related IEEE subcommunities and to inspire continued research and development toward realizing the field’s immense potential.

Given the multidisciplinary nature of this field, which encompasses multiple technical fields, psychology, and art, the barrier to entry can be considerable. Our aim is to provide researchers and developers with the essential knowledge required to tackle the numerous attractive open problems in the field. Interested readers are encouraged to delve deeper into the cited references for a more profound understanding of the topics discussed.

As active researchers in this domain, we strongly recommend that those interested in pursuing this research topic collaborate with others possessing complementary expertise. Although we anticipate the development and sharing of more large-scale datasets and continuous incremental progress, transformative solutions will not arise solely from the straight application of data-driven approaches. Instead, sustained collaboration among researchers in computational, social and behavioral, and machine learning and statistical modeling fields will likely lead to lasting contributions to this intricate research field.

Biographies

James Z. Wang (Senior Member, IEEE) received the bachelor’s degree (summa cum laude) in mathematics from the University of Minnesota, Minneapolis, MN, USA, in 1994, and the M.S. degree in mathematics, the M.S. degree in computer science, and the Ph.D. degree in medical information sciences from Stanford University, Stanford, CA, USA, in 1997, 1997, and 2000, respectively.

He was a Visiting Professor with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA, from 2007 to 2008. He is currently a Distinguished Professor of the data science and artificial intelligence area and the human–computer interaction area with the College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA, USA. His research interests include affective computing, image analysis, image modeling, image retrieval, and their applications.

Dr. Wang was a Lead Special Section Guest Editor of the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE in 2008 and a Program Manager with the Office of the Director of the National Science Foundation from 2011 to 2012.

Sicheng Zhao (Senior Member, IEEE) received the Ph.D. degree from the Harbin Institute of Technology, Harbin, China, in 2016.

He was a Visiting Scholar with the National University of Singapore, Singapore, from 2013 to 2014, a Research Fellow with Tsinghua University, Beijing, China, from 2016 to 2017, a Postdoctoral Research Fellow with the University of California at Berkeley, Berkeley, CA, USA, from 2017 to 2020, and a Postdoctoral Research Scientist with Columbia University, New York, NY, USA, from 2020 to 2022. He is currently a Research Associate Professor with Tsinghua University. His research interests include affective computing, multimedia, and computer vision.

Chenyan Wu received the B.E. degree in electronic information engineering from the School of the Gifted Young, University of Science and Technology of China, Hefei, China, in 2018. He is currently working toward the Ph.D. degree in the Informatics Program of the College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA, USA.

He worked as an Intern at Amazon Lab126, Bellevue, WA, USA, Microsoft Research Asia, Beijing, China, and SenseTime Research, Shenzhen, China. His research interests are affective computing, computer vision, and machine learning.

Reginald B. Adams Jr. received the Ph.D. degree in social psychology from Dartmouth College, Hanover, NH, USA, in 2002.

He is currently a Professor of psychology with The Pennsylvania State University, University Park, PA, USA. He is interested in how we extract social and emotional meaning from nonverbal cues, particularly via the face. His work addresses how multiple social messages (e.g., emotion, gender, race, and age) combine across multiple modalities and interact to form unified representations that guide our impressions of and responses to others. Although his questions are social psychological in origin, his research draws upon visual cognition and affective neuroscience to address social perception at the functional and neuroanatomical levels. His continuing research efforts have been funded through the NSF and the National Institute on Aging and the National Institute of Mental Health of the National Institutes of Health.

Dr. Adams Jr., before joining Penn State, was awarded the National Research Service Award (NRSA) from the NIMH to train as a Postdoctoral Fellow at Harvard University and Tufts University.

Michelle G. Newman received the Ph.D. degree in clinical psychology from Stony Brook University, Stony Brook, NY, USA, in 1992.

She completed a postdoctoral fellowship at Stanford University, Stanford, CA, USA, in 1994. She is currently a Professor of psychology and psychiatry, and the Director of the Center for the Treatment of Anxiety and Depression, The Pennsylvania State University, University Park, PA, USA. She has conducted basic and applied research on anxiety disorders and depression, and has published over 200 papers on these topics.

Dr. Newman is also a Fellow of the American Psychological Association Divisions 29 and 12, the Association for Behavioral and Cognitive Therapies, and the American Psychological Society. She was a recipient of the APA Division 12 Turner Award for distinguished contribution to clinical research, the APA Division 29 Award for Distinguished Publication of Psychotherapy Research, the ABCT Outstanding Service Award, the APA Division 12 Toy Caldwell-Colbert Award for Distinguished Educator in Clinical Psychology, and the Raymond Lombra Award for Distinction in the Social or Life Sciences. She is a Past Editor of Behavior Therapy. She is an Associate Editor of Journal of Anxiety Disorders.

Tal Shafir graduated from the Law School, The Hebrew University of Jerusalem, Jerusalem, Israel. She received the Ph.D. degree in neurophysiology of motor control from the University of Michigan, Ann Arbor, MI, USA, in 2003.

She then studied dance-movement therapy at the University of Haifa, Haifa, Israel, and completed two postdoctoral fellowships in brain–behavior interactions in infants, and in affective neuroscience at the University of Michigan. She developed research on movement–emotion interaction and its underlying brain mechanisms, behavioral expressions, and therapeutic applications.

Dr. Shafir was a recipient of the ADTA 2020 Innovation Award. She, certified also in laban movement analysis, was the Main Editor of The Academic Journal of Creative Arts Therapies and Frontiers in Psychology research topic: the state of the art in creative arts therapies. She has been serving on The American Dance Therapy Association (ADTA) Research Committee since 2016.

Rachelle Tsachor is currently an Associate Professor of movement with the University of Illinois at Chicago, Chicago, IL, USA. She is certified in mind-body medicine (CMBM), Laban Movement Analysis (CMA), and somatic movement therapy (RSMTISMETA). Her research investigates body movement to bring a human, experiential understanding to how movement affects our lives. She analyzes patterns in moving bodies in diverse projects, researching movement’s effects on our brains, emotions, health, and learning. She is a Co-PI on an NSF-funded project STAGE and the UI Presidential Initiative for the Young People’s Science Theater: CPS and UIC Students Creating Performances for Social Change. Both initiatives bring mind/body methods into Chicago Public Schools that educate primarily students of color to support learning in embodied ways.