What the Face Displays: Mapping 28 Emotions Conveyed by Naturalistic Expression

Abstract

What emotions do the face and body express? Guided by new conceptual and quantitative approaches (Cowen, Elfenbein, Laukka, & Keltner, 2018; Cowen & Keltner, 2017, 2018), we explore the taxonomy of emotion recognized in facial-bodily expression. Participants (N = 1,794; 940 female, ages 18–76 years) judged the emotions captured in 1,500 photographs of facial-bodily expression in terms of emotion categories, appraisals, free response, and ecological validity. We find that facial-bodily expressions can reliably signal at least 28 distinct categories of emotion that occur in everyday life. Emotion categories, more so than appraisals such as valence and arousal, organize emotion recognition. However, categories of emotion recognized in naturalistic facial and bodily behavior are not discrete but bridged by smooth gradients that correspond to continuous variations in meaning. Our results support a novel view that emotions occupy a high-dimensional space of categories bridged by smooth gradients of meaning. They offer an approximation of a taxonomy of facial-bodily expressions, visualized within an online interactive map.

As novelists, poets, and painters have long known, the dramas that make up human social life—flirtations, courtships, family affection and conflict, status dynamics, grieving, forgiveness, and personal epiphanies—are rich with emotional expression. The science of emotion is revealing this to be true as well: From the first moments of life the recognition of expressive behavior is integral to how the individual forms relationships and adapts to the environment. Infants as young as 6 months old identify expressions of anger, fear, joy, and interest (Peltola, Hietanen, Forssman, & Leppänen, 2013) that guide behaviors from approaching strangers to exploring new sources of reward (Hertenstein & Campos, 2004; Sorce, Emde, Campos, & Klinnert, 1985). Adults signal love and desire in subtle facial movements (Gonzaga, Keltner, Londahl, & Smith, 2001). We more often forgive those who express embarrassment (Feinberg, Willer, & Keltner, 2012), conform to the actions of those who express pride (Martens & Tracy, 2013), and concede in negotiations to those who express anger (Reed, DeScioli, & Pinker, 2014). Even shareholders’ confidence in a company depends on a CEO’s facial expression following corporate transgressions (ten Brinke & Adams, 2015). The recognition of fleeting expressive behavior structures the interactions that make up social life.

The study of emotion recognition from facial-bodily expression (configurations of the facial muscles, head, torso, limbs, and eye gaze) has been profoundly influenced by the seminal work of Ekman, Sorenson, and Friesen (1969) in New Guinea. In that work, children and adults labeled, with words or situations, prototypical facial expressions of anger, fear, disgust, happiness, sadness, and surprise. Enduring in its influence, that research, as we will detail, established methods that preclude investigating the kinds of emotion people recognize from expressive behavior, the degree to which categories (e.g., fear) and appraisals (e.g., valence) organize emotion recognition, and whether people recognize expressions as discrete clusters or continuous gradients—issues central to theoretical debate in the field. Based on new conceptual and quantitative approaches, we test competing predictions concerning the number of emotions people recognize, whether emotion recognition is guided by categories or domain-general appraisals and whether the boundaries between emotion categories are discrete.

The Three Properties of an Emotion Taxonomy

Emotions are dynamic states that are initiated by meaning-based appraisals and involve multiple response modalities—experience, expression, and physiology—that enable the individual to respond to ongoing events in the environment (Keltner, Sauter, Tracy, & Cowen, 2019). Within each modality, empirical study (and theoretical controversy) has centered upon the number of emotions with distinct responses, how people represent these responses with language, and how emotions overlap or differ from one another (Barrett, 2006; Keltner, Sauter, et al., 2019; Keltner, Tracy, Sauter, & Cowen, 2019; Lench, Flores, & Bench, 2011; Nummenmaa & Saarimäki, 2019).

To provide mathematical precision to these lines of inquiry, we have developed what we call a semantic space approach to emotion (Cowen & Keltner, 2017, 2018). As emotions unfold, people rely on hundreds, even thousands, of words to describe the specific response, be it a subjective feeling, a physical sensation, a memory from the past, or—the focus of this investigation—the expressive behavior of other people. How people describe emotion-related responses (e.g., experience, expression, physiology) manifests in what we call a semantic space of that response modality. A semantic space is defined by three properties. The first property is its dimensionality, or the number of distinct varieties of emotion that people represent within the semantic space. In the present investigation, we ask, how many emotions do people recognize in facial-bodily expressions? The second property is what we call conceptualization: To what extent do emotion categories and appraisals of valence, arousal, dominance, and other themes capture how people ascribe meaning to emotional responses within a specific response modality? The third property is the distribution of the emotional states within the space. In the present investigation, we ask how distinct the boundaries are between emotion categories recognized in expressive behavior.

Approximating a semantic space of emotion within a response modality requires judgments of emotion categories and appraisals of a vast array of stimuli that spans a wide range of emotions and includes variations within categories of emotion. In previous investigations guided by this framework, we examined the semantic space of emotional experience (Cowen & Keltner, 2017) and vocal expression (Cowen, Elfenbein, Laukka, & Keltner, 2018; Cowen, Laukka, Elfenbein, Liu, & Keltner, 2019). In these studies, 2,185 emotionally evocative short videos, 2,032 nonverbal vocalizations (e.g., laughs, screams), and 2,519 speech samples varying only in prosody (tune, rhythm, and timbre) were rated by participants in terms of emotion categories and domain-general appraisals (e.g., valence, dominance, certainty), and new statistical techniques were applied to participants’ judgments of experience, vocal bursts, and prosody. With respect to the dimensionality of emotion, we documented 27 kinds of emotional experience, 24 kinds of emotion recognized in nonverbal vocalizations, and 12 kinds of emotion recognized in the nuanced variations in prosody that occur during speech. With respect to the conceptualization of emotional experience, we found that emotion categories (e.g., awe, love, disgust), more so than broad domain-general appraisals such as valence and arousal, explained how people use language to report upon their experiences and perceptions of emotion. Finally, in terms of the distribution of emotional experiences and expressions, we documented little evidence of sharp, discrete boundaries between kinds of emotion. Instead, different kinds of emotion—for example, love and sympathy—were linked by continuous gradients of meaning.

In the present investigation, we apply this conceptual framework to investigate a more traditionally studied modality of expression: facial-bodily expressions. We do so because questions regarding the number of emotions signaled in the face and body, how emotions recognized in the face and body are best conceptualized, and the structure of the emotions we recognize in the face and body are central to the science of emotion and a central focus of theoretical debate (Barrett, 2006; Keltner, Sauter, et al., 2019; Keltner, Tracy, et al., 2019; Shuman, Clark-Polner, Meuleman, Sander, & Scherer, 2017). A semantic space approach has the potential to yield new advances in these controversial areas of inquiry.

Beyond Impoverished Methods: The Study of the Dimensionality, Conceptualization, and Distribution of Emotion Recognition

The classic work of Ekman and colleagues (1969) established prototypical facial-bodily expressions of anger, disgust, fear, sadness, surprise, and happiness (Elfenbein & Ambady, 2002). The expressions documented in that work have enabled discoveries concerning the developmental emergence of emotional expression and recognition (Walle, Reschke, Camras, & Campos, 2017), the neural processes engaged in emotion recognition (Schirmer & Adolphs, 2017), and cultural variation in the meaning of expressive behavior (e.g., Elfenbein, Beaupré, Lévesque, & Hess, 2007; Matsumoto & Hwang, 2017). The photos of those prototypical expressions may be the most widely used expression stimuli in the science of emotion—and the source of the most theoretically contested data.

Critiques of the methods used by Ekman and colleagues (1969) can often be reframed as critiques of how a narrow range of stimuli and constrained recognition formats fail to capture the richness of the semantic space of emotion recognition (Keltner, Sauter, et al., 2019; Keltner, Tracy, et al., 2019; Russell, 1994). First, the focus on prototypical expressions of anger, disgust, fear, sadness, happiness, and surprise fails to capture the dimensionality, or variety of emotions, signaled in facial-bodily expression (see Cordaro et al., in press; Keltner, Sauter, et al., 2019; Keltner, Tracy, et al., 2019). Recent empirical work finds that humans are capable of producing a far richer variety of facial-bodily signals than traditionally studied, using our specially adapted facial morphology with 30 to 40 facial muscles, pronounced brow ridges, accentuated sclera, and the emotional information conveyed in subtle movements of the head, torso, and gaze activity (Godinho, Spikins, & O’Higgins, 2018; Jessen & Grossmann, 2014; Reed et al., 2014; Schmidt & Cohn, 2001). Studies guided by claims concerning the role of emotion in attachment, hierarchies, and knowledge acquisition point to candidate facial-bodily expressions for over 20 states, including amusement (Keltner, 1995), awe (Shiota et al., 2017), contempt (Matsumoto & Ekman, 2004), contentment (Cordaro, Brackett, Glass, & Anderson, 2016), desire (Gonzaga, Turner, Keltner, Campos, & Altemus, 2006), distress (Perkins, Inchley-Mort, Pickering, Corr, & Burgess, 2012), embarrassment (Keltner, 1995), interest (Reeve, 1993), love (Gonzaga et al., 2001) pain (Prkachin, 1992), pride (Tracy & Robins, 2004), shame (Keltner, 1995; Tracy & Matsumoto, 2008), sympathy (Goetz, Keltner, & Simon-Thomas, 2010), and triumph (Tracy & Matsumoto, 2008). In this investigation, we seek to account for the wide range of emotions for which facial-bodily signals have been proposed, among other candidate states, although future studies could very well introduce further displays. With this caveat in mind, we, for the first time in one study, investigate how many emotions can be distinguished by facial-bodily expressions.

Second, beginning with Ekman and colleagues (1969), it has been typical in emotion recognition studies to use either reports of emotion categories or ratings of valence, arousal, and other appraisals to capture how people recognize emotion (for a recent exception, see Shuman et al., 2017). These disjoint methodological approaches—that measure emotion categories or appraisals but not both—have failed to capture the degree to which emotion categories and domain-general appraisals such as valence, arousal, and dominance organize emotion recognition.

Finally, the focus on expression prototypes fails to capture how people recognize variations of expression within an emotion category (see Keltner, 1995, on embarrassment, and Tracy & Robins, 2004, on pride), leading to potentially erroneous notions of discrete boundaries between categories (Barrett, 2006). Guided by these critiques, we adopt methods that allow for a fuller examination of the semantic space of emotion recognition from facial-bodily behavior.

Basic and Constructivist Approaches to the Semantic Space of Emotion Recognition

Two theories have emerged that produce competing predictions concerning the three properties of the semantic space of emotion recognition. Basic emotion theory starts from the assumption that different emotions are associated with distinct patterns of expressive behavior (see Keltner, Sauter, et al., 2019; Keltner, Tracy, et al., 2019; Lench et al., 2011). Recent evidence suggests that people may recognize upward of 20 states (e.g., Cordaro et al., in press). It is asserted that emotion categories, more so than appraisals of valence, arousal, or other broad processes, organize emotion recognition (e.g., Ekman, 1992; Nummenmaa & Saarimäki, 2019; Scarantino, 2017) and that the boundaries between emotions are discrete (Ekman & Cordaro, 2011; Nummenmaa & Saarimäki, 2019).

Constructivist approaches, by contrast, typically presuppose that what is most core to all emotions are domain-general appraisals of valence and arousal, which are together referred to as core affect (Russell, 2009; Watson & Tellegen, 1985). Layered on top of core affect are context-specific interpretations that give rise to the constructed meaning of expressive behavior (Barrett, 2017; Coppin & Sander, 2016; Russell, 2009). This framework posits that the recognition of emotion is based on the perception of a limited number of broader features such as valence and arousal (Barrett, 2017), and that there are no clear boundaries between emotion categories.

Although often treated as two contrasting possibilities, basic emotion theory and constructivism are each associated with separable claims about the dimensionality, conceptualization, and distribution of emotion. In Table 1, we summarize these claims, which remain largely untested given the empirical focus on prototypical expressions of six emotions.

Table 1.

Past Predictions Regarding the Semantic Space of Emotion Recognition

Theory	Dimensionality How many emotions are signaled in facial-bodily behavior?	Conceptualization What concepts capture the recognition of emotion in expression?	Distribution Are emotions perceived as discrete clusters or continuous gradients?
Basic emotion theory	Perceptual judgments rely on a variety of distinct emotions, ranging from six in early theorizing to upwards of 20 in more recent theorizing.	Emotion categories capture the underlying organization of emotion recognition. Ekman (1992): “Each emotion has unique features … [that] distinguish emotions from other affective phenomena.”	Boundaries between categories are discrete. Ekman (1992): “Each of the basic emotions is … a family of related states. … I do not propose that the boundaries between [them] are fuzzy.”
Constructivist theory	Perceptual judgments of expressions will be reducible to a small set of dimensions.	Valence, arousal (and perhaps a few other appraisals) will organize the recognition of emotion.	Categories, constructed from broader properties that vary smoothly and independently, will be bridged by continuous gradients.
	Barrett (2017): “Emotional phenomena can be understood as low dimensional features.”	Russell (2003): “Facial, vocal … changes … are accounted for by core affect [valence and arousal] and … instrumental action.”	Barrett (2006): “evidence … is inconsistent with the view that there are kinds of emotion with boundaries that are carved in nature.”
Methods	Multidimensional reliability assessment.	Statistical modeling applied to large, diverse data sets.	Visualization techniques, direct analysis of clusters, and continuous gradients.

The Present Investigation

To test these competing predictions about the semantic space of emotion recognition, we applied large-scale data analytic techniques (Cowen et al., 2018; Cowen & Keltner, 2017; Cowen et al., 2019) to the recognition of emotion facial-bodily behavior. Departing from the field’s typical focus on prototypical expressions of six emotions, we (a) compiled a rich stimulus set incorporating numerous, highly variable expressions of dozens of recently studied emotional states (addressing pervasive limitations of low sample sizes and the reliance on prototypical expressions of a limited number of emotions); (b) collected hundreds of thousands of judgments of the emotion categories and dimensions of appraisals perceived in these expressions; (c) applied model comparison and multidimensional reliability techniques to determine the dimensions (or kinds of emotion categories) that explain the reliable recognition of emotion from facial/bodily expression (moving beyond the limitations of traditional factor analytic approaches; see Supplemental Discussion section in the online supplemental materials); (d) investigated the distribution of states along the distinct dimensions people recognize, including the boundaries between emotion categories; and finally, (e) created an online interactive map to provide a widely accessible window into the taxonomy of emotional expression in its full complexity.

Guided by findings of candidate expressions for a wide range of emotions, along with recent studies of the semantic space of subjective experience and vocal expression, we tested the following predictions.

Hypothesis 1: With respect to the dimensionality of emotion recognition, people will be found to recognize a much wider array of distinct emotions in expressive behavior than typically thought (i.e., greater than 20 distinct dimensions of emotion, as opposed to fewer than 10).

Hypothesis 2: With respect to the conceptualization of emotion recognition, emotion categories more so than domain general appraisals will capture the emotions people recognize in facial bodily expressions.

Hypothesis 3: With respect to the distribution of emotion recognition, emotion categories will not be discrete but joined by continuous gradients of meaning. These gradients in the categories of expression people recognize in facial-bodily expressions will correspond to smooth variations in appraisals of affective features.

We also answered more exploratory questions about the influence of context and ecological validity on emotion recognition relevant to recent debate (Aviezer et al., 2008; Barrett, 2017; Matsumoto & Hwang, 2017).

Exploratory Question 1: Features of the context—what the person is wearing, clues to the social nature of the situation, and other expressive actions—can strongly influence the recognition of facial-bodily expression. For instance, a person holding a soiled cloth is more likely to be perceived as disgusted than a person poised to punch with a clenched fist (Aviezer et al., 2008). However, the extent to which context is required for the recognition of emotion more generally from facial-bodily expression is not well understood. Here, we determine whether context is necessary for the recognition of a wide array of emotions.

Exploratory Question 2: A longstanding concern in the study of facial bodily expression has been the ecological validity of experimental stimuli (e.g., Russell, 1994). Are people only adept at recognizing facial bodily expressions that are extreme and caricature like (e.g., the photos used by Ekman and colleagues [1969])? Do people encounter a more limited variety of expressions in everyday life (Matsumoto & Hwang, 2017)? We will ascertain how often people report having perceived and produced the expressions investigated here in their everyday lives.

Method

To address the aforementioned hypotheses and questions, we created a diverse stimulus set of 1,500 facial-bodily expressions and gathered ratings of each expression in four response formats: emotion categories, affective scales, free response, and assessments of ecological validity.

Participants

Judgments of expressions were obtained using Mechanical Turk. Based on effect sizes observed in a similarly designed study of reported emotional experiences (Cowen & Keltner, 2017), we recruited 1,794 U.S.-English-speaking raters aged 18 to 76 years (940 female, M_age = 34.8 years). Four separate surveys were used: one for categorical judgments, one for appraisal judgments, one for free-response judgments, and one for judgments of ecological validity. The experimental procedures were approved by the Institutional Review Board at the University of California, Berkeley (see the online supplemental materials for details regarding surveys and payment).

Creation of Stimulus Set of 1,500 Naturalistic Expressions

Departing from past studies’ focus on a narrow range of prototypical expressions, and informed by the crucial dependence of multidimensional analytic results on sample size (MacCallum, Widaman, Zhang, & Hong, 1999), we collected widely varying stimuli—1,500 naturalistic photographs in total—centered on the face and including nearby hand and body movements. We did so by querying search engines with contexts and phrases (e.g., “roller coaster ride,” “flirting”) that range in their affect- and appraisal-related qualities and their associations with a rich array of emotion categories. The appraisal properties were culled from theories of the appraisal/construction of emotion. The categories were derived from taxonomies of prominent theorists (see Keltner, Sauter, et al., 2019), along with studies of positive emotions such as amusement, awe, love, desire, elation, and sympathy (Campos, Shiota, Keltner, Gonzaga, and Goetz, 2013); states found to occur in daily interactions, such as confusion, concentration, doubt, and interest (Benitez-Quiroz, Wilbur, & Martinez, 2016; Rozin & Cohen, 2003); and more nuanced states such as distress, disappointment, and shame (Cordaro et al., 2016; Perkins et al., 2012; see also Table S2 of the online supplemental materials for references). These categories by no means represent a complete list of emotion categories, but instead those categories of expressive behavior that have been studied thus far.¹ Given large imbalances in the rates of expressions that appear online (e.g., posed smiles), we selected expressions for apparent authenticity and diversity of expression but not for resemblance to category prototypes. Nevertheless, given potential selection bias and the scope of online images, we do not claim that the expressions studied here are exhaustive of facial-bodily signaling.

Judgment Formats

In each of four different judgment formats, we obtained repeated (18–20) judgments from separate participants of each expression. This was estimated to be sufficient to capture the reliable variance in judgments of each image based on findings from past studies (Cowen et al., 2018; Cowen & Keltner, 2017). One set of participants (n = 364; 190 female; M_age = 34.3 years; mean of 82.4 faces judged) rated each image in terms of 28 emotion categories derived from an extensive review of past studies of emotional expression (e.g., Benitez-Quiroz et al., 2016; Campos et al., 2013; Cordaro et al., 2016; Rozin & Cohen, 2003; see Keltner, Sauter, et al., 2019, for review). The categories included amusement, anger, awe, concentration, confusion, contemplation, contempt, contentment, desire, disappointment, disgust, distress, doubt, ecstasy, elation, embarrassment, fear, interest, love, pain, pride, realization, relief, sadness, shame, surprise, sympathy, and triumph, along with synonyms (see Table S2 of the online supplemental materials). To our knowledge, this list encompasses the emotions for which facial-bodily expressive signals have previously been proposed (and more), but we do not claim that it is exhaustive of all facial-bodily expression. A second set of participants (n = 1,116; 584 female; M_age = 34.3 years; mean of 24.2 faces judged) rated each expression in terms of 13 appraisals derived from constructivist and appraisal theories (Osgood, 1966; Smith & Ellsworth, 1985; Table S3 of the online supplemental materials). Given the potential limitations of forced-choice recognition paradigms (DiGirolamo & Russell, 2017), a third set of participants (n = 193; 110 female; M_age = 37.1 years; mean of 69.9 faces judged) provided free-response descriptions of each expression. In this method, as participants typed, a drop-down menu appeared displaying items from a corpus of 600 emotion terms containing the currently typed substring. For example, typing the substring “lov-” caused the following terms to be displayed: love, brotherly love, feeling loved, loving sympathy, maternal love, romantic love, and self-love. Participants selected as many emotion terms as desired. To address potential concerns regarding ecological validity (Matsumoto & Hwang, 2017), a final group of participants (n = 121; 56 female; M_age = 36.8 years; mean of 37.2 faces judged) rated each expression in terms of its perceived authenticity, how often they have seen the same expression used in real life, how often they themselves have produced the expression, and whether it has reflected their inner feelings. Questions from each survey are shown in Figure S1 of the online supplemental materials.

All told, this amounted to 412,500 individual judgments (30,000 forced-choice categorical judgments, 351,000 9-point appraisal scale judgments, 13,500 free-response judgments, and 18,000 ecological validity judgments). These methods capture 87.6% and 88.0% of the variance in population mean category and appraisal judgments, respectively (see the Supplemental Methods section of the online supplemental materials), indicating that they accurately characterized people’s responses to each expression.

Overview of Statistical Analyses

Hypothesis 1: Dimensionality.

Four methods were used to assess how many distinct varieties of emotion were required to explain participants’ recognition judgments: (a) interrater agreement levels, (b) split-half canonical correlations analysis (SH-CCA) applied to the category judgments, (c) partial correlations between individual judgments and the population average, and (d) canonical correlations analysis (CCA) between category and free-response judgments.

Hypothesis 2: Conceptualization.

Linear and nonlinear predictive models (linear regression and nearest neighbors) were applied to determine how much of the explainable variance in the categorical judgments can be explained by the domain-general appraisals, and vice versa.

Hypothesis 3: Distribution.

Visualization techniques were applied, and we assessed whether gradients between categories of emotion corresponded to discrete shifts or smooth, recognizable transitions in the meaning of expressions by interrogating their correspondence with changes in more continuous appraisals.

Exploratory Question 1: Context.

Judgments of context-free versions of each expression were compared with judgments of the original 1,500 images. We estimated correlations between the population average judgments of expressions with and without context, and performed CCA between the original and context-free judgments to determine the number of preserved dimensions.

Exploratory Question 2: Ecological validity.

We examined how often perceivers rated expressions from each category of emotion as authentic and how often they reported having seen them in real life, produced them in the past, and produced them in reflection of internal feelings.

Results

Distinguishing 28 Distinct Categories of Emotion From Naturalistic Expressions

We first analyzed how many emotions participants recognized at above-chance levels in the 1,500 expressions. Most typically, emotion recognition has been assessed in the degree to which participants’ judgments conform to experimenters’ expectations regarding the emotions conveyed by each expression (Cordaro et al., 2016; Elfenbein & Ambady, 2002). We depart from this prescriptive approach, as we lack special knowledge of the emotions conveyed by our naturalistic stimuli. To capture emotion recognition, we instead focus on the reliability of judgments of each expression, operationalized as interrater agreement (see Jack, Sun, Delis, Garrod, & Schyns, 2016).

We first examined whether the 28 categories of emotion were recognized by traditional standards, that is, with significant interrater agreement rates across participants. We found that the 28 emotions used in our categorical judgment task were recognized at above-chance rates, each category reliably being used to label between three and 125 expressions (M = 65.6 significant expressions per category; see Figure S2 of the online supplemental materials). Ninety-two percent of the 1,500 expressions were reliably identified with at least one category (25%+ interrater agreement, false discovery rate [FDR] < .05, simulation test; see Supplemental Methods section of the online supplemental materials) and over a third of the expressions elicited interrater agreement levels of 50% or greater (FDR < 10⁻⁶, simulation test; see Figure S2 of the online supplemental materials for full distribution of ratings). These interrater agreement levels are comparable with those documented in metaanalyses of past studies of the recognition of emotion in prototypical expressions (Elfenbein & Ambady, 2002), even though the present study included a larger set of emotion categories and variations of expressions within categories. Thus, the 28 categories serve as fitting descriptors for facial-bodily expressions (see Figure S3 of the online supplemental materials for other metrics of importance of each category).

Although we find that the 28 emotion categories were meaningful descriptors of naturalistic expressions, further analyses are needed to show that they each had distinct meaning in how participants labeled each of the 1,500 expressions. How many kinds of emotion were represented along separate dimensions within the semantic space of emotion recognition from facial-bodily expressions? Within our framework, every expression corresponds to a point along some combination of dimensions. Each additional category beyond the first could require a separate dimension, for a maximum of 27, because of the forced-choice nature of the task. Or, as an alternative, the categories could reduce to a smaller number, for example, a low-dimensional space of emotion defined by valence and arousal (Russell, 2003).

Addressing these competing possibilities requires moving beyond traditional methods such as factor analysis or principal components analysis, which are limited in important ways. Most critically, the derivation of the factors or principal components is based on correlations or covariance between different judgments, not on the reliability of reports of individual items (Cowen et al., 2019). Thus, these methods cannot identify when an individual category, like fear, is reliably distinguished from every other rated category (see Supplemental Video 1 of the online supplemental materials for illustration). To determine how many emotion categories are reliably distinguished in the recognition facial-bodily expression, we used the recently validated multidimensional reliability assessment method called SH-CCA (Cowen & Keltner, 2017). In SH-CCA, the averages obtained from half of the ratings for each expression are compared using CCA with the averages obtained from the other half of the ratings. This yields an estimate of the number of dimensions of reliable variance (or kinds of emotion that emerge) in participants’ judgments of the 1,500 naturalistic expressions (see online supplemental materials and Figure S4 for details of the method and validation in 1,936 simulated studies). Using SH-CCA, we found that 27 statistically significant semantic dimensions of emotion recognition (p ≤ .013, r ≥ .019, cross-validated; see Supplemental Methods section of the online supplemental materials for details)—the maximum number for 28-way forced-choice ratings—were required to explain the reliability of participants’ recognition of emotion categories conveyed by the 1,500 expressions (see Figure S5 of the online supplemental materials). This would suggest that, in keeping with our first hypothesis, the expressions cannot be projected onto fewer semantic dimensions without losing information about the emotional states people reliably recognized. Participants recognized 28 distinct states in the 1,500 facial-bodily expressions.

Varieties of Emotion Recognized in Free-Response Description of Expressions

Forced-choice judgment formats may artificially inflate the specificity and reliability of emotion recognition (DiGirolamo & Russell, 2017; Russell, 1994). Given this concern, we assessed how many dimensions of emotion recognition were also reliably identified when participants used their own free-response terms to label the meaning of each expression (see Figure 1). In other words, how many of the dimensions of expression captured by the categorical judgments (e.g., fear, awe) were also reliably described using distinct free-response terms (e.g., shock, wonder)?

Figure 1.

Comparing categorical to free-response and appraisal judgments: All 28 categories are reliable, distinct, and accurate descriptors of perceived emotion and cannot reduced to a small set of domain-general appraisals. (a) Relative usage rates of free-response terms. Font size is proportional to the number of times each term was used. Participants employed a rich array of terms. (b) Canonical correlations between categories and free response. All 27 dimensions of variance of the categorical judgments were recognized via free response (p <.05), indicating that the 28 categories are reliable, semantically distinct descriptors of emotional expression. (c) Free-response terms correlated with each forced-choice category. Correlations (r) were assessed between category judgments and free-response terms, across images. The free-response terms most highly correlated with each category consistently include the categories themselves and synonyms. (d) Proportion of variance explained in 28 categories by 13 affective scales and vice versa using linear and nonlinear regression. Affective scale judgments do not fully explain category judgments (39.9% and 56.1% of variance explained, Ordinary least squares [OLS] linear regression and k-nearest neighbors [kNN] regression, respectively), whereas category judgments do largely explain affective scales (91.4% and 87.1% of variance, OLS and kNN, respectively). Categories offer a richer conceptualization of emotion recognition. The smaller Venn diagrams illustrate results predicted by different theories. Our findings are consistent with theories that expressions convey emotion categories or are high dimensional and cannot be captured by 13 scales (Skerry & Saxe, 2015). See the online article for the color version of this figure.

We answered this question by applying CCA between the category and free-response judgments of the 1,500 expressions. Here, CCA extracted linear combinations among the category judgments of 28 emotions that correlated maximally with linear combinations among the free-response terms. This analysis revealed 27 significant linearly independent patterns of shared variance in emotion recognition (p ≤ .0008, r ≥ .049, cross-validated; see of the online supplemental materials for details)—the maximum attainable number (Figure 1b). Furthermore, forced-choice judgments of each of the 28 categories were closely correlated with identical or near-identical free-response terms (Figure 1c), indicating that they accurately capture the meaning attributed to expressions (ruling out methodological artifacts such as process of elimination; see DiGirolamo & Russell, 2017). Together, these results confirm that all 28 categories are reliable, semantically distinct, and accurate descriptors for the emotions perceived in naturalistic expressions.

How Do People Recognize Emotion in Expressive Behavior: Categories or Domain-General Appraisals?

In comparable studies of emotional experience and vocal expression recognition (Cowen et al., 2018, 2019; Cowen & Keltner, 2017), with respect to how people conceptualize emotion, we found that emotion categories largely explained, but could not be explained by, domain-general appraisals (valence, arousal, dominance, safety, etc.). These results informed our second hypothesis, that emotion categories capture a richer conceptualization of emotion than domain-general appraisals. To test this hypothesis, we ascertained whether judgments of each expression along the 13 appraisal scales would better explain its categorical judgments, or vice versa. Do judgments of the degree to which a sneer expresses dominance, unfairness, and so on better predict whether it will be labeled as “contempt,” or vice versa? Using regression methods (Figure 1d), we found that the appraisal judgments explained 56.1% of the explainable variance in the categorical judgments, whereas the categorical judgments explained 91.4% of the explainable variance in the appraisal judgments (see the online supplemental materials for details). That the categorical judgments explained the appraisal judgments substantially better than the reverse (using linear [91.4% vs. 39.9%] and nonlinear models [87.1% vs. 56.1%]; both ps < 10⁻⁶, bootstrap test) suggests that categories have more value than appraisal features in explicating emotion recognition of facial-bodily expression. Emotion recognition from expression cannot be accounted for by 13 domain-general appraisals.

The Nature of Emotion Categories: Boundaries Between and Variations Within Categories

Our third hypothesis regarded the distribution of emotion recognition. In prior studies, it was found that categories of emotional experience and vocal expression were not discrete but joined by smooth gradients of meaning. We thus predicted that the same would be true of emotion categories perceived in facial-bodily expressions. Our results were in keeping with this prediction. Namely, if we assign each expression to its modal judgment category, yielding a discrete taxonomy, these modal categories explain 78.9% of the variance in the 13 appraisal judgments, substantially less than the 91.4% of variance explained using the full proportions of category judgments (p < 10⁻⁶, bootstrap test). Confusion over the emotion category conveyed by each expression, far from being noise, instead carries information regarding how it is perceived.

To explore how the emotions recognized in facial-bodily expression were distributed, we used a visualization technique called t-distributed stochastic neighbor embedding (t-SNE; Maaten & Hinton, 2008), which projects data into a two-dimensional space that preserves the local distances between data points without imposing assumptions of discreteness (e.g., clustering) or linearity (e.g., factor analysis). The results are shown in Figure 2 and online at https://s3-us-west-1.amazonaws.com/face28/map.html . The map reveals within-category variations in the relative location of each expression, in keeping with the notion that within an emotion category there is heterogeneity of expressive behavior (Ekman, 1992, 1993; Keltner, 1995). For example, some expressions frequently labeled “surprise” were placed closer to “fear” and others closer to “awe.”

Figure 2.

Mapping of expression judgment distributions with t-distributed stochastic neighbor embedding (t-SNE). t-SNE projects the data into a two-dimensional space that approximates the local distances between expressions. Each number corresponds to a single naturalistic expression and reflects the most frequent category judgment. The t-SNE map reveals smooth structure in the recognition of emotion, with variations in the proximity of individual expressions within each category to other categories. These variations are exemplified by the example expressions shown. Face images have been rendered unidentifiable here via artistic transformation. The expressions and their assigned judgments can be fully explored in the interactive online version of this figure: https://s3-us-west-1.amazonaws.com/face28/map.html. See the online article for the color version of this figure.

To quantify the gradients revealed by t-SNE, we tested statistically for potential nondiscrete structure in the categorical judgments. We investigated whether expressions rated with two emotion categories (e.g., awe and fear) scored intermediate ratings between the categories on our 13 appraisal scales. If categories were perceived as discrete, then the meaning attributed to each expression would (a) shift abruptly across category boundaries, and (b) have higher standard deviation across raters at the midpoint between categories, as expressions would be perceived as belonging to either one category or the other. Instead, we found that patterns in the 13 appraisal scales were significantly correlated with patterns in the categorical judgments within (not just across) modal categories, and location near the midpoint along gradients between categories was typically uncorrelated with standard deviation in the appraisal features (mean correlation r = .00039; see the online supplemental materials for details).² That the appraisal scales vary smoothly both within and across modal category boundaries confirms that the categories occupy continuous gradients.

The Role of Context: Does the Recognition of 28 Emotion Categories Depend on Contextual Cues?

In natural settings, facial-bodily expressions are framed by contextual cues: clothing, physical settings, activities individuals may be engaged in, and so forth. In the interest of ecological validity, many of the expressions presented here included such cues. To ascertain whether these contextual cues were necessary to distinguish certain categories of emotion from expression, we eliminated them from each image. To do so, we employed facial landmark detection and photographic interpolation techniques to crop out or erase visual details besides those of the face and body. We eliminated scenery (e.g., sporting fields, protest crowds), objects (e.g., headphones), clothing, jewelry, and hair when its position indicated movement.³ We were able to remove contextual cues from all but 44 images in which contextual cues overlapped with the face (we allowed hand and body gestures to be fully or partially erased from many of the photographs; see Figure 3a for examples). We then collected 18 forced-choice categorical judgments of each of these context-free expressions from a new set of participants (N = 302; 151 female, M_age = 37.6 years).

Figure 3.

The recognition of 28 distinct categories of emotion from naturalistic expression does not require contextual cues. (a) Examples of images before and after the removal of contextual cues. (b) Correlations between judgments of 1,456 expressions in their natural context and in isolation, adjusted for explainable variance. (c) Canonical correlations between judgments of expressions in context and in isolation. All 27 dimensions of recognized emotion were preserved after the removal of contextual cues (p < .01; leave-one-rater-out cross-validation; see the online supplemental materials for details). (d) Categorical judgments of the original and isolated images. Here, colors are interpolated to precisely capture the proportion of times each category was chosen for each expression. The resulting map reveals that the smooth gradients among distinct categories of emotion are preserved after the removal of contextual cues. Face images have been rendered unidentifiable here via artistic transformation. The isolated expressions and their assigned judgments can be fully explored within another online interactive map: https://s3-us-west-1.amazonaws.com/face28/nocontext.html. See the online article for the color version of this figure.

A comparison of judgments of the 1456 context-free expressions with the corresponding judgments of the original images revealed that context played, at best, a minor role in the recognition of the 28 categories of emotion. As shown in Figure 3b, judgments of the 28 emotion categories of the context-free expressions were generally highly correlated with judgments of the same categories in the original images (r > .68 for all 28 categories, and r > .9 for 18 of the categories). Furthermore, applying CCA between judgments of the original and context-free expressions again revealed 27 significant dimensions (p ≤ .033, r ≥ 030, cross-validated; see the online supplemental materials for details), the maximum number (Figure 3c). And finally, projecting the judgments of the context-free expressions onto our chromatic map revealed that they are qualitatively very similar to those of the original images (Figure 3d).

These results indicate that 28 categories of emotion can be distinguished from facial-bodily posture absent contextual clues. Although a few expressions are more context-dependent—especially triumph, love, and pride (see Figure 3d)—the vast majority are perceived similarly when isolated from their immediate context. Isolated facial and bodily expressions are capable of reliably conveying 28 distinct categories of emotion absent contextual cues.

Ecological Validity of Expressions of 28 Emotions

The focus of past studies on posed expressions raises important questions about how often the 1,500 expressions we have studied occur in daily life (Matsumoto & Hwang, 2017; Russell, 1994). To examine the ecological validity of the 1,500 expressions presented here, we asked an independent group of participants to rate each expression in terms of authenticity, how often they have seen the same expression in daily life, how often they have produced the expression, and whether it has reflected their inner feelings (for a similar approach, see Rozin & Cohen, 2003). Overall, expressions were reported as authentic 60% of the time, as having been seen in daily life 92% of the time, as having been produced in daily life 83% of the time, and as having been produced in reflection of raters’ inner feelings 70% of the time. Figure 4a presents these results broken down by emotion category. Expressions labeled with each category were reported to have been witnessed, on average, by 82% (“desire”) to 97% (“sympathy”) of participants at some point in their lives, confirming that many of the expressions in each emotion category were ecologically valid.

Figure 4.

Expressions of each of the 28 emotion categories are widely reported to occur in real life. (a) Percentage of perceivers who rated expressions from each category of emotion as authentic, as having been seen in real life, as having been produced by the perceiver in the past, and as having been produced by the perceiver in reflection of internal feelings. Percentages are weighted averages for each category; that is, an expression that was rated “amusement” by 50% of raters in the categorical judgment task would count twice as much toward the averages shown here as an expression rated “amusement” by 25% of raters. Expressions in every category were rated as authentic and as having been perceived and produced in real life by a substantial percentage of raters. (b) How often people reported having perceived and produced expressions from each category. Expressions from all of the emotion categories were rated as having been perceived and produced somewhat regularly (e.g., daily, weekly or monthly) by a substantial proportion of raters. See the online article for the color version of this figure.

To verify that all 28 categories of emotion could be distinguished in ecologically valid expressions, we applied CCA between judgments of the original and context-free expressions, this time restricting the analysis to 841 expressions that were rated as authentic and having been encountered in real life by at least two thirds of raters. This analysis still revealed 27 significant dimensions (p < .01; leave-one-rater-out cross-validation; see online supplemental materials for details), the maximum number. Ecologically valid expressions convey 28 distinct emotion categories.

A further question is how frequently these expressions occur in daily life. People’s self-reports suggest that a wide range of expressions occur regularly. Figure 4b shows the frequency of reported occurrence of expressions in each category, in terms of reported perception and production of each expression. Fifty-seven percent of the expressions were reported as having been perceived in real life on at least a monthly basis, 34% on at least a weekly basis, and 16% on at least a daily basis. The percentage of raters reporting at least weekly perception of individual expressions from each emotion category varied from 17% (expressions of “pain”) to 52% (expressions of “amusement”). Weekly rates of expression production varied from 15% (“pain”) to 49% (“amusement”). Presumably, expressions within an emotion category can vary considerably in how often they occur—for instance, expressions of extreme pain are likely rarer than expressions of slight pain. Nevertheless, the present findings suggest that at least 28 categories of emotion are regularly perceived and experienced by a considerable portion of the population.

Inspection of Figure 4b also reveals striking findings about the profile of the expressions that people observe and produce in their daily lives (Rozin & Cohen, 2003). The most commonly studied emotions—“anger,” “disgust,” “fear,” “sadness,” and “surprise”—were reported to occur less often than expressions of understudied emotions such as “awe,” “contempt,” “interest,” “love,” and “sympathy” in terms of daily, weekly, and monthly perception and production. These findings reinforce the need to look beyond the six traditionally studied emotions.

Discussion

Evidence concerning the recognition of emotion from facial-bodily expression has been seminal to emotion science and the center of a long history of debate about the process and taxonomy of emotion recognition. To date, this research has largely been limited by its focus on prototypical expressions of a small number of emotions; by conceptual approaches that conflate the dimensionality, conceptualization, and distribution of emotion; and by limitations in statistical methods, which have not adequately explored the number of reliably recognized dimensions of emotion, modeled whether emotion recognition is driven by emotion categories or dimensions of appraisal, or mapped the distribution of states along these dimensions.

Guided by a new conceptual approach to studying emotion-related responses (Cowen et al., 2018, 2019; Cowen & Keltner, 2017, 2018), we examined the dimensionality, conceptualization, and distribution of emotion recognition from facial-bodily expression, testing claims found in basic emotion theory and constructivist approaches. Toward these ends, we gathered appraisal, categorical, free-response, and ecological validity judgments of over naturalistic 1,500 facial-bodily expressions of 28 emotion categories, many of which have only recently been studied.

In keeping with our first hypothesis, large-scale statistical inference yielded evidence for the recognition of 28 distinct emotions from the face and body (Figure 2, and Figures S2–S4 of the online supplemental materials). These results were observed both in forced-choice and free-response formats (DiGirolamo & Russell, 2017; Russell, 1994). We document the most extensive, systematic empirical evidence to date for the distinctness of facial-bodily expressions of positive states such as amusement, awe, contentment, desire, love, ecstasy, elation, and triumph, and negative states such as anger, contempt, disappointment, distress, doubt, pain, sadness, and sympathy. These results overlap with recent production studies showing that over 20 emotion categories are expressed similarly across many cultures (Cordaro et al., 2018) and suggest that the range of emotions signaled in expressive behavior is much broader than the six states that have received most of the scientific focus.

It is worth reiterating that the expressions studied here are likely not exhaustive of human expression—rather, our findings place a lower bound on the complexity of emotion conveyed by facial-bodily signals. The present study was equipped to discover how facial-bodily expressions distinguish up to 28 categories of emotion of wide-ranging theoretical interest, informed by past studies. We were surprised to find that perceivers could reliably distinguish facial-bodily expressions of every one of these 28 emotion categories (see Cowen et al., 2018, and Cowen & Keltner, 2017, for studies in which perceivers could only distinguish a subset of categories tested). These results should motivate further studies to examine an even broader array of expressions. It is also worth noting that, as with any dimensional analysis, the semantic space we derive is likely to be one of many possible schemas that effectively capture the same dimensions. For example, an alternative way to capture “awe,” “fear,” and “surprise” expressions might be with categories that represent blends of these and other emotions, such as “horror,” “astonishment,” and “shock.”

Our second hypothesis held that the recognition of emotion categories would not be reducible to domain-general scales of affect and appraisal, including valence and arousal, as is often assumed. In keeping with this hypothesis, we found that 13 domain-general appraisals captured 56.1% of the variance in categorical judgments, whereas emotion categories explained 91.4% of the variance in these appraisals (Figure 1d). This pattern of results poses problems for the perspective that the recognition of emotion categories is constructed out of a small set of general appraisals including valence and arousal (Barrett, 2017; Russell, 2003, 2009). Instead, our results, now replicated in several modalities, support an alternative theoretical possibility: that facets of emotion—experience, facial/bodily display, and, perhaps, peripheral physiology—are more directly represented by nuanced emotion categories, and that domain-general appraisals capture a subspace of—and thus may potentially be inferred from—this categorical semantic space.

With respect to the distribution of emotion, our third hypothesis held that the distribution of emotion categories would not be discrete but instead linked by smooth dimensional gradients that cross boundaries between emotion categories. This contrasts with a central assumption of discrete emotion theory, that at the boundaries of emotion categories—for example, awe and interest—there are sharp boundaries defined by step-like shifts in the meaning ascribed to an emotion-related experience or expression (Ekman & Cordaro, 2011). Our results pose problems for this assumption. As shown in Figure 2, smooth gradients of meaning link emotion categories. In addition, we were able to demonstrate, using the appraisal scales in conjunction with the category judgments, that confusion over the category of each expression is meaningful (Figures 1 and 3, and Figures S5–S7 of the online supplemental materials). In particular, the position of each expression along a continuous gradient between categories defines the precise emotional meaning conveyed to the observer. This latter finding suggests that emotion categories do not truly “carve nature at its joints” (Barrett, 2006).

A further pattern of results addressed the extent to which the recognition of emotion is dependent upon context. Most naturalistic expressions of emotion occur in social contexts with myriad features—signs of the setting, people present, evidence of action. Such factors can, in some cases, powerfully shape the meaning ascribed to an expressive behavior. For example, perceptions of disgust depend on whether an individual is engaged in action related to disgust (e.g., holding a soiled article of clothing) or another emotion (e.g., appearing poised to punch with a clenched fist; Aviezer et al., 2008). However, our results suggest that such findings concerning the contextual shaping of the recognition of emotion from facial-bodily expression may prove to be the exception rather than the rule. Namely, across 1,500 naturalistic expressions of 28 emotion categories, many sampled from actual in vivo contexts, all 28 categories could reliably be recognized from facial-bodily expression alone, and their perception was typically little swayed by nuanced contextual features such as clothing, signs of the physical and social setting, or cues of the actions in which people were engaged (see Figure 3).

Our final pattern of findings addressed the ecological validity of our results. People report having perceived and produced most expressions of each of the 28 distinct emotion categories in daily life (see Figure 3). Judging by people’s self-report, none of the 28 categories of emotional expression we uncovered are particularly rare occurrences; on any given day, a considerable proportion of the population will witness similar expressions of each emotion category.

Taken together, the present results yield a nuanced, complex representation of the semantic space of facial-bodily expression. In different judgment paradigms, observers can discern 28 categories of emotion from facial-bodily behavior. Categories capture the recognition of emotion more so than appraisals of valence and arousal. However, in contrast to discrete emotion theories, expressions are distributed along smooth gradients between many categories.

To capture the emotions conveyed by facial-bodily expression, we focused here on recognition by U.S. English speakers, leaving open whether people in other cultures attribute the same emotions to these expressive signals. On this point, recent studies have suggested that people in at least nine other cultures similarly label prototypical expressions of many of the emotions we uncovered (e.g., Cordaro et al., in press). However, such studies have not yet incorporated broad enough arrays of expressions to understand what is universal about the conceptualization, dimensionality, and distribution of emotion recognition. Moving in this direction, a recent study of emotion recognition from speech prosody found that at least 12 dimensions of recognition are preserved across the United States and India, and that cultural similarities are driven by the recognition of categories such as “awe” rather than appraisals such as valence and arousal (Cowen et al., 2019). Similar methods can be applied to examine universals in the recognition of facial-bodily expression, guided by the semantic space we derive here.

Another important limitation of our findings, specifically those regarding the dimensionality of emotion, is that they rely on linear methods. We find that expressions convey at least 27 linearly separable dimensions of emotion (Figures 1b, 2, and 3). Given that the vast majority of psychology studies employ only linear models, such findings are critical. A linear model of emotion must utilize variables that linearly capture the variance in emotion-related phenomena, and Figure 1d indicates that emotion categories linearly capture nearly all of the explainable variance in appraisal judgments (valence, arousal, and so on). Using nonlinear predictive models, we have also ruled out hypotheses that valence, arousal, dominance and 10 other domain-general appraisals can account for the recognition of emotional categories. These findings rule out most low-dimensional theories of emotional expressions (Lang, Greenwald, Bradley, & Hamm, 1993; Mauss & Robinson, 2009; Russell, 2003, 2009; Watson & Tellegen, 1985). Nevertheless, it is possible that emotion recognition can be represented nonlinearly using other undiscovered dimensions, possibly ones that are not directly represented in language. Testing for such nonlinear dimensions is a difficult problem that will require methods such as training a deep autoencoder (a type of artificial neural network) on a sufficiently rich data set.

By applying high-dimensional statistical modeling approaches to large-scale data, we have uncovered an initial approximation of the semantic space of emotion conveyed by facial-bodily expression. It is worth noting that 17 of the emotions we find to be distinguished by facial-bodily expression were previously found to be distinguished by both vocal expression (Cowen et al., 2018, 2019) and experiences evoked by video (Cowen & Keltner, 2017), including amusement, anger, anxiety/distress, awe, confusion, contentment/calmness, desire, disgust, elation/joy, embarrassment/awkwardness, fear, interest, love/adoration, pain, relief, sadness, and surprise. Taken together, these findings illuminate a rich multicomponential space of emotion-related responses.

Facial-bodily signals have received far more scientific focus than other components of expression, perhaps because of their potency: They can imbue almost any communicative or goal-driven action with expressive weight, can be observed passively in most experimental contexts, and are routine occurrences in daily life. More focused studies of particular facial-bodily expressions have revealed their powerful influence over critical aspects of our social lives, including the formation of romantic partnerships (Gonzaga et al., 2001), the forgiveness of transgressions (Feinberg et al., 2012), the imitation of role models (Martens & Tracy, 2013), the resolution of negotiations (Reed et al., 2014), and even the allocation of investments (ten Brinke & Adams, 2015). Such findings can be refined and profoundly expanded upon by applying measurements and models guided by a broader semantic space of expression. In many ways, the understanding of how emotion-related responses are arranged within a high-dimensional space has the promise of revamping the study of emotion.

Biographies

Alan S. Cowen

Dacher Keltner