Seeing Mixed Emotions: The Specificity of Emotion Perception From Static and Dynamic Facial Expressions Across Cultures

Abstract

Although perceivers often agree about the primary emotion that is conveyed by a particular expression, observers may concurrently perceive several additional emotions from a given facial expression. In the present research, we compared the perception of two types of nonintended emotions in Chinese and Dutch observers viewing facial expressions: emotions which were morphologically similar to the intended emotion and emotions which were morphologically dissimilar to the intended emotion. Findings were consistent across two studies and showed that (a) morphologically similar emotions were endorsed to a greater extent than dissimilar emotions and (b) Chinese observers endorsed nonintended emotions more than did Dutch observers. Furthermore, the difference between Chinese and Dutch observers was more pronounced for the endorsement of morphologically similar emotions than of dissimilar emotions. We also obtained consistent evidence that Dutch observers endorsed nonintended emotions that were congruent with the preceding expressions to a greater degree. These findings suggest that culture and morphological similarity both influence the extent to which perceivers see several emotions in a facial expression.

Facial expressions serve as a primary means to communicate emotions in social interactions and thereby convey critical information about a person’s appraisal of a situation and about his or her goals, desires, and social intentions (Frijda & Mesquita, 1994; Manstead & Fischer, 2001; Van Kleef, 2009). Recognizing the “correct” emotion from an expresser’s facial expression is therefore crucial to successful communication and social interaction. But although perceivers often agree about the primary emotion that is conveyed by a particular expression (Elfenbein & Ambady, 2002, 2003; Scherer & Wallbott, 1994), viewers may concurrently perceive several additional emotions from a given facial expression, which may also influence their interpretation of the expresser and the situation. However, the perception of such nonprimary emotions from emotional expressions is poorly understood.

The communication of emotions via nonverbal expressions comprises two distinct but interrelated processes—expression and perception (Scherer, Clark-Polner, & Mortillaro, 2011; Van Kleef, 2016). The expression consists of an externalization of internal emotion processes resulting in, for example, the production of a facial expression (see Lench, Flores, & Bench, 2011). Perception refers to observers inferring what emotion(s) the expresser is experiencing, based on the physical signal that the observer perceives. In the present research, we set out to establish the role of two factors in the perception of nonintended emotions from both static and dynamic facial expressions, namely, the morphological features of facial expressions of emotion (relating to expression) and the culture of the perceiver (relating to perception).

Morphological Similarity Between Facial Expressions

Previous research has shown that observers can perceive multiple emotions in a facial expression (Ekman et al., 1987; Kayyal & Russell, 2013; Yrizarry, Matsumoto, & Wilson-Cohn, 1998). To account for this phenomenon, Yrizarry and colleagues (1998) proposed a potential explanation suggesting that expressions that share facial components (i.e., morphological similarities) may lead observers to partially endorse multiple emotions. For example, observers are likely to perceive both anger and disgust in an angry face because the facial expression of disgust is morphologically similar to the expression of anger. The facial expressions of anger and disgust share key component features, namely, the eyes narrowing (Action Unit [AU] 7 in Ekman, Friesen, and Hager’s [2002] Facial Action Coding System) and the eyebrows lowering (AU4; Ekman, 1993; Ekman, Friesen, & Hager, 2002). In addition to anger and disgust, facial expressions of fear and surprise are also morphologically similar to one another (Aviezer et al., 2008; Susskind, Littlewort, Bartlett, Movellan, & Anderson, 2007; Young et al., 1997) as they both involve the eyes widening (AU5) and the eyebrows raising (AU1/AU2; Ekman, 1993; Ekman et al., 2002).

Based on these considerations, we hypothesized that the extent to which other (i.e., nonintended) emotions are perceived from a facial expression should depend in part on the morphological similarity to the intended emotion.¹ For example, when confronted with a facial expression of anger, perceivers are also likely to perceive some degree of disgust; in contrast, the morphological dissimilarity between the facial expressions of anger and fear makes perceivers less likely to infer fear from facial expressions of anger. In the current study, we compared the extent to which perceivers infer multiple concurrent emotions from facial expressions, testing the prediction that a given facial expression (e.g., anger) would be rated higher on morphologically similar emotion dimensions (e.g., disgust) than on dissimilar emotion dimensions (e.g., fear; Hypothesis 1). Building on this basic prediction, we developed one more substantive hypothesis relating to culture, as well as two exploratory predictions, which is outlined in the following sections.

Culture and Emotion Perception

Although the intended emotion perceived in an expression may be shared across cultures (e.g., Elfenbein & Ambady, 2003; Sauter, Eisner, Ekman, & Scott, 2010; Scherer & Wallbott, 1994), the extent to which other emotions are perceived from that expression may differ. Previous findings indicate that East Asians (henceforth Easterners) tend to experience multiple different emotions concurrently, while North Americans and Europeans (henceforth Westerners) are likely to experience specific feelings (Grossmann, Huynh, & Ellsworth, 2015; Miyamoto, Uchida, & Ellsworth, 2010).² This implies that Easterners are more inclined to experience blends of emotions than Westerners. In addition to this difference in emotional experience, Easterners also use more nonintended emotion words than Westerners to describe facial expressions (Beaupr & Hess, 2005; Leu, Mesquita, Masuda, Ellsworth, & Karasawa, 2003). This may reflect Easterners’ interdependent, holistic view of the self in relation to the social environment. Specifically, Easterners emphasize the relation of the self with others, tend to attend to the entire field, and make relatively little use of categories. In contrast, Westerners emphasize the independent self and pay attention primarily to the focal object and the categories to which it belongs (Ji, Peng, & Nisbett, 2000; Markus & Kitayama, 1991; Nisbett, Peng, Choi, & Norenzayan, 2001). As a result, Easterners may be more attuned to the presence of multiple different emotions (and their interrelations) within an expression compared with Westerners, who would be attuned primarily to the most salient emotion category.

In addition to cultural differences in terms of interdependence/independence and holistic/analytic cognitive processes, Easterners are considered to be more dialectical thinkers than Westerners (Peng & Nisbett, 1999; Peng, Spencer-Rodgers, & Nian, 2006). This means that Easterners are less troubled by apparent contradictions in their own thoughts, emotions, and behaviors as well as those of others (Hideg & Van Kleef, 2017). In contrast, Westerners typically look to reconcile inconsistencies and are more disturbed by discrepancies (Spencer-Rodgers, Peng, Wang, & Hou, 2004). As specific emotions are associated with distinct combinations of appraisals and action tendencies (Ellsworth & Smith, 1988; Frijda, Kuipers, & ter Schure, 1989; Scherer, 1984), endorsing concurrent distinct emotions is likely to represent a contradictory belief. Easterners should therefore be more likely to endorse multiple emotions than Westerners. Based on these interrelated accounts, we predicted that nonintended emotions would be perceived as stronger by Eastern compared with Western perceivers (Hypothesis 2). It is worth noting that this inference is different from the argument that Eastern perceivers cannot reliably distinguish certain facial expressions of emotion in forced-choice tasks due to, for example, perceptual biases (see Jack, Blais, Scheepers, Schyns, & Caldara, 2009; Jack, Caldara, & Schyns, 2012). We return to this issue in the “General Discussion” section.

Furthermore, we examined whether the tendency of Easterners to endorse nonintended emotions to a greater degree than Westerners would vary as a function of the morphological similarity of emotions. Since an interdependent/holistic view is reflected in the perception of overlapping of elements (Ji et al., 2000; Markus & Kitayama, 1991; Nisbett et al., 2001), groups that differ in terms of interdependent/holistic thought may vary more strongly on the endorsement of elements that are similar to the target (see Beaupr & Hess, 2005). In the present context, Easterners, who hold an interdependent/holistic view, may thus be expected to perceive more nonintended emotions that are close to the intended emotion (i.e., morphologically similar emotions) than Westerners, who hold an independent/analytic view. Given the limited theoretical and empirical support for this account, we conducted an explorative analysis to test whether cultural differences would be more pronounced for morphologically similar emotions than for morphologically dissimilar emotions.

Dynamic Facial Expressions

Though the majority of research on facial expressions of emotion employs static images, in real life, facial expressions are typically dynamic, changing from one state (henceforth the start emotion) to another state (the end emotion; Krumhuber, Kappas, & Manstead, 2013). We therefore tested the above-mentioned hypotheses using dynamic emotion displays as well as static emotional expressions, to improve both ecological validity and generalizability. In addition, the inclusion of dynamic displays allowed for an internal replication of our findings from static displays.

Unlike static emotion displays, dynamic emotional expressions include a preceding expression as well as an end expression. Despite the ubiquity of successive contexts in real-life emotion perception, surprisingly, little research has studied the influence of a preceding expression on the perception of an end facial expression. One study by Russell and Fehr (1987) provided initial evidence for an effect of successive contextual information. In their experiment, participants saw two photographs of different facial expressions in succession. The authors found that viewing a preceding expression shifted the judgment of the end expression in the opposite direction to that of the preceding expression. For example, a relatively neutral face was perceived as more sad when presented after a happy face. More recent work has used morphs of emotional changes to present facial expressions sequentially (Sacharin, Sander, & Scherer, 2012), again highlighting that how an end expression is decoded can be influenced by its context, in this case, the preceding expression.

Related to interdependence/independence and holistic/analytic cognitive processes, the contextual effects may vary across cultures because Easterners and Westerners tend to assign attention differently to contextual information. In general, Easterners attend more to contextual information, whereas Westerners tend to pay more attention to the target and to perceive the target independently from the background (Nisbett et al., 2001). This difference in attention to context has been linked to the perception of emotional expressions in Eastern and Western observers. A study by Masuda and colleagues (2008) found that when seeing a cartoon figure displaying an emotional facial expression surrounded by other expressions, Easterners’, but not Westerners’, perceptions were influenced by the emotion of the surrounding figures. Stanley, Zhang, Fung, and Isaacowitz (2013) suggested that Westerners saw emotions as individual feelings, whereas Eastern observers saw them as inseparable from the feelings of the group. The temporal context used in the present research (the expresser’s preceding expression) is different from the concurrent context manipulations (e.g., other individuals’ expressions) used in previous cross-cultural research on contextual influences on emotion perception. Attending to the relationships between the self and others might make Easterners focus more on others’ expressions but may not necessarily lead to Easterners paying more attention to their own preceding expressions. However, research in cognitive psychology suggests that contextual effects can occur across domains, including spatial and temporal contexts (e.g., Hansen, Walter, & Gegenfurtner, 2007; Indefrey & Levelt, 2004). Cultural differences in contextual effects exerted by the concurrent context may thus generalize to the temporal context. Based on these considerations, we exploratively examined whether the temporal context effects on the perception of nonintended emotions would vary across cultures (Easterners vs. Westerners).

The Present Research

We conducted two experimental studies to test the above hypotheses. Since perceivers from both East and West were to be tested, both Eastern and Western facial expressions of emotion were included. In each study, participants were asked to rate the intensities of three specified emotions from both static expressions (static task) and dynamic expressions (dynamic task). In Study 1, the emotions included were anger, disgust, and fear, which are all negative in valence and high in arousal (Russell & Barrett, 1999). Among them, anger and disgust are associated with morphologically similar facial expressions, whereas anger versus fear and disgust versus fear have morphologically dissimilar expressions (Aviezer et al., 2008; Susskind et al., 2007). In the static task, participants were presented with photographs of facial expressions, while in the dynamic task, participants were presented with morphs of dynamic facial expressions that changed between two emotions. In Study 2, we used a different set of emotions to examine whether the results of Study 1 were robust. Specifically, Study 2 examined the perception of facial expressions of anger, fear, and surprise, a set that also contains morphologically similar facial expressions (fear–surprise) and morphologically dissimilar facial expressions (anger–surprise and anger–fear; Aviezer et al., 2008; Susskind et al., 2007).

It is noteworthy that the focus of the present research is on the perception of multiple emotions, particularly nonintended emotions, from facial expressions. Although there is a great deal of cross-cultural research on emotion recognition from facial expressions (e.g., Ekman et al., 1987; Jack et al., 2009; Yrizarry et al., 1998), that work has emphasized the accuracy of emotion recognition; that is, the correct identification of intended emotions from facial expressions. Here, emotion perception is described as the extent to which emotions are endorsed from facial expressions, rather than considered in terms of accuracy. One of our goals with this research is to underscore the importance of investigating the perception of nonintended emotions from facial expressions, particularly in cross-cultural studies.

Development of Stimuli and Pilot Study

Six Dutch actors (three male) posing facial expressions of anger, disgust, fear, and surprise were selected from the Radboud Faces Database (Langner et al., 2010). Six Chinese actors expressing the same emotions were selected from the Taiwan Corpora of Chinese Emotions (Shyi, Huang, & Yeh, 2013), based on one-to-one correspondence to the Dutch stimuli in terms of gender, appearance, and approximate age (see supplementary Figure S1 for examples of facial expression stimuli). Both facial expression databases were produced using instructions based on the Facial Action Coding System (FACS; Ekman et al., 2002). Thus, the Eastern and Western facial expression stimuli used in the present research were matched in terms of the muscles (“action units”) activated for each emotion (see supplementary Table S5 for an overview of action units and supplementary Table S6 for a test of the morphological similarities between the facial expression stimuli).

A pilot study was conducted to ensure a match of intensity between the Dutch and Chinese stimuli. A photograph of one of the Dutch facial expressions was presented on the left side of the screen, while the corresponding Chinese facial expression (morphed to change from neutral to a specific emotion in 26 frames) was presented on the right side of the screen. Participants were asked to drag the slider bar under the Chinese morph to choose the frame that was most similar in terms of intensity to the Dutch stimulus on the left side. Each comparison between a Chinese stimulus and a Dutch stimulus included two trials with different initial positions of the slider bar under the Chinese morph, with one starting from the first frame (minimum intensity, that is, fully neutral with no emotion) and the other one starting from the last frame (maximum intensity, that is, fully emotional with no neutral). In total, each participant completed 48 trials (6 actors × 4 emotions × 2 initial positions of the slider bar) in a random order. Twenty Dutch participants from the University of Amsterdam and 20 Chinese participants from Zhejiang University were recruited for the pilot test. No significant differences were found between the two groups of perceivers, so the average frame across the two groups of perceivers was used as the final stimulus for each individual and emotion in the Chinese set.

We used a digital morphing program (FantaMorph; http://www.fantamorph.com/) to produce 26-frame morphs, in which a start emotional expression changed into an end expression for the same individual’s face (see Figure 1 for an illustration). The morphs were presented at the speed of 30 frames per second, which has been found to adequately reflect natural changes in the dynamic facial expressions (Ambadar, Schooler, & Cohn, 2005; Sato & Yoshikawa, 2004). Consequently, each frame lasted for 33.33 ms. Based on the unfolding time of natural dynamic emotional expressions (Hoffmann, Traue, Bachmayr, & Kessler, 2010), the exposure time of the first frame (i.e., the start emotion) and the last frame (i.e., the end emotion) were extended to 600 ms each. Thus, each clip lasted for 2,000 ms. In total, 72 morphs (3 actors × 2 actor ethnicities × 2 genders × 3 emotion pairs × 2 directions) were included for anger–disgust–fear (Study 1) and the same number of morphs was included for anger–fear–surprise (Study 2).

Figure 1.

The procedure and materials adopted in the static and dynamic tasks of Studies 1 and 2.

Note. (a) In the static task, each trial started with a fixation cross displayed for 500 ms, followed by a photograph shown for 1,000 ms, and ending with a screen with three scales asking participants to indicate the emotions they perceived on the face. In the dynamic task, each trial started with a fixation cross lasting for 500 ms, followed by a dynamic (morphed) clip lasting a total of 2,000 ms, and ending with a screen with three scales asking participants to indicate the last emotional expression they perceived on the face. (b) A facial expression of anger, as used in the static task (above), and a subset of the morphs shown in a dynamic clip of disgust-to-anger, as used in the dynamic task (below).

The emotion terms in the current research were taken from the respective facial expression databases (i.e., the terms are used to denote the various facial expressions in each stimulus set; see Table 1). They were back-translated into English by a native speaker of each language to verify accurate translation. The instructions were translated from English into Chinese and Dutch by means of the standard translation/back-translation procedure.

Table 1.

Chinese and Dutch Translations of Emotion Terms.

English	Dutch	Chinese
Anger	Boosheid	愤怒
Disgust	Walging	厌恶
Fear	Angst	恐惧
Surprise	Verrassing	惊讶

Study 1

Method

Participants

The study included 222 individuals, 11 participants were excluded because they merely clicked through the trials without performing the task. The final sample consisted of 134 Dutch participants (M_age = 22.3, SD = 3.46; 106 females) from the University of Amsterdam and 77 Chinese participants (M_age = 21.1, SD = 2.02; 49 females) from Zhejiang University. Among the Dutch participants, 95 people were presented with Dutch facial expressions (M_age = 22.8, SD = 3.77; 71 females) and 39 people (M_age = 21.1, SD = 2.32; 32 females) were presented with Chinese facial expressions. Among the Chinese participants, 38 people (M_age = 20.9, SD = 1.98; 25 females) were presented with Chinese facial expressions and 39 people (M_age = 21.4, SD = 2.08; 22 females) were presented with Dutch facial expressions.³ In return for participation, Dutch participants received one course credit or €10, whereas Chinese participants received 30 yuan (about €4).

Apparatus

The tests were controlled by a custom-written PsychoPy program (Psychophysics software in Python; Peirce, 2007) and implemented on a Windows 7 computer. The monitor was 24 inch, with a screen resolution of 1920 × 1080 pixels, and a screen refresh rate of 60 Hz.

Procedure

In the static task, each trial started with a fixation cross displayed in the center of the screen for 500 ms, followed by a photograph shown for 1,000 ms. Participants were seated approximately 60 cm from the screen, and the photographs subtended 8° × 12° of their visual angle. Immediately following the photograph, a gray screen appeared with a prompt to judge the emotion expressed in the photograph. Ratings were made by moving sliders on three scales (anger, disgust, and fear), ranging from 0 (not at all) to 100 (very much). All scales were displayed on a single screen, with the order consistent for each subject, but counterbalanced between subjects. Participants completed three practice trials, followed by four blocks of 18 trials each. Each stimulus appeared once in each block, and the order was random.

In the dynamic task, the procedure was identical to that of the static task except that the photographs were replaced with movie clips, which were displayed for 2,000 ms. Participants were asked to judge the last emotion expressed in the video. Participants completed six practice trials, followed by four blocks of 36 trials each. Each stimulus appeared once in each block, and the order was random (see Figure 1).

The total study consisted of a static task (15 min), a dynamic task (30 min), and an unrelated filler task in between to separate the two conditions (15 min).⁴ The order of the static and dynamic tasks was counterbalanced between participants. On average, participants took 60 min to finish the experimental session.

Results and Discussion

Static task

For facial expressions of anger and disgust, in addition to the intended emotion scale, ratings were made of both morphologically similar (disgust/anger) and dissimilar (fear) emotions. In contrast, for facial expressions of fear, in addition to the intended emotion scale, ratings were only made of two nonintended emotion scales of morphologically dissimilar emotions (anger and disgust).⁵ As we were interested in possible intensity differences between perceptions of morphologically similar and dissimilar emotions, our analyses focused only on the nonintended ratings of anger and disgust expressions (48 out of 72 trials per participant).

We examined the nonintended emotion ratings in a 2 (Scale Similarity: Similar, Dissimilar) × 2 (Culture of Perceiver: Chinese, Dutch) × 2 (Culture of Expresser: Chinese, Dutch) mixed-design analysis of variance (ANOVA), with participants’ intensity ratings being the dependent variable. Culture of Perceiver and Culture of Expresser were between-subjects variables, and Scale Similarity was a within-subjects variable. A complete overview of effects can be found in supplementary Table S1; the results pertaining to our hypotheses are presented below. To rule out possible cultural differences in response styles, we followed Triandis, McCusker, and Hui’s (1990) recommendation to standardize all the ratings across variables and participants within each culture. We then conducted the same analyses on the standardized scores, and the results were identical (see supplementary Table S3 for the results).

In support of Hypothesis 1, there was a main effect of Scale Similarity, F(1, 207) = 565.11, p < .001, h2p = .732, with ratings for morphologically similar emotions (M = 38.78) being higher than those for dissimilar emotions (M = 14.60; difference = 24.18, 95% confidence interval [CI] = [21.64, 26.72]). This suggests that morphologically similar emotions were endorsed to a greater extent than dissimilar emotions.

Consistent with Hypothesis 2, there was a main effect of Culture of Perceiver, F(1, 207) = 88.76, p < .001, h2p = .300, with Chinese participants (M = 38.96) rating the expressions higher on the nonintended emotion scales than did the Dutch participants (M = 19.63; difference = 19.33, 95% CI = [15.61, 23.04]). This shows that the Chinese endorsed nonintended emotions to a greater extent than the Dutch.

We further tested whether the difference between the Chinese and Dutch would be greater for morphologically similar emotions or morphologically dissimilar emotions. The interaction effect between Scale Similarity and Culture of Perceiver was significant, F(1, 207) = 60.96, p < .001, h2p = .227, with the cultural difference being greater for ratings on the morphologically similar emotions (M_Chinese = 57.44, M_Dutch = 28.05; difference = 29.39, 95% CI = [24.34, 34.44]) than for ratings on the dissimilar emotions (M_Chinese = 20.48, M_Dutch = 11.22; difference = 9.26, 95% CI = [5.51, 13.02]; see Figure 2, left panel). This pattern of results indicates that the effect of culture was more pronounced for morphologically similar emotions than for dissimilar emotions.

Figure 2.

Nonintended emotion ratings as a function of similarity of rating scale and culture of perceiver for static facial expressions of emotion (Studies 1 and 2).

Note. (a) In Study 1, Chinese perceivers endorsed nonintended emotions to a greater extent than Dutch perceivers, and this effect was particularly pronounced for morphologically similar emotions. (b) In Study 2, this finding was only replicated for Chinese, but not for Dutch, faces. The error bars represent confidence interval.

Dynamic task

In the dynamic task, the last expression shown on the face was the expression to be evaluated. Therefore, only morphs to anger/disgust (i.e., disgust-to-anger, fear-to-anger, anger-to-disgust, and fear-to-disgust) yield ratings for both morphologically similar (disgust/anger) and dissimilar emotions (fear). Similar to the static task, only morphs to expressions of anger/disgust (96 out of 144 trials per participant) were therefore included in the analysis.

In addition to the factors included in the analyses of the data from the static task, another factor was included in the analysis of dynamic task data, namely, whether the temporal context (i.e., the start expression) was morphologically similar or different to the end expression. A 2 (Scale Similarity: Similar, Dissimilar) × 2 (Context Similarity: Similar, Dissimilar) × 2 (Culture of Perceiver: Chinese, Dutch) × 2 (Culture of Expresser: Chinese, Dutch) mixed-design ANOVA was conducted. Both hypotheses supported in the static task were further supported in the dynamic task (see supplementary Table S2 for all effects based on raw scores and Table S4 for all effects based on standardized scores, the results of which were identical). Specifically, morphologically similar emotions (M = 39.87) were endorsed to a greater extent than morphologically dissimilar emotions (M = 16.90; Hypothesis 1), F(1, 207) = 600.33, p < .001, h2p = .744 (difference = 22.97, 95% CI = [20.53, 25.41]), Chinese (M = 39.10) endorsed nonintended emotions to a greater extent than Dutch (M = 22.23; Hypothesis 2), F(1, 207) = 56.11, p < .001, h2p = .213 (difference = 16.88, 95% CI = [12.98, 20.77]). In addition, this cultural difference was greater for morphologically similar emotions (M_Chinese = 57.30, M_Dutch = 29.85; difference = 27.45, 95% CI = [22.47, 32.42]) than for dissimilar emotions (M_Chinese = 20.91, M_Dutch = 14.60; difference = 6.30, 95% CI = [2.13, 10.48]), F(1, 207) = 81.00, p < .001, h2p = .281.

Furthermore, we tested temporal context effects across cultures. A marginal three-way interaction was found between Scale Similarity, Context Similarity, and Culture of Perceiver, F(1, 207) = 2.98, p = .086, h2p = .014. The three-way interaction was broken down for Chinese and Dutch observers separately. For Chinese observers, the two-way interaction between Scale Similarity and Context Similarity was not significant, F(1, 76) = 0.01, p = .935, h2p < .001. For Dutch observers, however, the interaction between Scale Similarity and Context Similarity was significant, F(1, 133) = 11.10, p = .001, h2p = .077. This two-way interaction was analyzed further, revealing that ratings on the morphologically similar emotion were higher when preceded by a morphologically similar emotion (M = 31.87) compared with a dissimilar emotion (M = 27.83), t(133) = 2.95, p = .004, Cohen’s d = .25 (difference = 4.04, 95% CI = [1.33, 6.75]). Ratings on the morphologically dissimilar emotion, however, were higher when the end emotion was preceded by a morphologically dissimilar emotion (M = 17.43) as compared with a morphologically similar emotion (M = 11.77), t(133) = 3.53, p = .001, Cohen’s d = .30 (difference = 5.66, 95% CI = [2.49, 8.83]; see Figure 3, left panel). This pattern suggests a congruency effect in Dutch observers, with an increase in judgments on the rating scales of the preceding emotions. However, the temporal context did not influence Chinese observers’ ratings.

Figure 3.

Nonintended emotion ratings as a function of similarity of rating scale, similarity of context, and culture of perceiver for dynamic facial expressions of emotion (Studies 1 and 2).

Note. A greater influence of context was found among Dutch perceivers than among Chinese perceivers. (a) In Study 1, context made Dutch perceivers endorse nonintended emotions (congruent with the preceding expressions) to a greater extent for ratings on both morphologically similar and dissimilar emotions, but context did not affect Chinese perceivers’ judgments. (b) In Study 2, context made Dutch perceivers endorse nonintended emotions (congruent with the preceding expressions) to a greater extent for ratings on the morphologically dissimilar emotions but not for ratings on the morphologically similar emotions; context did not affect Chinese perceivers’ judgments. The error bars represent confidence interval.

In summary, both hypotheses were supported in Study 1, with morphologically similar emotions being endorsed to a greater extent than dissimilar emotions (Hypothesis 1), and Chinese participants endorsing nonintended emotions to a greater extent than Dutch participants (Hypothesis 2). Furthermore, the difference between Chinese and Dutch was greater for morphologically similar emotions than for dissimilar emotions. In addition, the present research showed that the temporal context affected Dutch, but not Chinese, observers’ judgments.

In Study 2, we set out to replicate these findings with a different set of emotions to test whether the pattern of results would be robust across distinct emotion constellations.

Study 2

Method

Participants

The study included 210 individuals, nine participants were excluded because they merely clicked through the trials without performing the task. The final sample consisted of 122 Dutch participants (M_age = 21.5, SD = 3.01; 100 females) and 79 Chinese participants (M_age = 19.2, SD = 1.78; 51 females), who were recruited in the same way as in Study 1. Among the Dutch participants, 84 (M_age = 21.7, SD = 2.85; 62 females) were presented with Dutch facial expressions and 38 (M_age = 21.0, SD = 3.48; 32 females) were presented with Chinese facial expressions. Among the Chinese people, 39 (M_age = 19.7, SD = 1.76; 30 females) were presented with Chinese facial expressions and 40 (M_age = 18.6, SD = 1.48; 20 females) were presented with Dutch facial expressions. In return for participation, Dutch participants received one course credit or €10, and Chinese participants received 30 yuan (about €4).

Apparatus and procedure

Besides the use of different emotion categories (anger–fear–surprise) and the accompanying rating scales, the apparatus and procedure were identical to Study 1.

Results and Discussion

Static task

Like in Study 1, the analyses focused on static expressions for which both morphologically similar and dissimilar rating scales were included (i.e., fear and surprise for Study 2). The data were analyzed identically to those of Study 1, with a 2 (Scale Similarity: Similar, Dissimilar) × 2 (Culture of Perceiver: Chinese, Dutch) × 2 (Culture of Expresser: Chinese, Dutch) mixed-design ANOVA. A complete overview of effects based on raw scores can be found in supplementary Table S1 (results based on standardized scores can be found in supplementary Table S3; the pattern of results was identical.). The results pertaining to the hypotheses are presented below.

In support of Hypothesis 1, there was a main effect of Scale Similarity, F(1, 197) = 775.79, p < .001, h2p = .797, with ratings on the morphologically similar emotion scales (M = 47.92) being higher than ratings on the dissimilar emotion scales (M = 11.76; difference = 36.17, 95% CI = [33.71, 38.63]). This is consistent with the finding from Study 1 that similar emotions are endorsed to a greater extent than dissimilar emotions.

Consistent with Hypothesis 2, there was a main effect of Culture of Perceiver, F(1, 197) = 22.71, p < .001, h2p = .103, with Chinese participants (M = 35.21) rating the expressions higher on the nonintended emotion scales than Dutch participants (M = 26.37; difference = 8.84, 95% CI = [5.05, 12.64]). This indicates that Chinese observers endorsed nonintended emotions to a greater extent than Dutch observers.

We further explored whether cultural differences would be greater for morphologically similar emotions or morphologically dissimilar emotions. The anticipated interaction between Scale Similarity and Culture of Perceiver was not significant, F(1, 197) = 1.77, p = .185, h2p = .009. However, a three-way interaction between Scale Similarity, Culture of Perceiver, and Culture of Expresser was found, F(1, 197) = 4.23, p = .041, h2p = .021. This three-way interaction was broken down for Chinese and Dutch expressers separately since we were interested in the interaction between Scale Similarity and Culture of Perceiver. When judging Chinese stimuli, ratings on the morphologically similar emotions were higher than ratings on the dissimilar emotions, F(1, 75) = 341.78, p < .001, h2p = .820, and Chinese observers’ ratings were higher than Dutch observers’ ratings, F(1, 75) = 20.55, p < .001, h2p = .215. Furthermore, these two effects interacted, F(1, 75) = 4.61, p = .035, h2p = .058, such that the effect of Culture of Perceiver was greater for ratings on the morphologically similar emotions (M_Chinese = 58.56, M_Dutch = 41.13; difference = 17.43, 95% CI = [8.76, 26.10]) than for ratings on the dissimilar emotions (M_Chinese = 16.10, M_Dutch = 7.51; difference = 8.59, 95% CI = [3.66, 13.52]; see Figure 2, right panel). When judging Dutch stimuli, however, no significant interaction was found, F(1, 122) = 0.33, p = .567, h2p = .003. Thus the difference between Chinese and Dutch observers was greater for ratings on the morphologically similar emotions than ratings on the dissimilar emotions for Chinese, but not Dutch, stimuli.

Dynamic task

As in Study 1, only end emotions for which both morphologically similar and dissimilar rating scales were included (i.e., anger-to-fear, surprise-to-fear, anger-to-surprise, and fear-to-surprise).

The data were analyzed in the same way as those of Study 1, using a 2 (Scale Similarity: Similar, Dissimilar) × 2 (Context Similarity: Similar, Dissimilar) × 2 (Culture of Perceiver: Chinese, Dutch) × 2 (Culture of Expresser: Chinese, Dutch) mixed-design ANOVA (see supplementary Table S2 for all effects based on raw scores and Table S4 for all effects based on standardized scores). Both hypotheses supported in the static task were further supported in the dynamic task. Specifically, morphologically similar emotions (M = 43.40) were endorsed to a greater extent than dissimilar emotions (M = 13.46; Hypothesis 1), F(1, 197) = 544.32, p < .001, h2p = .734 (difference = 29.94, 95% CI = [27.48, 32.41]), the Chinese (M = 31.84) endorsed nonintended emotions to a greater extent than the Dutch (M = 26.22; Hypothesis 2), F(1, 197) = 9.59, p = .002, h2p = .046 (difference = 5.62, 95% CI = [1.45, 9.79]). Furthermore, this cultural difference was greater for morphologically similar emotions (M_Chinese = 48.93, M_Dutch = 39.82; difference = 9.11, 95% CI = [3.72, 14.51]) than for morphologically dissimilar emotions (M_Chinese = 14.75, M_Dutch = 12.62; difference = 2.13, 95% CI = [–2.15, 6.41]), F(1, 197) = 9.84, p = .002, h2p = .048.

To test whether contextual influence varies depending on the culture of perceiver, we examined the three-way interaction between Scale Similarity, Context Similarity, and Culture of Perceiver, which was significant, F(1, 197) = 6.53, p = .011, h2p = .032. The three-way interaction was broken down for Chinese and Dutch observers separately. As in Study 1, for Chinese observers, the two-way interaction between Scale Similarity and Context Similarity was not significant, F(1, 78) = 2.56, p = .114, h2p = .032, whereas for Dutch observers, the interaction was significant, F(1, 121) = 9.13, p = .003, h2p = .070. Follow-up simple effects analyses showed no difference between morphologically similar and dissimilar contexts for ratings on the morphologically similar emotions, t(121) = 1.53, p = .128, Cohen’s d = .14 (difference = 2.61, 95% CI = [–0.76, 5.99]), but a significant difference was found for ratings on the morphologically dissimilar emotions, t(121) = 4.18, p < .001, Cohen’s d = .38, with ratings on the morphologically dissimilar emotions being higher when preceded by a morphologically dissimilar (M = 16.86) rather than similar (M = 8.38) emotion for Dutch observers (difference = 8.48, 95% CI = [4.46, 12.50]). Thus Study 2, like Study 1, showed that start expressions did not influence Chinese observers’ perception of end expressions, whereas start expressions elicited a congruence effect in Dutch observers. The pattern of the Dutch participants was largely consistent with the results of Study 1, with a congruency effect emerging in ratings on the morphologically dissimilar emotions, that is, ratings on the dissimilar emotions being higher in dissimilar contexts. Unlike in Study 1, however, no congruence effect was found for ratings on the morphologically similar emotions in Dutch participants in Study 2.

In sum, three main findings from Study 1—morphologically similar emotions being endorsed to a greater extent than dissimilar emotions, Chinese endorsing nonintended emotions to a greater extent than Dutch, and the start expressions influencing Dutch, but not Chinese, observers—were replicated in Study 2. In addition, the finding from Study 1 that the cultural differences were greater for similar as compared with dissimilar emotions was partly replicated in Study 2, with this pattern found for the perception of Chinese, but not Dutch, faces.

General Discussion

Although a considerable body of research has studied the recognition of intended emotions, we know little about the perception of nonintended emotions from facial expressions. The present research investigated the perception of multiple concurrent emotions from both static and dynamic expressions in Chinese and Dutch observers. Four main findings emerged consistently across the two studies (we also conducted a total analysis on the two studies together and obtained the same results; see supplementary Tables S1, S2, S3, and S4). First, morphologically similar emotions were endorsed to a greater extent than dissimilar emotions. For example, facial expressions of anger were rated higher on perceived disgust (which is morphologically similar to anger) than on perceived fear (which is morphologically dissimilar to anger). Second, Chinese observers endorsed nonintended emotions to a greater extent than Dutch observers. When observing a facial expression of anger, for instance, Chinese participants reported perceiving disgust and fear to a greater degree than did Dutch participants. Third, the difference between Chinese and Dutch observers was more pronounced for morphologically similar emotions than for dissimilar emotions (e.g., for expressions of anger, cultural differences were greater for judgments of disgust compared with judgments of fear). Fourth, the start emotions had a greater influence on Dutch than on Chinese observers. Specifically, Dutch observers endorsed nonintended emotions that were congruent with the preceding expressions to a greater degree. The consistency of these findings in Studies 1 and 2 across two different emotion constellations suggests that these results are robust and persist across distinct emotional expressions.

Unlike previous emotion perception studies using multiscalar intensity ratings of emotional expressions (Ekman et al., 1987; Hall & Matsumoto, 2004; Yrizarry et al., 1998), the current research focused on the perception of nonintended emotions. In using this paradigm, we found that Chinese endorsed nonintended emotions to a greater extent than Dutch and that this cultural difference was moderated by the morphological similarity between emotional expressions. Below we consider some theoretical implications of the present findings, discuss the strengths and limitations of our approach, and outline directions for future research.

Theoretical Implications

Previous research has found that anger and disgust, and fear and surprise, share some morphological features, which make them relatively more difficult for perceivers to distinguish from both static and dynamic emotional expressions (Jack et al., 2009; Sacharin et al., 2012). However, participants in previous research could only specify one emotion category (or a “none of the above” option) when asked to indicate which emotion they perceived from a given expression. Instead of using this categorical classification, the current paradigm allowed participants to indicate that they had seen several concurrent emotions (or no emotion) in an expression. In Study 1, for example, disgust was endorsed to a greater extent than fear for facial expressions of anger, and in Study 2, surprise was endorsed to a greater extent than anger for facial expressions of fear. This implies that observers can perceive several concurrent emotions in a given expression and that the perceived intensity of each emotion depends on its morphological similarity to the intended emotion.

The differential emotion perception we observed here appears to be driven mainly by the morphological (dis)similarities between emotional expressions, rather than by (dis)similarities between emotions in terms of valence, arousal, or approach/avoidance tendencies. Facial expressions of anger, disgust, and fear are typically perceived as high in arousal and negative in valence (see Russell, 1980). Yet anger was endorsed to a larger extent than fear from a disgusted face, even though arousal and valence ratings of disgust and fear expressions were more similar than those of disgust and anger expressions (see supplementary Table S7 for valence and intensity ratings). In terms of approach/avoidance tendencies, anger is typically thought to be approach-related, while disgust and fear are thought to be avoidance-related (Frijda et al., 1989). Thus, similarities in terms of approach/avoidance cannot account for the finding that anger was endorsed to a larger extent than fear from a disgusted facial expression.

The present finding, that Chinese participants endorsed nonintended emotions to a greater extent than the Dutch participants did, is consistent with previous findings showing that Easterners make a less clear distinction than Westerners between anger and disgust and between fear and surprise (Jack et al., 2009; Jack et al., 2012). Previous accounts have suggested that this may be due to Easterners and Westerners using different perceptual strategies when viewing facial expressions: Easterners’ perception emphasizes the eye region, whereas Westerners attend more to the mouth area. Facial expressions of anger and disgust (and those of fear and surprise) show few differences around the eye region. This explanation, however, cannot account for the cultural differences in the ratings of morphologically dissimilar emotions found in the current study. For example, despite expressions of anger and fear differing greatly in the eye region, Chinese still perceived a greater degree of fear from facial expressions of anger relative to Dutch observers. Thus, cultural differences in the endorsement of multiple emotions cannot be fully accounted for by attentional biases. It is worth noting that participants from both cultures reliably perceived the intended emotions conveyed by the facial expressions studied here, as evidenced by that fact that, in every condition, ratings for the intended emotion were significantly higher than ratings for all nonintended emotions (Fang, Van Kleef, & Sauter, 2017).⁶ This pattern of results suggests that Easterners’ stronger inclination of perceiving multiple emotions from a face is not due to them failing to perceive the intended emotion.

Previous research has also shown that Easterners experience more complex blends of emotions than Westerners, which is in line with our finding that Chinese observers perceived multiple emotions to a greater degree than did Dutch observers. Studies in which participants were asked to report their feelings in different situations have revealed positive correlations between positive and negative emotions in Asian cultures but negative correlations in Western cultures; that is, mixed emotions co-occurring in Easterners but not in Westerners (Grossmann et al., 2015; Miyamoto et al., 2010). The present finding extends this research from an intrapersonal level (perceiving self-emotional experience) to an interpersonal level (perceiving other’s emotions). Notably, the present research is the first to establish that Easterners are able to perceive mixed emotions consisting of multiple distinct negative emotions, besides blends of positive and negative emotions.

The finding that cultural differences were more pronounced for morphologically similar emotions than for dissimilar emotions suggests that the cultural differences in emotional complexity discussed above were primarily driven by the perception of morphologically similar emotions. For example, both Chinese and Dutch rated expressions of anger low on the fear scale, with fear being morphologically dissimilar to anger. In contrast, judgments of disgust, which share a stronger morphological similarity with anger, entail larger overlapping elements to the target. Accordingly, Chinese observers, who tend to perceive close elements as more overlapping (interdependent/holistic), rated anger expressions higher on the disgust scale than did Dutch observers, who are more independent and analytical. Here, the effects of culture and morphological similarity interacted to determine the extent to which participants perceived several emotions in a facial expression, with greater cultural differences in the endorsement of morphologically similar rather than dissimilar emotions.

In contrast to documented East–West differences in concurrent context effects (e.g., Masuda et al., 2008; Stanley et al., 2013), we consistently found that the temporal context influenced Dutch observers’ but not Chinese observers’ judgments. Specifically, we found a congruency effect with preceding expressions in Dutch observers. This temporal context effect on Dutch observers’ perception of nonintended emotion is, to some extent, consistent with previous findings that concurrent contexts (gender and job status) influence the perception of nonintended emotions from facial expressions in Western perceivers (Algoe, Buswell, & DeLamater, 2000). In contrast, Chinese observers’ judgments of nonintended emotions did not differ across preceding expressions. Two explanations may account for this phenomenon. First, relative to Dutch observers, Chinese observers already endorsed a greater degree of nonintended emotions from static expressions, and context may therefore not have caused a substantial increase in ratings of nonintended emotions. However, this explanation is undermined by the fact that Chinese observers’ ratings on the nonintended scales of static expressions were far from ceiling in the current studies (see Figures 2 and 3).

Instead, we propose a second possibility, namely, that cultural differences vary across different types of contexts. The cultural differences found in previous research on concurrent context effects (i.e., that Easterners are more strongly influenced by context than Westerners) do not generalize to cultural differences in temporal context effects. A recent study has suggested that the agency of the contextual information may have differing degrees of influence on how faces are evaluated across cultures (Ito, Masuda, & Li, 2013). Ito and colleagues found that when other figures’ emotional expressions served as context, Westerners’ judgments of the target person were not influenced by surrounding people’s expressions. In contrast, when there was no agency information in the context (e.g., affectively salient landscape scenery), Westerners were influenced by the context just like Easterners. They argued that this difference might be due to Western perceivers considering individuals as agents of their own emotions, while Easterners construe emotions more as products of an individual’s social context. In the present study, the context was the expresser’s own preceding expression, reflecting the history of the expresser’s own feelings rather than those of others. This may have resulted in the start expression having a more marked influence on Dutch observers’ judgments.

Notably, Chinese participants endorsed nonintended emotions to a greater extent than did Dutch participants, both in the static and dynamic tasks. This suggests that Easterners’ higher endorsement of nonintended emotions was not driven by temporal context, given that there was no preceding expression in the static task. This pattern of results implies that we should be cautious in concluding that Easterners are overall more strongly influenced by context than Westerners. Cultural differences may vary across different types of contextual information (e.g., concurrent vs. temporal context) and perhaps also as a function of the relationship between context and expression.

Limitations and Future Directions

One limitation of the current paradigm is that the change from one expression to another does not necessarily reflect the natural emotional change in emotional facial expressions that we come across in real life. For example, with natural emotional change from anger to disgust, the mouth area may change ahead of the eyes, which would not be captured in computer-generated morphs in which changes in the upper and lower face happen simultaneously. Despite this criticism, morphs have clear merits, such as the easily adjustable rate of change and the lack of noise, and they are commonly used in dynamic emotion research (e.g., Sacharin et al., 2012; Sato & Yoshikawa, 2010). Nevertheless, this methodology could be improved in future research using videos of changing emotional expressions as they happen in real time. Although databases with naturalistic dynamic emotional facial expressions exist for changes to and from neutral expressions (e.g., Van der Schalk, Hawk, Fischer, & Doosje, 2011), comparable stimulus materials for expression changes between emotions have not yet been developed. Even greater ecological validity would be gained by examining emotional expressions occurring in ongoing interactions. But it is worth noting that establishing the intended emotion in a genuine expression and ensuring the absence of additional emotions in the expresser are a challenge. These research approaches complement each other, and future studies should establish whether the current results hold for genuine dynamic expressions.

Another limitation of the current research is that only emotions of negative or neutral valence were included in this study. Positive emotions, such as happiness, may be particularly interesting to examine cross-culturally. In Eastern cultures, smiles do not necessarily function as a positive signal, as smiling is deemed an appropriate reaction in social interaction even when feeling angry with the interaction partner (Sun, 2010). Westerners are comparatively more encouraged to express what they are feeling, and smiling is consequently more often regarded as a positive social signal (Matsumoto, 1991). Future work could evaluate whether the current pattern of results (e.g., Easterners perceiving emotions as more mixed than Westerners) extends to expressions of positive emotions.

Other potential limitations include the number and type of emotion scales used as response categories; we do not know whether the conclusion that Easterners perceive nonintended emotions to a larger extent than Westerners also holds for other emotions besides the emotion scales used in the present studies. It should also be noted that the facial expressions were Western-based prototypes and we studied only two cultures. As a result, the current findings do not necessarily generalize to other facial expressions and other populations. Regardless of these limitations, however, the present results challenge previous work in this area and raise important questions about the influence of culture and morphological similarity on the judgments of emotional expressions.

Finally, although there is a wealth of research showing that Easterners are more interdependent, holistic, and dialectical than Westerners (e.g., Nisbett et al., 2001; Peng et al., 2006), we did not include these measures in the current research. Future research may benefit from measuring interdependence/independence, holistic/analytic cognition, and dialectical thinking to establish the mechanism(s) underlying the cultural differences in emotion perception. It would also be interesting to investigate whether Easterners also produce more mixed expressions of emotion than Westerners. This would complement previous findings showing that Easterners are more likely to experience mixed feelings (Grossmann et al., 2015; Miyamoto et al., 2010) as well as the current finding that this mixedness extends to perceptual judgments.

Conclusion

Two studies provided consistent evidence that morphologically similar emotions were endorsed to a greater extent than dissimilar emotions, that Chinese observers endorsed nonintended emotions to a greater extent than Dutch observers, and that this cultural difference was more pronounced for morphologically similar emotions than for dissimilar emotions in both static and dynamic expressions. We also found that temporal contextual cues led Dutch observers to endorse nonintended emotions that were congruent with the preceding expressions to a greater extent but that context did not affect Chinese observers’ judgments.

Research on emotion perception tends to focus on the perception of intended emotions, often ignoring the degree to which other emotions are perceived. The present research underscores the importance of investigating the perception of nonintended emotions from facial expressions, particularly in cross-cultural studies. The increasingly popular idea that emotional expressions serve to facilitate social interaction by providing information to observers about an expresser’s inner feelings and future intentions hinges on the assumption that emotional expression and emotion perception align (for a comprehensive discussion, see Van Kleef, 2016). The current data indicate that such alignment varies as a function of characteristics of the expression (morphological similarity to other emotions) as well characteristics of the perceiver (cultural background).