Formalizing emotion concepts within a Bayesian model of theory of mind

Abstract

Sensitivity to others’ emotions is foundational for many aspects of human life, yet computational models do not currently approach the sensitivity and specificity of human emotion knowledge. Perception of isolated physical expressions largely supplies ambiguous, low-dimensional, and noisy information about others' emotional states. By contrast, observers attribute specific granular emotions to another person based on inferences of how she interprets (or “appraises”) external events in relation to her other mental states (goals, beliefs, moral values, costs). These attributions share neural mechanisms with other reasoning about minds. Situating emotion concepts in a formal model of people's intuitive theories about other minds is necessary to effectively capture humans' fine-grained emotion understanding.

Introduction

“I’d rather write an encyclopedia about common emotions,” he admitted. “From A for ‘Anxiety about picking up hitchhikers’ to E for ‘Early risers’ smugness’ through to Z for ‘Zealous toe concealment, or the fear that the sight of your feet might destroy someone’s love for you.’ ” — The little Paris bookshop by Nina George.

If your friend is experiencing early risers’ smugness, how would you know? From a quick glance at her face and posture, you see she is experiencing a low-arousal positive emotion. To refine this attribution, though, you would need knowledge of the context and cause of the emotion. She is more likely to feel smug, you know intuitively, if she chose to wake up early (rather than being woken involuntarily by a screaming baby) and if she used those extra hours to her relative advantage (rather than wasting them counting sheep). As this example illustrates, human observers can recognize and reason about highly-differentiated, or fine-grained, emotions. Here we propose that fine-grained emotion concepts are best captured in a Bayesian hierarchical generative model of the intuitive theory of other minds [1*].

The role of concepts in emotion has been much disputed [2–5]. This question is particularly hard for first person emotions: when I myself feel anxious, what is the role of my concept of “anxiety” in the construction of my experience? Here, we selectively tackle an easier problem: the problem of other minds. We recognize anxiety in our friends, distinguish their anxiety from their disappointment or regret, and try to respond in appropriate ways [6]; but how do we make such specific and accurate emotion attributions to another person? In order to formally address that question, we situate emotion concepts in a computational model of the intuitive theory of mind [1*,7]. (Note that intuitive or lay theories are causally structured, but generally not explicit, declarative, or introspectively accessible [8]).

Situating emotion concepts within an intuitive theory of mind

Initial scientific descriptions of an “intuitive theory of mind” focused on its application to predicting others’ intentional actions [9]. Minimally, intentional actions can be predicted (and explained) as consequences of the agent’s beliefs and desires, and modeled as inverse planning [10]. Subsequent models have considerably extended this basic premise to capture causal relations between other kinds of mental states. For example: Greg’s choices additionally depend on (what he believes about) the costs of his actions [11]; his beliefs update in response to new evidence [7]; his actions are influenced by his habits [12]; and so on. A hierarchical Bayesian model of this intuitive causal theory can explain both observers’ forward inferences (predicting Greg’s actions given his beliefs and desires) and inverse inferences (inferring Greg’s beliefs and desires given his actions) [10].

People readily incorporate emotions in their intuitive reasoning about other minds [13–15] but only recently have computational models of theory of mind been elaborated to include emotion concepts. Minimally, in the intuitive theory, emotions (or emotional reactions) are caused by how the person interprets (or “appraises”) external events in relation to his other mental states (goals, beliefs, moral values, costs, traits, etc; Figure 1). For example, Greg’s emotional reactions will depend on whether (according to Greg) external events fulfill his goals, contradict his beliefs, reduce the constraints or costs of his preferred actions, violate his values, and so on. As with intentional actions, the same intuitive theory also supports inverse inferences. In the intuitive theory, emotions (which are internal mental states) cause emotional expressions (which are externally observable behaviors), so observers can use perceived emotional behaviors to infer underlying emotions (i.e. perform an inverse inference from observed effects to unobserved cause). Situating emotion concepts within the intuitive theory of mind in this way may seem obvious, but has many ramifications, some of which we explore in the remainder of this article.

Figure 1.

A box and arrow simplification of part of the intuitive causal theory of other minds. Ovals denote unobservable, internal states; rectangles are externally observable. Constraints include appraisals of the costs of actions, what is possible, what is controllable, and other beliefs. Goals and values include both local goals and intentions, but also long term values like relationships and status, and therefore can directly influence expressions. At the core of the model is the inferred “appraisal” process: interpreting external events through the lens of their relevance for one’s goals, beliefs, costs, and so on. Inferred appraisals cause emotions (internal states) which cause expressions (observable behaviors). An observer can therefore predict emotions based on inferred appraisals (following the causal arrows) or from the observed expressions (inverse of the causal arrows). Compare similar models in [14,18**,19].

Specificity and development of emotion inference

First, this approach offers a natural, systematic way to formalize highly-differentiated predictions of others’ emotions, and the links between those predictions and the rest of our sophisticated reasoning about other minds. Although no existing model has yet fulfilled this promise, parts of the intuitive theory of mind have already been well-described in Bayesian generative causal models [16*,17]. Capitalizing on this progress, the same formalizations can be used to model (some) human emotion predictions. For example, in a simple lottery context, two parameters of the target’s appraisal could be inferred directly from a description of the event — his overall reward, and his prediction error — and combined to capture in quantitative detail the emotions that observers predicted [18**]. Relatedly, Wu and colleagues showed participants simple moral scenarios, in which Grace puts white powder in another girl’s coffee [19]. The powder turns out to be poison, and the girl dies. Participants use Grace’s smiling facial expression to infer both that Grace knew the powder was poison, and that she wanted the girl to die. These inferences could be precisely described as inverse inferences in the participants’ intuitive theory of mind. (In the real world, observers make similarly momentous inverse inferences based on emotional reactions [20,21].)

Even children’s earliest understanding of others’ emotions implies (simple) inferred appraisals. Based on an agent’s observed motion path (and a principle of rational action), preverbal infants can infer the agent’s goal (e.g. to get over the wall); then, relative to that goal, infants can distinguish between outcomes that the agent would appraise as goal-consistent or not [22]. Critically, by 10 months old, infants also appear to predict a relevant emotion (or affective state) that causes subsequent expressions (laughing or crying) and are surprised if the agent whose goal was fulfilled then shows a negative-valence behavior, crying [23**] (see also [24]). During development, children’s intuitive theory of mind becomes more sophisticated, and their third-person emotion attributions follow suit [15,25–30]. (Note that while some developmental psychologists reserve the term “theory of mind” for a meta-representational understanding of beliefs, e.g. [30], here the Bayesian model of theory of mind is a generative causal theory, encompassing goals and actions as well as beliefs, costs, and values [10,16*].)

The long-term goal, however, is not just to capture one or two components of observers’ emotion knowledge; rather, it is to develop a formal model that captures all of the same inferred appraisals as human observers do. Promising for this line of work, when given human labels for the target’s appraisals, computational models can already capture a relatively wide and differentiated range of human emotion predictions. Two recent studies provide converging evidence. Using human ratings for 25 appraisal features, a model correctly chose an emotion label (out of 14) for 51% of 6000 real-life events; only 10% of the model’s choices were judged “wrong” by human observers [31]. Similarly, using human ratings for 38 inferred appraisals, a simple model correctly chose the emotion label (out of 20) for 57% of 200 short stories; human accuracy on the same test was 63% [32**]. These models do not yet capture the link from the event to the target’s values, goals, beliefs, and costs, and thus to inferred appraisals. Still, the models’ success suggests that once these links are included, the intuitive theory of mind will capture a substantial portion of shared human knowledge about emotions.

Ambiguous perception and precise predictions

Second, our proposal offers novel insight into predictions based on combinations of inferred appraisals (forward inference) and perceived emotional expressions (inverse inference). People intuit that faces contain the most revealing information about others’ emotions [33,34]. Perhaps surprisingly, mounting scientific evidence shows that human emotion attribution from faces is actually uncertain, noisy, and low-dimensional [35–37]. Many different emotions can be attributed to the same facial configuration [38*–41]; and the space of emotions perceived in faces can be captured in just a handful of dimensions [42,43*]. Even the valence of the event (goal-congruent or not) is not reliably perceived in high-intensity faces: the exact same facial configuration can be attributed to extreme joy (the unexpected return of a child from military service), extreme distress (witnessing a terrorist attack), extreme pleasure (orgasm), or extreme pain (unanesthetized nipple piercing) with equal plausibility [33]. To disambiguate these emotions, observers rely on body posture (open arms, lifted chest [44]) or inferred appraisals of the event (“he won the race” [45]).

Although both body posture and event information are known to disambiguate emotion recognition [33,37,45,46], our model makes a novel distinction between inverse inferences (from bodies) and forward inferences (from event-appraisals). On one hand, observers intuitively infer a common cause (an underlying emotion) of observable face, body and vocal cues. Thus, integrating facial and body configuration, as well as vocal tone, can improve the reliability and specificity of inverse inferences [44,47,48]. Postural information is less ambiguous than facial configuration when perceived at high intensity, from a distance, etc. [44]; similarly, vocal bursts are more informative for distinguishing among positive emotions [49]. As a result, depending on the context, the modality with the most reliable information will appear to dominate emotion attributions [18**,46]; when one cue is ambiguous, cues from other modalities can “sharpen” the inferred cause by shifting attributions among similar, or nearby, emotions [37]. On the other hand, event information is intuitively relevant to the cause of the emotion, rather than its consequences. Additional event information can make emotion attributions more reliable not only by continuously shifting among similar emotions, but also by selecting among separated possibilities [50], because partial event knowledge can generate predictions of distinct (dissimilar, non-overlapping) alternative emotions (e.g. how will he feel after he asks his crush on a first date?). This difference between forward and inverse inferences has been obscured in prior research that confused postural and event-context cues: for example, a photograph of nipple piercing [33] contains mainly event information supporting inferred appraisals, not an emotional posture.

Relatedly, we can distinguish between two ways that “dynamic” facial expressions contain more information than static ones [51]. On the one hand, dynamic change can more precisely differentiate expressive from structural facial features (e.g. a person with dark brows from a person making an angry expression) [52–55]. Dynamic change can also provide more clarity on mixed expressions, by separating the mixture in time [56]. In these ways, dynamic expressions may lead to more specific or more confident inverse inferences (though observers can also be surprisingly insensitive to dynamic information per se [36,57]). On the other hand, when temporal change in the face coincides with temporal change in the external event structure, dynamics support forward inference by highlighting the emotionally-relevant aspect of an event [58]. For example, observers are generally quite insensitive to elements of surprise (“wide-eyed”) in mixed expressions [19,42]. When a change of expression is temporally coincident with an event outcome, though, observers accurately infer that the information was unexpected and change their inferred appraisals accordingly [19]. The temporal sequence of emotions can further constrain inferred appraisals; if people intuit that cognitive processes occur at different speeds then the order of expressions can indicate which hidden mental variable is associated with which emotion.

Third, we propose that there is a key asymmetry between forward and inverse inferences of emotion. The forward inference depends on inferred appraisals which are highly differentiated and granular. However, people’s intuitive theory of mind is also biased and based on simplifying heuristics, inducing systematic errors [59]. We assume people share our desires, values, norms [60]. We underestimate people’s ability to cope, recover, and rebound from significant events [61,62]. These biases in the intuitive theory of mind translate into systematic errors in predictions of emotions. By contrast, inverse inference from emotional expressions is uncertain and low-dimensional, but also relatively accurate and unbiased. Combining both sources is therefore uniquely powerful: forward inferences from inferred appraisals can suggest highly specific, granular, differentiated predictions of another person’s emotions; perception of that person’s expressions can confirm or contradict these predictions, allowing for rapid correction within a reduced possibility space.

Neural representations of fine-grained emotion concepts

Finally, situating emotion concepts within the intuitive theory of mind fits well with recent neuroscientific evidence. Highly-differentiated representations of others’ emotions are almost exclusively found in brain regions associated with theory of mind, especially in temporo-parietal and medial frontal cortex [32**,63,64] (Figure 2). These representations are abstract and amodal, generalizing across emotions inferred from stories, events, and expressions [32**,65]. By contrast, perception of emotional expressions, and even integration of those expressions across modalities, depends on distinct brain regions, especially the superior temporal sulcus [66–70]. These two processes are dissociable in individual differences [71–73], and in neurodegenerative disorders [74]. Taken together, these lines of evidence strongly support the link between emotion concepts and the rest of an observer’s intuitive theory of mind.

Figure 2.

(A) Fine-grained discrimination of others’ emotions (20-way classification) from short verbal narratives (in red) depends on the same brain regions that are involved in intuitive reasoning about other minds (green), especially temporo-parietal junction and medial prefrontal cortex. (B) The behavioral confusion matrix for human participants: bright colors show the probability of a human choosing each label, averaged over 10 stories in each of 20 categories. (C) The confusion matrix based on patterns of response in brain regions associated with theory of mind. Adapted with permission from [32**].

Conclusion

Two lines of scientific research have made substantial progress in parallel, and now stand to make even more progress in concert. On the one hand, formal computational models have begun to capture the core of people’s intuitive theory of mind. These models can accurately model inferences over continuous quantitative variables, within abstract hierarchical structures. As of yet, however, these models have made limited progress in the domain of emotion understanding. On the other hand, the conceptual act theory of emotion attribution identifies the powerful influence of emotion concepts on emotion attribution (though emotion concepts are usually operationalized as words, or labels) [75,76]. Appraisal theory describes some of the content of shared knowledge about emotional events (though as a hand-picked and manually-coded list, rather than a generative causal model) [31]. Using the intuitive theory of mind as a framework to formalize observers’ inferences about a target’s appraisals offers a powerful tool to capture, and even recreate in a computer [77], our detailed knowledge of how others feel.

Highlights.

• Humans recognize and reason about specific fine-grained emotions in others.

• Generative causal models of people's intuitive theory of mind can formalize human emotion attributions.

• Observed emotional expressions yield low-dimensional, noisy and uncertain emotion attributions.

• Forward predictions of others' emotions based on event structure are precise but subject to systemic biases.

• These complimentary inferences can synergize to yield rich, granular understandings of others' minds.