Mixed emotions in the predictive brain

Abstract

Understanding complex or mixed emotions first requires an exploration of the human nervous system underlying emotions, and indeed all experience. We review current research in neuroscience, which describes the brain as a predictive, internal model of the world that flexibly combines features from past experience to construct emotions. We argue that “mixed emotions” result when these features of past experience correspond to multiple emotion categories. Integrating event perception and cognitive linguistic theories, we propose that “mixed emotions” are perceived as an episode of distinct, linked emotional events due to attentional shifts which update the predicted model of experience. These proposed mechanisms have profound implications for the study of emotion; we conclude by suggesting methodological improvements for future research.

Emotions can feel pure and simple, multi-faceted, or confusing and muddy. To understand differences in emotional complexity, or what is often termed “mixed emotions”, we begin with the structure and function of the human nervous system and ask “what emotions is a nervous system like this capable of constructing?” Using this approach, we find a brain that does not trigger or retrieve multiple elemental emotions in parallel, but rather generates singular, high-dimensional representations comprised of mental features from past, heterogeneous experiences of emotion. These representations are constructed in anticipation of their use, and then dynamically updated by sensory input from the body and the world, unfolding over time as a series of emotional events that can be viewed across multiple times scales and perspectives. In other words: emotions are not your reactions to the world; they are your actively constructed experiences of it.

The human brain is a predictive, internal model of the world

The last two decades of neuroscience research are converging on the hypothesis that all mental events, emotional and otherwise, are generated as predictions, not reactions [1**,2**,3,4]. Your brain runs an internal model [5,6] of the world from the perspective of someone who has your body; it is theorized that this internal model functions as a Bayesian filter [7] for incoming sensory input, driving actions and experience, including emotions [1**,8,9**]. Prediction optimizes energy efficiency by anticipating your body’s needs in a situated fashion and attempting to meet those needs before they arise; this process is called allostasis [10,11*].

We begin with hypothesis that your brain’s internal model consists of patterns of previous experience that are reconstructed so that they best match the situation at hand [5,12]. These patterns function as expectations of what sensations are most likely to occur next, and what actions will be most beneficial in the current context, given past experience. Incoming sensory input that is divergent from, or unanticipated by, these predictions is encoded as prediction error. Neuroscientists debate the mechanisms by which neurons compute prediction error, but most agree that it is information to be learned to update the internal model [for a discussion, see 9**]. Your running internal model (the predictions issued a moment ago), plus prediction errors, together construct your experience (Figure 1). Your brain’s predictions are context-sensitive and continuous, and errors of prediction are encoded as needed to arrive at “progressively better guesses” [13**] about the state of the body in the world (i.e., a more accurate internal model). The result is our lived experience, realized as our perceptions and actions, as the brain makes meaning by constructing predictions and resolving prediction error.

Fig. 1.

Schematic depiction of dynamic brain states over time. At time t, a cascade of predicted experience (light blue) is initiated in the limbic cortices of the allostatic/interoceptive network [14]; incoming sensory input (in this simplified example, from primary visual cortex) is anticipated, so there is no prediction error. At t+1, predictions in light blue are compared against input (prediction error, darker blue); differences may be incorporated into updated predictions (darker blue, t+2). Significant changes in incoming sensory input (red, t+2) may result in further tuning of predictions (purple, t+3), and the initiation of an entirely new cascade of predictions (red, t+4). Incoming sensory input that is unreliable or uncertain (green checkers, t+4) can be ignored rather than incorporated into updated predictions (t+5). (For a detailed depiction of the neural basis of prediction, see Figure 5, Barrett, 2017 [9**].)

Internal models run on similarity

Fundamentally, we suggest that emotions are states that link allostasis (and its interoceptive consequences) to events in the world in a meaningful way. To identify what caused incoming sensory inputs and determine how to best deal with them, your brain is performing similarity computations, asking: “what is new sensory input most similar to in my past?” [8,9**,15]. This is the way that your brain creates the best match for current situation with some probability (referred to in Bayesian terms as the prior probability, or prior). Prediction signals (each of which has some prior probability) are thought to anticipate sensory inputs and motor actions, changing the firing of sensory and motor neurons in anticipation of sensory inputs [16,17]. When they arrive in the brain, incoming sensory inputs from the body and the world constrain these priors to create representations with strong posterior probabilities (i.e., probabilities that the predictions are true given the relevant observations, including adjustments for prediction error). These posterior probabilities then serve as beliefs about the causes of the sensory inputs and how to best act on them [18].

Predictions are concepts that categorize sensations, guide action, and make meaning

Predictive coding provides the specific hypotheses that fill in the computational and neural gaps in the authors previous theoretical account of emotion [the conceptual act theory; 19]. The past experiences that are available within your brain as it implements its internal model can be thought of as a conceptual system [8,9**]. Predictions can be described as ad hoc concepts [8,9**]: groups of representations that are similar for some purpose [20], instantiated as dynamic patterns of information in response to cues in context [21,22**]. We hypothesize that your brain constructs arrays of competing predictions, each with some prior, and these potential futures are compared to the incoming input, which either completes the pattern to which it is most similar, or modifies the prediction. Completion is not necessarily based on similarity of sensory features, but on functional similarities that relate to a goal. This is the same as saying that predictions, as concepts, are constructed in the service of accomplishing context-specific goals [23,24; for a discussion applied to emotion, see 8]. Once a prediction is completed, sensations have been processed and categorized. Categorization is how the brain makes meaning of the sensory world (which includes sensory inputs from the body): sensations are constructed, causal explanations are inferred, and functional action plans are initiated. When incoming sensations are categorized using past experiences of emotions (i.e., when the brain constructs emotion concepts in the service of allostasis, thereby guiding action and constructing perception), the result is an instance of emotion.

Generativity allows the brain to perform conceptual combination when constructing emotions

Conceptual knowledge in the moment (i.e., a population of predictions) is created with preexisting accumulations of experience, whether directly lived or indirectly acquired from the stories of others (including TV shows, movies, and books). The brain’s ability to create new interpretations out of bits and pieces of the past is referred to as generativity. Generativity is what allows the brain to improvise and imagine. Generativity ensures that the brain is able to form novel concepts as needed; in the cognitive psychology literature, this is called conceptual combination [25]. Conceptual combination holds the key to understanding the neural implementation of “mixed emotions”: a population of emotion predictions (i.e., an emotion concept) can be constructed using conceptual combination of past experiences of more than one emotion category. Even a single prediction can be constructed via conceptual combination from past experiences belonging to different emotion categories.

The brain inherently constructs instances that belong to multiple emotion categories

To optimally predict and deal with incoming sensations, the brain will sample from a diverse set of past experiences, or previous categorizations, each of which likely shares similar features with current circumstances (e.g., a past ‘fear’ experience with the same goal, a past ‘sadness’ experience with a similar cause, a past ‘anger’ experience with similar interoceptive sensations, etc.). When the brain constructs a population of predictions dynamically, each individual prediction can be formed by combining these past experiences. In a toy example (Figure 2): a single prediction (A) may be 40% probabilistically similar to past ‘fear’ experience(s), 30% to past ‘sadness’ experience(s), and 30% to past ‘anger’ experience(s); another (B) may be 90% similar to past ‘fear’ and only 10% similar to past ‘sadness’. The prediction (A or B) with the highest posterior probability (i.e., the one that “wins”) will likely be the one with the strongest prior – but not necessarily (e.g., if the situation changes dramatically and unexpectedly).¹

Fig. 2.

Simplified array of predictions of emotional experience, created through conceptual combination. Past experience(s) of the given emotion categories are represented by colored circles; predicted experienced (conceptual combinations) are indicated by the translucent circle. Prediction A shares probabilistic similarity with features of all three categories, whereas prediction B has most features in common with past ‘fear’ experience(s), with only a few features shared with past ‘sadness’ experience(s).

Figure 2 is an abstract depiction meant to illustrate that emotion categories are overlapping and probabilistic, rather than fixed and discrete. It is based on the hypothesis that the brain creates predictions that are generative combinations of prior experiences constructed as specific emotions (i.e., past instances of sensations distinctly categorized as ‘anger’, ‘fear’, ‘sadness’, etc. establish the priors for the current array of predictions). The ‘mixedness’ of an emerging emotion is, in part, determined by a person’s ability to use emotion category knowledge with precision [i.e., his or her level of emotional granularity; 8]. What looks like a unitary instance of a single emotion category can still be constructed as a mixed emotion, to the extent that instances of other emotion categories also contribute to the priors. This hypothesis is consistent with observations that emotion categories are populated by highly variable, situated instances [8,9**,26]. For example, the category ‘fear’ includes instances of physical fear (e.g., being attacked by a dangerous animal), social fear (e.g., public speaking), and even pleasant fear (e.g., riding a rollercoaster). A given instance of fear contains some set of mental and physical features that can vary considerably from other instances both past and present: you can tremble in fear, jump in fear, freeze in fear, scream in fear, gasp in fear, hide in fear, attack in fear, and even laugh in the face of fear [27,28]; your heart rate or skin conductance can increase in fear, decrease in fear, or stay the same in fear [29]. Some of these mental and physical features occur in instances of other emotion categories, as well [i.e., emotion categories exhibit high cross-category similarity in addition to high within-category variability; 30].

The experience of multiple emotions “at the same time” is also due to the various timescales at which predictions, actions, and experiences occur

Consider the fact that predictions cascade across the brain to visceromotor, motor and primary sensory regions, controlling action and creating the complex sensory array of experience. Prediction errors are processed in as feedforward information flow [31] that takes, on average, about 100 ms from primary visual cortex to the limbic regions that initiate predictions [32], and a physical movement takes about 700 ms to organize and execute (yet psychology experiments still consider a meaningful reaction time to be 300 ms). By comparing these various timescales, it becomes clear that by the time you can report the experience of emotion, multiple iterations of prediction and prediction error have already occurred. Yet instead of having access to the iterative tuning process that the brain is continually performing, you only have access to its products, experiencing emotions as distinct phenomena that can occur “at the same time”.

We contend that the perception of distinct emotions is the result of event segmentation, as the brain dynamically constructs and updates predictions according to which contextual cues are most salient [33]. New sensory input continually arrives in the brain, and your brain predicts which will be most relevant for allostasis based on past experience [called precision estimates: 3,34–36]. Effectively, these precision estimates regulate attention, adjusting the internal model to the conditions of the sensory periphery. If incoming sensory input is sufficiently different (i.e., prediction error is sufficiently great), then the brain may construct a new population of predictions, using a different sample of past experiences. In event perception theory, shifts in the internal model are also the result of encoding of large amounts of prediction error, and coincide with reportable event boundaries [37]. We propose that these shifts correspond with the experienced boundaries that define the beginning and ending an emotional event.

Emotional events correspond to complex situations that can be apprehended from multiple dimensional and temporal vantage points [appendix B of 38, also 39]. In cognitive linguistics, the ability to conceptualize the same situation in alternate ways is called construal [40,41]. Aspects of construal include the allocation of attention in terms of specificity (i.e., the locus of attention) and centrality (i.e., the focus of attention) [42, also 43]. As applied to emotions, we understand construal to be the phenomenological equivalent of event perception: shifts in the locus and focus of attention have the potential to update predictions about your internal state, initiating new emotional events (see Figure 3 for an example). Because emotional events generated via changes in construal (a.k.a. conceptualization or categorization) are based on the same situation (e.g., “giving a speech”, figure 3), we speculate that they are perceived as intrinsically linked. When considered in summary, a collection of linked emotional events forms a more extended episode in which sequential perceptions appear to have occurred “at the same time” yielding “mixed emotions”.

Fig. 3.

Schematic depiction of an emotional episode. The same situation is construed from three different foci of attention, generating three linked emotional events, each corresponding to a different categorization of experience.

The mechanisms underlying the experience of mixed emotions have profound consequences for the study of emotion

In sum, we propose that “mixed emotions” are a product of a predictive, internal model of the world that flexibly combines features from previous experience in the service of allostasis and action regulation in a specific situation. New emotional events are perceived when there is sufficient prediction error that the brain implements an event boundary. This account sketches an innovative research program for the study “mixed emotions” (see Table 1).

Table 1.

Open Questions for Predictive Coding as Applied to Mixed Emotions

How are predictions issued and instantiated?

Is only one prediction in a population instantiated (i.e., one representation in an ad hoc concept confirmed by incoming sensory input) or might multiple predictions (i.e., simultaneous cascades) be supported with some probability? Is the conceptual cascade probabilistic at each level of the predictive hierarchy or does a prediction unfold in a deterministic way within the brain?

Are mixed emotions caused by featural conceptual combination, temporal consolidation, or both?

Given that emotions, as predictions, are built with experiential features from multiple previous categories of experience (i.e., conceptual combination), does one necessarily ‘win out’ (i.e., is every instance of emotion uniquely categorized)? Or, is the experience of mixed emotions a product of distinct cascades unfolding over time, and consolidated into summary episodes?

How should the experience of mixed emotions be understood?

Are all episodes of mixed emotions directly reportable, or can they occur outside of awareness? Does the experience of mixed emotions depend on the level of granularity [8] with which an individual experiences emotions (e.g., would very granular individuals report feeling mixed emotions between ‘irritated’ and ‘frustrated’ , whereas less granular individuals would feel mixed only between ‘anger’ and ‘sadness’ , or even between positive and negative affect)?

First, we focus less on individual emotion words as unitary descriptors of experience, and more on mental features of emotion (that can vary across instances of the same word). In our approach, an individual emotion word stands for varying sets of features that need not be physically similar to one another, but that are, instead, serving the same goal or function in a particular context [8,44; see also cognitive accounts of semantic representation, e.g., 45,46–49]. When one person speaks an emotion word to another, it forms the basis of conceptual synchrony and, thereby, efficient communication across the two people [50]. We have revised our measurement strategy accordingly. Instead of calculating the correlations between a variety of endorsed emotion labels across instances over time, as is typically done when measuring mixed emotions [e.g., 51,52], we now rely on high-dimensional analyses in which each emotion word is assumed to refer to a (set of) experiential feature(s) (e.g., E Demiralp, PhD thesis, University of Michigan, 2012).² By considering an individual combination of emotion words as a description of a unique emotional instance, we can better capture the nuanced complexity of emotional experiences (i.e., the diversity of priors that are sampled in conceptual combination) and individual variation therein [see also the emotional patterns approach to measurement; 53]. Indeed, recent investigations of mixed emotions are highly consistent with this conceptual combination account [54,55].

Second, there is a need to model temporal dynamics. The difference in meaningful timescales between neural processing and behavioral measurements makes it impossible to answer computational or mechanistic questions about emotions “at the same time” with self-report data alone [56]. To fully address the phenomenon of mixed emotions, experiments will be optimal when they bridge world and lab to capture multiple features of a brain state (physical, mental, and situational) that can be modeled continuously and at multiple timescales. The segmentation tasks used in tests of event perception theory [e.g., 57; for recent work in language and cognition, see 58; for a review, see 59] can be applied to measure emotional episodes extending over time to test whether they are comprised of distinct emotional events that can indeed be parsed, and at what timescale this parsing occurs. When emotional events are related to a single situation via shifts in attention (i.e., construal), the result may be reported as an episode of mixed emotions. To account for shifting perspectives, future research could complement emerging methods in ambulatory assessment [e.g., 60*] with classic methods for thick description of emotional episodes [e.g., structured diary entries; 61].

Highlights.

• An instance of emotion is constructed using generative patterns of past experience.

• Every instance can combine features from multiple emotion categories.

• The experience of mixed emotions is also a question of timescale and attention.