Understanding Mixed Emotions: Paradigms and Measures

Abstract

In this review, we examine the paradigms and measures available for experimentally studying mixed emotions in the laboratory. For eliciting mixed emotions, we describe a mixed emotions film library that allows for the repeated elicitation of a specific homogeneous mixed emotional state and appropriately matched pure positive, pure negative, and neutral emotional states. For assessing mixed emotions, we consider subjective and objective measures that fall into univariate, bivariate, and multivariate measurement categories. As paradigms and measures for objectively studying mixed emotions are still in their early stages, we conclude by outlining future directions that focus on the reliability, temporal dynamics, and response coherence of mixed emotions paradigms and measures. This research will build a strong foundation for future studies and significantly advance our understanding of mixed emotions.

It is widely agreed that emotions are multicomponential responses that consist of coordinated changes in subjective feeling, motor expression, and physiology [1]. Research over the past several decades generally has focused on discrete or pure emotions. Interest in mixed emotional states has only recently emerged. On the level of subjective feelings, mixed emotional states are characterized by the co-occurrence of positive and negative feelings [2, 3]. While research on mixed emotions has predominantly focused on studying the subjective feeling component, much remains to be learned about how to study mixed emotions objectively, with respect to both paradigms and measures. This review focuses on the experimental study of mixed emotions, where emotional responses are generated and assessed in the controlled environment of a research laboratory.

Paradigms for Generating Mixed Emotions

For objectively studying mixed emotions, we need psychometrically sound paradigms that not only generate mixed emotions, but also permit a contrast with pure states, as well as a neutral state. To demonstrate distinctness of the mixed emotion from other emotional states, the mixed emotion needs to be contrasted to pure versions of the emotions that make up the mixed state, as well as a neutral emotional state. To demonstrate shared and specific features of the mixed emotion, the mixed emotion needs to be contrasted to a second mixed emotion. The addition of a second mixed emotion, however, considerably inflates the experimental design through addition of not only the mixed emotion itself but also of its own matched constituent emotional states.

Induction of the mixed emotional state needs to be reliable, in the sense that it consistently generates the target emotion within and between individuals. The occurrence of positive and/or negative emotional states in mixed emotions should be simultaneous, stable, and ideally last for the full duration of the trial. To increase measurement reliability, the mixed emotional state should be either sustainable for a relatively long period or highly repeatable. The latter requires homogeneity of the mixed emotion over repeated trials.

Different stimuli have been used to induce mixed emotions in the laboratory, such as recalling naturally occurring situations [4, 5, 6], fictitious press advertisements [7, 8, 9, 10], listening to self-selected music [11], the paired-picture paradigm [12, 13, 14], or excerpts from motion pictures [15, 16, 17, 18]. However, these stimuli have been either ideographic (e.g., recall, self-selected music) or not naturalistic (e.g., scenes from motion pictures), less clear regarding simultaneity, stability, and time course of the mixed emotion (e.g., paired-picture paradigm), focused on a specific type of mixed emotion (e.g., nostalgia, poignancy), limited in number, or missing a neutral comparison condition.

To provide a paradigm that allows generation of an exemplary, highly-repeatable mixed emotional state, we recently created a mixed emotions film library for systematically examining mixed emotions in relation to relevant comparison states, including matched pure states and an emotionally neutral state [19]. Film clips have long been used for eliciting emotions in the laboratory. Film clips are a convenient method for standardized emotion induction and can elicit intense emotions in viewers [20, 21]. Several empirically validated emotion film libraries have been created for inducing specific target emotions [e.g., 22, 23, 24, 25]. However, few libraries provide data on the multiplicity of emotions elicited by a film stimulus [26] and none had been specifically aimed at eliciting mixed emotions. Moreover, film libraries typically use clips from motion pictures, which are heavily edited and often lack realism. Amateur video footage, in contrast, represents an excellent alternative to motion pictures with more realistic emotion-inducing content.

The mixed emotions film library [19] draws on amusement as a prototypical positive emotion and disgust¹ as a prototypical negative emotion [27]. Mixed film clips generate an amusing–disgusting state as a prototypical mixed emotion. Prior research has found the combination of amusement and disgust to frequently co-occur in mixed emotional states [17, 28]. Film clips are of 20–33 seconds duration and reflect naturalistic experiences through amateur film-capture. Film clips were validated based on participants’ subjective ratings of positive and negative emotional feelings. The film library comprises approximately 50 instances each of mixed, positive, negative, and neutral emotional film clips. The film library is extensible to include additional mixed and pure contrast states. These film clips have been demonstrated to allow for the repeated generation of mixed, pure, and neutral emotional states [29, 30, 19]. Given their relatively short duration and high immersiveness, these film clips are expected to sustain the generated emotional state for the duration of the stimulus. However, simultaneity, stability, and time course of emotional states elicited by these film clips remain to be demonstrated. A recent comprehensive catalog of films used in research for emotion elicitation [20] shows that this mixed emotions film library is unique in its domain.

Subjective Measures for Assessing Mixed Emotions

For assessing mixed emotions, we also need psychometrically sound measurement tools. Various criteria may be applied: We may select measures that are face valid or “look like” they are going to measure the essence or qualia of mixed emotions; that have previously been validated to separately assess the pure positive and negative constituent emotions of the mixed emotion; or that together perform well in discriminating mixed emotions from other emotional states. We may additionally require that new mixed emotions measures converge with other mixed emotions measures and diverge with measures that are assumed to assess states other than mixed emotions; produces a reliable and, ideally, scalable indication of the mixed emotion each time it is present; and neither is influenced by nor itself influences the measured phenomenon.

One approach to studying mixed emotional states employs subjective measures, i.e., self-reports of emotional feelings. Different measures have been used.

Subjective Univariate Mixed Emotions Measures

Univariate or direct measures of mixed emotions directly indicate the presence and map the degree of mixed emotions onto a unipolar scale by probing their phenomenological qualia [e.g., 31]. As Figure 1a illustrates, this type of mixed emotions measure shows greater responsivity to mixed emotions than to positive, negative, or neutral emotional states, which would not differ from each other (here and throughout, we will assume a mixed-valence mixed emotional state, but other mixed states are possible). Subjective univariate measures of mixed emotional feelings ask people directly to what degree they experience, for example, mixed [4], conflicting [10], or bittersweet feelings [6].

Figure 1.

Univariate, bivariate, and multivariate measures of mixed emotions. (a) A univariate mixed emotions measure probes the phenomenological qualia of mixed emotion, such that it shows greater responsivity to mixed emotion than to positive, negative, or neutral emotional states. (b) A bivariate mixed emotions measure consists of two independent measures, one of which is uniquely sensitive to positive emotion and the other is uniquely sensitive to negative emotion, that are subsequently combined using some mathematical function, such as their minimum (translating this into a univariate mixed emotions measure). (c) A multivariate mixed emotions measure assesses multiple measures of positive and negative emotion facets at one occurrence (left), which allows for the construction of a multivariate discrimination of the mixed emotional state from its matched pure positive and negative constituent and neutral emotional states (middle). Orthogonal rotation, dimensionality reduction, and creation of discriminant functions can be used to derive composite measures of positive and negative emotion (right; translating this into a bivariate mixed emotions measure). Abbreviations: ME_u – univariate mixed emotions measure; ME_b1 – bivariate mixed emotions measure of pure positive constituent emotion; ME_b2 – bivariate mixed emotions measure of pure negative constituent emotion; Me_mx – multivariate mixed emotions measure of positive or negative emotion facet; min – mathematical minimum function.

Subjective Bivariate Mixed Emotions Measures

Bivariate measures of mixed emotions separately assess the presence and map onto unipolar scales the degree of the pure positive and pure negative constituent emotional states of the mixed emotion. As Figure 1b illustrates, this type of mixed emotions measure requires that one measure be uniquely sensitive to positive emotion, the other measure uniquely sensitive to negative emotion, and their activation independent.

The bivariate positive and negative emotion measures are subsequently combined (on a participant-by-participant and occasion-by-occasion basis) using some mathematical function into a univariate co-occurrence measure of mixed emotions (Figure 1a). Various mathematical projections from two- to one-dimensional space have been proposed for creating this co-occurrence measure [as reviewed in 32], with the MIN score currently being the favored one based on theoretical considerations and empirical evaluation. MIN is calculated as the minimum of the degree of positive emotions and negative emotions [33, 34] and indexes the intensity of mixed emotions [e.g., 17, 35, 18, 36].

Subjective continuous bivariate measures of mixed emotional feelings ask people for two separate ratings regarding the degree of experiencing positive and negative feelings. They may also take the form of retrospective drawing of the degree of positive and negative feelings over time [15, 37], retrospective or real-time ratings of the degree of positivity and negativity assessed within a two-dimensional space [continuous evaluative space grid; 38], or continuous ratings of the degree of positive and negative emotional feelings on dual response boxes [39]. Subjective dichotomous bivariate measures of mixed emotional feelings may simply ask for open-ended retrospective self-report of emotion labels [40, 2] or continuous dual (positive feeling/negative feeling) keyboard button pressing [35].

Subjective Multivariate Mixed Emotions Measures

Multivariate measures of mixed emotions assess multiple measures of positive and negative emotion facets at one occurrence, as shown in Figure 1c, left. Emotion facets, in this context, are understood as characteristics of emotion, which—each individually—may not discriminate mixed from positive and negative emotions but—if considered jointly—may do so, with the limitation that variables may not be linear combinations of each other [cf. multicollinearity; 41]. This type of mixed emotions measure may be of interest if no obvious univariate or bivariate mixed emotions measures existed for the emotional response component of interest or if it were desirable to increase measurement reliability and create empirically derived composite measures of positive and negative emotion.

Unlike univariate and bivariate mixed emotions measures, emotional contrast states included in the experimental design are paramount for the construction of the multivariate mixed emotions measure with respect to differentiation and interpretation: As Figure 1c, middle, shows, multiple measures of positive and negative emotion facets allow for the construction of a multivariate discrimination of the mixed emotion from its matched pure positive and pure negative constituent and neutral emotional states. Orthogonal rotation, dimensionality reduction, and creation of discriminant functions are used to derive independent composite measures of positive and negative emotions [Figure 1c, right; see 42, for an example of this approach]. Discriminant functions would be chosen to maximally differentiate (a) mixed and positive from negative and neutral emotional states (Figure 1c, right, vertical axis) and (b) mixed and negative from positive and neutral emotional states (Figure 1c, right, horizontal axis). This process would result in a translation of the multivariate mixed emotions measure into a bivariate one (Figure 1b). Projected locations of pure positive and pure negative constituent and neutral emotional states within this two-dimensional space are crucial for interpreting derived dimensions as ones of positive and negative emotions (see Figure 1c, right).

Subjective multivariate mixed emotions measures ask people for multiple ratings regarding the experience of various facets of positive and negative emotional feelings (e.g., discrete emotional feelings; Figure 1c left). In the self-report domain, the simplest approach for reducing their multivariate representation into a bivariate projection of—albeit possibly not independent—positive and negative feeling composites has typically been guided by a priori hypotheses about their likely clustering and calculating separate averages over facets of positive and negative emotional feelings. More elaborate pattern analysis approaches may derive superior solutions.

Objective Measures for Assessing Mixed Emotions

While research on the subjective feeling component of mixed emotions has significantly advanced our understanding of this phenomenon, subjective measures in general and of mixed emotions in particular may be subject to biases derived from memory [43], demand effects, or lay theories of mixed emotions [e.g., 44, 2]. Affect labels may be misinterpreted [32, 6] or unipolar rating scales may be misused as bipolar ones [45]. Many measures are limited in their temporal resolution (e.g., retrospective summary measures), place a multi-task burden of self-monitoring on the participants (e.g., dual rating dial or dual button pressing), or may not tap into mixed emotions altogether [e.g., emotional complexity; 32, 46].

A second approach to studying mixed emotions focuses on measuring other emotional response components, which could provide an objective measure of mixed emotions. Expressive (e.g., behavioral reflex response, facial emotional expressions) and physiological (e.g., peripheral and central nervous system) measures allow for the objective and unobtrusive measurement of emotional responses with higher temporal resolution than retrospective subjective mixed emotions measures. They also eliminate the need to self-monitor and multi-task, as is the case for continuous subjective mixed emotions measures. However, theory and research in the domain of objective mixed emotions measures are much less developed than in the domain of subjective measures. We suggest candidate measures for each of the measurement categories below.

Objective Univariate Mixed Emotions Measures

Just as for subjective univariate mixed emotions measures (e.g., mixed, conflicting, or bittersweet feelings), a measure would qualify as an objective univariate mixed emotions measure if it exclusively responded to the presence of mixed emotions, i.e., had greater responsivity to mixed emotions than to positive, negative, or neutral emotional states, which would not differ from each other (see Figure 1a). A candidate for an objective univariate mixed emotions measure could be cardiovascular response. It may be measured at the heart (e.g., timing and contractility) or at the vasculature [e.g., blood pressure; 47]. Systolic blood pressure indicates the maximum arterial pressure when the heart is contracting. It is thought to be involved in effort mobilization [48, 49], increasing with task difficulty [e.g., 50, 51]. Systolic blood pressure has been found to show greater reactivity and recovery if subjects reported experiencing both positive and negative, i.e., mixed, feelings in a stressful speech task than if subjects reported experiencing pure positive or pure negative feelings [52].

Another candidate for an objective univariate mixed emotions measure could be activity of neural networks, assessed using functional magnetic resonance imaging [53]. A fronto-parietal network has recently been suggested to be involved in higher-level cognitive value comparisons of conflicting choice outcomes [54]. This network has also been found to show greater activation during mixed emotional feelings in response to disappointing wins (winning the lesser of two amounts) and relieving losses (losing the lesser of two amounts) than during experience of pure positive (outright wins) or pure negative (outright losses) feelings [55].

Objective Bivariate Mixed Emotions Measures

Similarly, just as we have subjective bivariate mixed emotions measures (i.e., constituent positive and negative feelings), two measures would qualify as objective bivariate mixed emotions measures if one responded exclusively to the presence of positive emotion (Figure 1b, horizontal axis), the other responded exclusively to the presence of negative emotion (Figure 1b, vertical axis), and their responses were independent. (For the purpose of the present discussion, we take a simplified view, assuming that certain expressive or physiological measures relate to positive emotion, and other measures relate to negative emotion, while we, of course, acknowledge that the evidence in the literature is more complex.)

Objective bivariate mixed emotions measures may take the form of behavioral reflexes. These are generated by an eliciting probe, such as an auditory stimulus, and assessed through electromyography, where electrodes are placed on the skin over specific muscle sites to record their activity [56]. The postauricular reflex, which pulls the ear backward, is potentiated during positive emotions and remains relatively unchanged during negative and neutral emotional states [57, 58, 59]. The eyeblink startle reflex, which is part of the whole-body startle response, is potentiated during negative emotions and remains relatively unchanged during neutral emotional states [60, 61]. Some additionally report attenuation during positive emotions. Negative emotion, however, appears to have a stronger influence on the startle reflex than positive emotion [cf. negativity bias; 62, 63, 64]. Activation of postauricular and eyeblink startle reflexes is independent: They have different underlying neural circuitries [65, 66, 67, 68], can be concurrently evoked and measured [59, 69], their responses are uncorrelated [70, 71, 69], and have different time courses [72]. Although constituting prime candidate measures, we are not aware of any studies that examine postauricular and eyeblink startle reflexes in mixed emotions.

Objective bivariate mixed emotions measures may alternatively take the form of facial emotional expressions. These can be measured through electromyography [73, 74] or video recording and observer coding [e.g., 75]. Zygomaticus major reactivity, which pulls back the corners of the mouth in forming a smile, increases during positive emotion and remains relatively unchanged during negative and neutral emotional states [76, 77, 78]. Corrugator supercilii reactivity, which furrows the brows in forming a frown, increases during negative emotion and remains relatively unchanged during neutral emotional states [79, 80, 81, 82]. Like the eyeblink startle reflex, some report that it decreases during positive emotion. Again, negative emotion appears to have a stronger influence on corrugator reactivity than positive emotion [cf. negativity bias; 83, 84].

Zygomaticus and corrugator responses appear to be independent, as they can be separately activated [85, 86], likely have unrelated time courses [87, 88], and can be concurrently assessed [e.g., 89, 90, 91]. Mixed smiles, comprising of activation of both zygomaticus and corrugator, have been documented over the averaged time course of brief film clips [29, 30], with second-by-second simultaneous occurrence [92, 93, 94, 11] and coinciding onset [95]. To the best of our knowledge, however, their temporal correlation has up to now not been formally examined, which is important given their shared neural pathways [96].

Objective Multivariate Mixed Emotions Measures

Just as we can have subjective multivariate measures of mixed emotions (i.e., multiple facets of positive and negative emotional feelings), measures would qualify as an objective multivariate mixed emotions measure if they gave insight into some aspect of emotion, be it its mixed, positive, negative, or other affective qualities (see Figure 1c right), as long as included variables were not linear combinations of each other [cf. multicollinearity; 41]. Unlike the subjective domain, however, there may exist neither a clear spectrum of univariate positive or negative emotion indicators nor an intuitive (semantic) clustering of these into positive and negative emotion composites on an objective response component of emotion. Above outlined processing steps (i.e., orthogonal rotation, dimensionality reduction, and construction of specialized discriminant functions) would be used to derive positive and negative emotion composites.

Peripheral physiological responses make a good candidate measure for an objective multivariate mixed emotions measure. Peripheral physiological responses associated with emotions have been measured through reactivity changes in the effector organs innervated by sympathetic and parasympathetic branches of the autonomic nervous system [97, 98]. These include measures taken from the cardiovascular, electrodermal, and respiratory systems, such as heart rate, blood pressure, skin conductance, or respiration rate. While at current, we do not have generalizable univariate peripheral physiological indicators of positive or negative emotion [99, 100, 101], the search continues [e.g., 102, 103, 104, 105]. However, peripheral physiological responses may still reveal some facet of emotion, as discrimination of positive and negative emotion on the level of multiple physiological variables seems possible [106]. Research suggests that positive and negative emotions involve specific and distinct multivariate peripheral physiological responses [107, 108, 109, 110, 111]. This also seems to be true for the peripheral physiological response of mixed emotions, which is not simply reducible to that of its pure constituent emotions [29, 30].

Similar to peripheral physiological activity, broad sampling of brain activity, such as through electroencephalography, may qualify for an objective multivariate mixed emotions measure. In electroencephalography, an array of electrodes is placed on the scalp to capture the brain’s electrical activity. Currently, there does not seem to exist a single electroencephalography measure that could quantify presence and/or degree of positive or negative emotion. However, successful discrimination of positive and negative emotion based on multichannel electroencephalography has previously been reported [112, 113, 114], as has differentiation of a mixed emotional state [115]. Of course, above outlined analysis steps for creating a multivariate mixed emotions measure based on either peripheral physiological or electroencephalographic measures would need to be completed to test these measures’ capacity to truly quantify mixed emotions.

Future Directions

To make the objective study of mixed emotions, including paradigms and measures, a useful tool for research, several fundamental issues remain to be addressed.

Reliability, Validity, and Generalizability of Mixed Emotions Measures and Paradigms

In developing and selecting mixed emotions paradigms and measures, as in any other context, we face the challenge of circularity of their definition, i.e., the degree to which a paradigm is said to induce mixed emotions depends on the measures we use to evaluate it and vice versa. We iteratively refine methods for future research as we grow more knowledgeable based on past research. To continue this process, we need to pay careful attention to effect sizes and psychometric properties of mixed emotions paradigms and measures [cf. 116]. With respect to stability, consistency, repeatability, and generalizability, it seems crucial for future research to examine whether individuals show responses on objective mixed emotions measures that are stable over the duration of one trial; consistent from one trial to the next; and comparable across blocks, sessions, or visits, in different contexts, and between individuals.

Temporal Dynamics of Mixed Emotions Measures

Much remains to be learned about the temporal relation of positive and negative emotion in mixed emotions [117, 118]. As has been done for subjective mixed emotions measures [15, 16, 35, 37], temporal characteristics of objective mixed emotions measures need to be systematically studied to demonstrate that mixed emotions not only occur sequentially or on average but are simultaneous. This not only requires analyses of response stability over the course of one trial [15, 16, 35, 37] but also analysis of temporal dynamics of the mixed emotions response, including characteristics of onset (e.g., latency, rise time), continued activation (e.g., duration), and offset (e.g., recovery time). Of course, the time scale chosen for analysis determines the degree of temporal resolution.

Response Coherence of Mixed Emotions Measures

Emotion theories generally posit the synchronized, coordinated, and/or emergent combination of subjective, expressive, and physiological components of emotion, i.e., emotional response coherence, as a functional definition of emotion [39]. This also applies to mixed emotions. Emotional response coherence is an active area of research [119], but emotional response coherence in mixed emotions remains unexplored. Most typically, subjective or objective mixed emotions measures are utilized in separate analyses focusing on the empirical demonstration of the mixed emotion phenomenon per se. Future research should evaluate emotional response coherence between subjective and objective mixed emotions measures, ideally using multiple measures from multiple domains.

Conclusion

In this review, we have focused on paradigms and measures for studying mixed emotions as situation-related multicomponential emotional states rather than as dispositions or traits. We have also focused on paradigms and measures for examining differentiating features of mixed emotions from relevant contrast states (matched pure states as well as a relatively neutral emotional state) rather than examining shared and differentiating features of one type of mixed emotion from another or of different intensities of mixed emotions. There still exist many unexplored questions regarding mixed emotions. Their systematic study will be important not only in their own right but also to better understand affective space [2] and potential health impacts of mixed emotions [5, 120, 121].

Highlights.

• Emotions consist of subjective, expressive, and physiological responses.

• Research on mixed emotions has predominantly studied subjective feelings.

• Paradigms and measures for studying mixed emotions objectively are needed.

• We describe a mixed emotions film library for eliciting mixed emotions.

• We describe objective measures for assessing mixed emotions.