The dynamic process of ambiguous emotion perception

Abstract

Everyday social interactions hinge on our ability to resolve uncertainty in nonverbal cues. For example, although some facial expressions (e.g., happy, angry) convey a clear affective meaning, others (e.g., surprise) are ambiguous, in that their meaning is determined by the context. Here, we used mouse-tracking to examine the underlying process of resolving uncertainty. Previous work has suggested an initial negativity, in part via faster response times for negative than positive ratings of surprise. We examined valence categorizations of filtered images in order to compare faster (low spatial frequencies; LSF) versus more deliberate processing (high spatial frequencies; HSF). When participants categorized faces as “positive”, they first exhibited a partial attraction toward the competing (“negative”) response option, and this effect was exacerbated for HSF than LSF faces. Thus, the effect of response conflict due to an initial negativity bias was exaggerated for HSF faces, likely because these images allow for greater deliberation than the LSFs. These results are consistent with the notion that more positive categorizations are characterized by an initial attraction to a default, negative response.

Our daily lives are saturated with affective value (e.g., a visit from a friend, the ringing of an alarm clock, a beautiful sunset, a hot cup of coffee). When we encounter new information (new people, sounds, locations, flavors), we readily sort this information into emotional valence categories: good or bad, approach or avoid, reward or threat. Indeed, the human brain quickly predicts affective value based on previous experiences (i.e., whether something is pleasant/approachable or unpleasant/to-be-avoided; Cabanac, 2002). Facial expressions convey particularly rich information about another person and the environment. Some expressions are clear-cut (angry face predicts threat/avoidance), whereas others are more ambiguous, because they readily predict both rewarding or threatening outcomes. For example, a surprised expression is associated with both positive (a friend’s unexpected visit) and negative (hearing that a loved one was in a car accident) outcomes. In other words, when we perceive anger (or happiness) on another’s face, we infer a prototypically negative (or positive) context, whereas when we see a surprised expression, the valence of the context is relatively ambiguous.

Previous research has documented a wide range of individual differences in ‘valence bias,’ or the tendency to categorize ambiguous cues (e.g., surprised faces) as having a positive or negative valence (Kim et al., 2003; Neta et al., 2009). Some individuals overwhelmingly tend to categorize such cues as positive, while others overwhelmingly categorize them as negative. Despite these individual differences, Neta and colleagues proposed an initial negativity hypothesis, such that ambiguous cues initially activate a negative valence representation, which can be overridden by a positive representation via an additional mechanism putatively involved in emotion regulation (Petro et al., 2018).

Various behavioral measures and techniques have been used to provide indirect evidence for this initial negativity hypothesis. For example, reaction times are longer for positive than negative categorizations of surprised faces (Neta et al., 2009), and encouraging greater deliberation results in more positive categorizations (Neta & Tong, 2016). Experimental manipulations that promote hypervigilance and reduce cognitive control (i.e., stress induction, threat-of-shock), on the other hand, result in more negative categorizations (Brown et al., 2017). Finally, low spatial frequency (LSF) images emphasizing faster visual processing (Bar et al., 2006) result in more negative categorizations of surprised faces than the more elaborate processing of high spatial frequency (HSF) images (Neta & Whalen, 2010). However, while these findings are suggestive of an initial negativity (in that, for example, positivity is associated with more processing/response time), there is not clear evidence demonstrating that positive categorizations are characterized by an initial attraction toward a competing negative response.

More recently, research has begun exploiting online hand movements via mouse-tracking to index competing response tendencies during valence categorization. In mouse-tracking paradigms, participants move the mouse from the bottom-center of the screen to response options in either top corner (e.g., negative vs. positive). Despite participants’ explicit response, mouse trajectories may reveal a simultaneous attraction toward the unselected response option (on the opposite side of the screen). Thus, the paradigm moves beyond a delayed processing/response time for positive categorizations of surprise and directly measures competition from an unselected (negative) response alternative, including its particular millisecond-resolution temporal dynamics (Freeman, 2018). This paradigm therefore provides a sensitive window into the process rather than products of categorization. One prior study used mouse-tracking to explore valence categorizations of surprised faces, but focused on trajectories that differed as a function of valence bias and cognitive load (Mattek et al., 2016). Thus, rather than examining the dynamic process of valence categorization across individuals, this work examined trajectories in individuals with a negative versus positive bias and found that bias did not impact trajectories when under high load.

To more directly study the dynamic process of valence categorization when resolving ambiguity across all individuals, and to provide evidence for the initial negativity hypothesis, we used mouse-tracking to examine response trajectories for LSFs (faster visual processing) and HSFs (more elaborate processing) of facial expressions. We predicted a general negativity bias, such that when participants select the ‘negative’ response their trajectories are especially direct; and when they ultimately select the ‘positive’ response, their trajectories reveal an early bias toward the ‘negative’ response, producing response conflict. Importantly, this response conflict due to the negativity bias should be exacerbated by HSF images, as they promote more deliberative processing that would only serve to intensify the conflict, relative to LSF images. Thus, we should observe negative categorizations of surprise (putatively the default) to be characterized by direct trajectories, but positive categorizations (putatively overriding the initial negativity) to be characterized by indirect trajectories that show response conflict, and this should be pronounced for HSF images.

Method

Participants

A power analysis using G*Power3.1 suggested a total sample size of 101 participants would be necessary to achieve 90% power in detecting significant effects with an effect size comparable to previous work (d = .31; Neta et al., 2009). One-hundred and twenty-five participants were recruited from Amazon Mechanical Turk. Seven participants were excluded because they provided incomplete data (responding on less than 80% of trials), and ten were excluded for failing to accurately rate clearly valenced faces (angry/happy) on at least 60% of trials (as in previous work; Neta & Whalen, 2010). This resulted in a final sample of 106 participants (52 female; ages 18-43 years, M_age(SD)=28.71(4.14)). All procedures were approved by the New York University Committee for the Protection of Human Subjects.

Stimuli

The stimuli were taken from previous work (Neta & Whalen, 2010), including an equal number of male and female faces from NimStim (8 individuals; Tottenham et al., 2009), Pictures of Facial Affect (13 individuals; Ekman & Friesen, 1976), and Averaged Karolinska Directed Emotional Faces (KDEF) databases (39 individuals; Lundqvist, Flykt, & Öhman, 1998). Of interest were responses to surprised expressions; angry and happy expressions were included as response anchors and to validate performance on the task (Neta et al., 2009). All images were transformed to gray-scale with a resolution of 75 dots per inch. Facial expressions were validated by an independent set of raters labeling each expression; only faces correctly labeled by more than 60% of raters were included.

Further, the original image (broad spatial frequencies, BSFs) was filtered to create two versions of each face: one comprising primarily HSF information and one primarily LSF information (Fig. 1). We used a high-pass cutoff of 24 cycles per image for HSFs and a low-pass cutoff of 6 cycles per image for LSFs (Neta & Whalen, 2010). Prior to filtering, we equated contrast and luminance across stimuli. For the task, we used 99 stimuli (face identities posing a particular expression) that were counterbalanced such that each subject viewed a given face as either filtered (counterbalanced order of HSF and LSF) or intact (two presentations of the BSF), totaling 198 trials. Faces of all types (BSF, HSF, and LSF) were presented pseudorandomly within each of three runs of 66 trials each. We avoided presenting the same identity in BSF and filtered versions to a given subject so that BSF versions would not affect ratings of the filtered images (Neta & Whalen, 2010).

Figure 1.

Examples of the stimuli. Broad-spatial frequency (BSF) images were intact. These images were filtered to emphasise the high-spatial frequency (HSF) or low-spatial frequency (LSF) information, in order to promote more elaborate or faster processing, respectively. Each expression was presented either as intact (BSF) or filtered (HSF and LSF) to each participant, and this was counterbalanced across participants.

Procedure

Participants categorized faces as positive or negative in a mouse-tracking paradigm using a Javascript implementation of MouseTracker (Freeman & Ambady, 2010). Participants began each trial by clicking “Start” at the bottom center of the screen, after which a fixation cross appeared for 500 ms, followed by a face for 200 ms. Participants categorized the valence of the face’s emotion as quickly and accurately as possible by moving the computer mouse to a “positive” or “negative” response option, which appeared in the upper left and right corners of the screen (counterbalanced across participants), and clicking on the response. There was no time limit within participants needed to complete their response. However, in order to prevent mouse trajectories from being off-line with the decision process, participants were encouraged to start moving within 400 ms of face onset; if they did not, a message appeared following the trial encouraging earlier movement initiation, and the trial was excluded. This criterion resulted in removal of 964 trials out of a total 24,353 across participants (i.e., 3.96% of trials; M(SD)= 7.84(6.45) out of 198 trials per participant). During the categorization process, the streaming x, y coordinates of the mouse were recorded.

Results

Table 1 reports valence ratings scored as the percentage of trials the participant rated as negative out of the total number of trials for that condition (Neta et al., 2009; see Supplementary Material for analysis).

Table 1:

Frequency of negativity ratings by condition.

	Surprised			Angry			Happy
	BSF	LSF	HSF	BSF	LSF	HSF	BSF	LSF	HSF
Mean (SD)	77.45 (24.59)	78.24 (24.73)	76.83 (23.50)	98.07 (4.29)	94.19 (6.88)	93.69 (9.82)	3.14 (6.94)	4.19 (7.06)	9.48 (15.53)
Range (Min- Max)	0-100	0-100	3.45-100	77.78-100	63.16-100	42.11-100	0-38.89	0-33.33	0-100

Note: Dependent measure is Percent Negative Ratings, so mean frequency of positive ratings can be calculated by subtracting the mean frequency of negative ratings from 100

Mouse-trajectory data underwent standard preprocessing (Freeman & Ambady, 2010). All trajectories were rescaled into a standard coordinate space (top-left: ‘‘1, 1.5”; bottom-right: ‘‘1, 0”) and normalized into 100 time bins (101 time-steps) using linear interpolation to permit averaging of their full length across multiple trials. For comparison, all trajectories were remapped rightward. With this orientation, the selected response is located at x=1.0 and unselected response at x=−1.0. Thus, more positive x-coordinates indicated a more direct and facilitated trajectory, while less positive (or even negative) x-coordinates indicated a more indirect trajectory temporarily attracted to the unselected response.

Analyses focused on filtered images of surprised expressions so to emphasize the distinction between faster (LSF) versus more elaborate processing (HSF) of ambiguously valenced stimuli. We used generalized estimating equations multi-level regression (GEE) models, which permits analysis of trial-by-trial data while appropriately accounting for intracorrelations due to the nested data structure (Zeger & Liang, 1986). GEE models of initiation and response times are provided in Supplementary Material. For all GEE models, we report unstandardized regression coefficients (B) and Wald Z-statistics.

Plotting trajectories’ x-coordinates over 101 time-steps for the four conditions [Rating (positive, negative) x Filter (LSF, HSF)] supported our predictions (Fig. 2A): trajectories were more direct for negative than positive ratings, and this was exacerbated for HSF images. To probe this Rating x Filter interaction, at each time-step, we regressed the x-coordinate onto Rating (−0.5=negative, 0.5=positive), Filter (−0.5=LSF, 0.5=HSF), and the interaction using GEE. To solve the issue of multiple statistical testing at each time-step, we used bootstrapping on 10,000 simulated experiments (see Supplemental Material and Dale et al., 2007 for details). The bootstrapping showed that the experiment-wide significance of the Rating x Filter interaction was guaranteed at a criterion of p<.05, p<.01, or p<.001 if at least 5, 6, or 8 consecutive time-steps showed a significant effect, respectively.

Figure 2.

Positive ratings are characterised by an early attraction to negativity particularly in response to HSF faces. (a) There was a partial attraction to the competing (unselected) response option which was more evident on trials rated as positive (attraction to negative) than on trials rated as negative (attraction to positive). This pattern of results was exaggerated for the HSF images, which emphasise a slower, more elaborate processing than the LSF images. (b) We averaged the x-coordinates within the sequence of the trajectory showing a significantinteraction (time steps 48–57) and submitted these to GEE multi-level regression. We found that LSF trajectories demonstrated the negativity bias, with significantly more attraction (i.e. lower x-coordinates) toward the opposite response for “positive” rather than “negative” categorizations(B = −0.08; SE = 0.03; Z = −2.52; p = .01), but this effect was exacerbated in the HSF condition (B = −0.14, SE = 0.03, Z = −4.57, p < .001). Error bars denote standard error of the mean.

The Rating x Filter interaction effect was significant for time-steps 48-57, guaranteeing a significant experiment-wide interaction effect (p<.001). To characterize the nature of the interaction, we averaged x-coordinates within these time-steps and submitted them to GEE. There was a significant effect of Rating (B=−0.11, SE=0.03; Z=−3.88; p<.001), with negative-rating trials showing higher x-coordinates (more direct trajectories) than positive, evidencing a general negativity bias. There was also a significant effect of Filter (B=−0.06, SE=0.01, Z=−4.74, p<.001), with LSF trials showing higher x-coordinates than HSF trials, as expected given faster processing of LSFs. Notably, as expected given the bootstrapping, these effects were qualified by a significant interaction (B=−0.06, SE=0.03, Z=−2.10; p=.04), such that LSF trajectories demonstrated a negativity bias (i.e., more attraction toward the competing response for ‘positive’ than ‘negative’ categorizations; B=−0.08; SE=0.03; Z=−2.52; p= .01) that was exacerbated in the HSF condition (B=−0.14, SE=0.03, Z=−4.57, p<.001). Together, these results show that, even when participants selected the ‘positive’ response, they exhibited an initial attraction toward the ‘negative’ response overall, and this early ‘negative’ activation was stronger for HSF faces (Fig. 2B).

It is possible that what appears to be response conflict and early negativity in the average trajectories could be spuriously produced, for example, if a subpopulation of erroneous “flip-flopping” trajectories were averaged together with very direct trajectories (Freeman & Dale, 2013). To rule this out, we inspected the distribution of x-coordinates during time-steps 48–57, as multimodality would suggest multiple subpopulations of trajectories at play (Freeman & Dale, 2013). Hartigan’s dip statistic (HDS; Hartigan & Hartigan, 1985) provides an inferential test of multimodality against a null hypothesis of unimodality. There was no evidence of multimodality (LSFs rated as ‘positive’ HDS=.011, p=.88 or ‘negative’ HDS=.007, p=.52; HSFs rated as ‘positive’ HDS=.011, p=.931 or ‘negative’ HDS=.006, p=.855).

Another possible explanation is that our results merely reflect perceptual confounds, such that surprised faces show greater resemblance to angry than happy faces. We calculated pixel-based similarity (Pearson correlation between vectorized pixel-maps) between surprised and angry, and between surprised and happy faces, and compared the correlations (Meng et al., 1992). For HSFs, surprised faces were more similar to happy than angry faces (p<.001), contrary to this possibility; for BSFs and LSFs, there was no significant difference (ps=.26 and .12, respectively) with the pattern again biased toward happy faces. Such results favor a genuine negativity bias affecting the categorization process.

Discussion

We provide direct evidence supporting the initial negativity hypothesis, which posits that the more spontaneous interpretation of ambiguity is negative. Specifically, by using mouse-tracking, which provides a unique window into the dynamic process of categorization, we demonstrated that negative ratings of ambiguity show more direct response trajectories, but positive ratings are characterized by an attraction to the negative (competing) response option. Notably, filtering images into different spatial frequency bands enabled us to examine trajectories in response to images that receive priority processing (i.e., LSFs are processed first and fast) as compared to those processed more slowly and deliberately (HSFs; Bar et al., 2006). As predicted, the effect of response conflict due to negativity bias was exaggerated for HSFs, likely because these images allow for greater deliberation than the LSFs. Interestingly, this effect was evident despite there being no difference in categorizations of HSF and LSF surprised faces, suggesting that it was the HSF nature of the images that resulted in increased conflict. Having said that, we note that we did not replicate the finding that LSF surprised faces are rated more negatively than HSFs (Neta & Whalen, 2010). Given that other work using a button press response (i.e., similar to the original report; Park et al., 2016) has established replicability, we propose that response modality may impact this finding.

These findings extend work demonstrating that, despite individual differences in valence bias (Kim et al., 2003; Neta et al., 2009), the initial response is negative (Neta & Whalen, 2010) and positivity arises from greater deliberation (Neta & Tong, 2016). This is also consistent with evidence that the default response to uncertainty is negative (Brosschot et al., 2016). We extend these findings by suggesting that increased deliberation (e.g., longer reaction times) may be associated with greater response conflict. Specifically, we suggest that reaction times are longer for positive than negative trials because positive trials are characterized by a greater attraction to the initial, negative (competing) response option, whereas the negative trials result from more direct trajectories. Notably, this increased response conflict due to negativity bias was exaggerated in response to HSFs, which allow for slower and more deliberate processing; hence, HSFs are more vulnerable to this response conflict. Thus, we leveraged filtered images to more explicitly probe the processes associated with a more direct/automatic (negative) versus more indirect (positive) categorization.

Recent work suggests that the bottom-up negativity is associated with amygdala activation (Kim et al., 2003), and that a top-down mechanism promotes positivity by overriding the initial negativity using an emotion regulation mechanism akin to cognitive reappraisal (Petro et al., 2018). Cognitive reappraisal is a strategy whereby one reframes or reinterprets an emotionally evocative event in order to change the emotional response (e.g., decreasing negative affect; Gross, 1998). With particular relevance to surprised faces, which have a dual-valence ambiguity, we have proposed that participants override the initial negativity by allowing for a more elaborate (re)interpretation of the expression as positive. The present findings support this notion by demonstrating that a positive interpretation is preceded by an attraction toward a negative rating (i.e., participants are not disengaging from the stimuli, but likely overriding the initial negativity). Importantly, this reinterpretation may result from an implicit form of reappraisal (Braunstein et al., 2017), as individuals are not likely aware of this override process. For this reason, using filtered images here was crucial for disentangling trajectories during faster versus more elaborate processing. Future work could use mouse-tracking to examine processes associated with reappraisal and draw a more direct link to the response competition associated with positive ratings.

Finally, we recently reported that experimental manipulations that promote hypervigilance and reduce cognitive control (e.g., stress) result in more negative categorizations (Brown et al., 2017). Notably, individuals that were more sensitive to the stressor (greater cortisol reactivity) showed more direct trajectories toward the ‘negative’ response when under stress. Thus, stress was not only associated with greater negativity, but also decreased response competition in the context of negativity. We have also recently demonstrated that individuals that use reappraisal more frequently in their daily lives are more resilient to this stress-related negativity (Raio, Brown & Neta, submitted). Future work will be useful for determining if other simpler methods for promoting elaborate processing (HSF images) might also help to mitigate stress-related negativity.

Several limitations are worth noting. Although we focus on the ambiguity of surprised faces, there is a broader context-dependency of emotional expressions (Aviezer et al. 2012; Barrett, 2014). Indeed, a particular facial configuration is always “ambiguous” to some extent (e.g., Fernandez-Dols & Crivelli, 2013). However, our approach stemmed from a social signaling standpoint, suggesting that angry expressions are inferred to be prototypically negative and happy positive, but one is more likely to encounter surprised expressions in a range of contexts (positive and negative). Thus, we do not wish to imply a fundamental distinction between angry/happy and surprised expressions’ potential for ambiguity, and future work could test how our findings generalize to cases in which angry/happy expressions are also considered relatively ambiguous. Also, although this paradigm characterizes individual differences in negativity bias, faces are almost never seen without context in the real world. However, even then, the context is not always sufficient for determining a clear emotional valence (e.g., seeing people in a dark alley either in search of a night club or someone to rob). Our valence bias task intentionally withholds context so that individuals must resolve the uncertainty by relying on their own biases. Having said that, it could be that this paradigm is a better measure of one’s tendency to infer that situations are negative or positive, and we are generally agnostic to this distinction; the important feature here is that we can identify one’s bias to perceive a more negative or positive meaning in circumstances where both alternatives are valid with relatively equal probability. Future research could explore these open questions, for example, using paradigms lacking categorical emotion labels.

Conclusions

By leveraging mouse-tracking technology and filtered images that emphasize faster versus more deliberate processing, this work provides direct evidence in support of the initial negativity hypothesis. Specifically, we demonstrate the presence of an early negativity bias that weighs in during the process of resolving emotional ambiguity, even when participants ultimately arrive at a ‘positive’ evaluation. This work lends insight into the underlying mechanism of classifying new, and particularly ambiguous information into emotional valence categories.