Through the Eyes of the Beholder: Simulated Eye-movement Experience (“SEE”) Modulates Valence Bias in Response to Emotional Ambiguity

Abstract

Although some facial expressions provide clear information about people’s emotions and intentions (happy, angry), others (surprise) are ambiguous because they can signal both positive (e.g., surprise party) and negative outcomes (e.g., witnessing an accident). Without a clarifying context, surprise is interpreted as positive by some and negative by others, and this valence bias is stable across time. When compared to fearful expressions, which are consistently rated as negative, surprise and fear share similar morphological features (e.g., widened eyes) primarily in the upper part of the face. Recently, we demonstrated that the valence bias was associated with a specific pattern of eye movements (positive bias associated with faster fixation to the lower part of the face). In this follow-up, we identified two participants from our previous study who had the most positive and most negative valence bias. We used their eye movements to create a moving window such that new participants view faces through the eyes of one our previous participants (subjects saw only the areas of the face that were directly fixated by the original participants in the exact order they were fixated; i.e., Simulated Eye-movement Experience). The input provided by these windows modulated the valence ratings of surprise, but not fear faces. These findings suggest there are meaningful individual differences in how people process faces, and that these differences impact our emotional perceptions. Furthermore, this study is unique in its approach to examining individual differences in emotion by adapting a methodology previously used primarily in the vision/attention domain.

Extant research has demonstrated that, in contrast to clearly valenced emotional expressions (e.g., angry, happy)—which are rated consistently across participants—there are individual differences in valence ratings of surprised faces (Neta, Kelley, & Whalen, 2013; Neta, Norris, & Whalen, 2009; Neta & Tong, 2016). This valence bias—an individual’s tendency to rate surprise as positive or negative—is consistent across time (Neta et al., 2009), and related to stable traits (Neta & Brock, under review). By using images that emphasize high (HSFs) or low spatial frequencies (LSFs), we previously examined automatic versus controlled responses to surprise (Neta & Whalen, 2010). More recently, we demonstrated that the valence bias is associated with a specific pattern of eye movements when viewing LSFs and HSFs of surprised and fearful faces, such that, for LSFs (faster processing), individuals with a positive bias attended faster to the mouth (a feature that distinguishes surprise from fear) than those with a negative bias (Neta, Cantelon et al., 2017). To the extent that this pattern of eye movements is important for valence bias, we would predict that we could use the perceptual input from the most positive and most negative participants to manipulate surprise ratings in new participants.

Previous research has used a variety of techniques for constraining visual processing. For example, some work has used a “bubbles” method to vary the perceptual information available from face stimuli by revealing only small and specific parts of the face (e.g., one eye; Gosselin & Schyns, 2001). This method has been useful in demonstrating that individuals with autism show abnormal processing of facial information, including less fixation specificity to the eyes and mouth and a greater tendency to look away from the eyes (Spezio, Adolphs, Hurley, & Piven, 2007). Other work has used the gaze-contingent window in which participants’ eye movements are tracked while they view a stimulus and they are only able to see the locations they are directly fixating via a window that moves in real time to each new location they fixate (the rest of the image is obscured; Figure 1). This technique is commonly applied to studies of reading (Rayner, 2014) but has also been applied in other contexts in which experimenters want to ensure that individuals can only attend to information that is being directly fixated (e.g., McDonnell, Mills, McCuller, & Dodd, 2015). This is particularly important given that individuals can attend to areas that they are not directly fixating (Posner, 1980). Some work has used this method to demonstrate that simultaneous availability of the information from the entire face is crucial for efficient (holistic) face recognition (Maw & Pomplun, 2004; Van Belle, De Graef, Verfaillie, Rossion, & Lefèvre, 2010), that there are cross-cultural universals in face processing (Caldara, Zhou, & Miellet, 2010), and even restored fixation to the eyes in a patient with amygdala damage (Kennedy & Adolphs, 2010). Still other research has used moving windows, for example, where a mouse (held by the participant) controls the perceptual input rather than the participant’s gaze. This method, originally developed to study reading (McConkie & Rayner, 1975) and visual search (Gilchrist, North, & Hood, 2001), is thought to provide a more natural viewing experience than the gaze-contingent window (Dalrymple, Birmingham, Bischof, Barton, & Kingstone, 2011). Again, in the context of face processing, this method has been used to study development of emotion recognition (Birmingham et al., 2013). Generally, however, either the moving window is under the control of the participant (e.g., gaze-contingent window) or the window is designed to move from location to location in a systematic fashion (e.g., to determine the speed with which information can be processed). In the present study, we created a moving window that was based on the actual eye movements of previous participants(i.e., Simulated Eye-movement Experience; “SEE” method). There is evidence in other domains that the manner in which faces are processed—for example, how quickly and for how long specific regions of the face are fixated—can impact subsequent recognition memory (e.g., McDonnell, Bornstein, Laub, Mills, & Dodd, 2014), and our previous work demonstrates that individual differences in eye movements are related to the valence bias. This leads to the question of whether constraining the manner in which faces are perceived will influence how emotional faces are interpreted.

Figure 1.

(A) An example of the perceptual input received by the participants via the moving window. Two windows were created from the eye movements of two previous participants who deemed surprise the most positive (SEE Pos) or the most negative (SEE Neg), with window type manipulated between subjects. The window moved smoothly and continuously in real time with the same timing parameters and fixation locations of our previous participants such that images could be viewed through the eyes of another individual. (B) A significant Expression × SEE Group × Valence Bias interaction revealed that the SEE Neg group rated surprise more negatively than the SEE Pos group, specifically in the individuals with a positive valence bias (Spos group: p = .006), but not those with a negative valence bias (Sneg group: p = .781). Errors bars denote standard errors.

Importantly, much of the previous findings have focused on examining universal effects of face processing or group-level differences (e.g., autism; Spezio et al., 2007). More recent work used modification training and adaptation effects to shift biases in emotion recognition, such that individuals perceived more happiness than anger in ambiguous expressions (happy-angry morphs; Penton-Voak et al., 2013). However, the current study is unique in its approach to examining individual differences in emotion processing of intact facial expressions with greater ecological validity(i.e., not morphed across two incongruent expressions). Moreover, while there have been a variety of methods that constrain perceptual input (bubbles, gaze contingent, and moving windows), this study is unique in creating a moving window that is entirely based on the eye movements of another individual, allowing these participants to view the faces “through their eyes.”

The participants in Neta, Cantelon et al. (2017) viewed the same surprised and fearful faces (in the same order as our present participants) and were required to indicate whether they perceived each expression as positive or negative while their eye movements were tracked. In the present study, we took the eye movement data of the individual who evaluated surprise the most positively and the individual who evaluated surprise the most negatively in the Neta, Cantelon et al. (2017) study and used their real-time saccades and fixation locations to create a moving window for our new participants. Thus, the present participants viewed the images in the exact same manner as one of these previous participants. This study represents the first attempt to determine whether the specific visual input is important for emotion perception, specifically in the context of resolving the ambiguity of emotional facial expressions. We predicted that the perceptual input provided by the window will modulate valence ratings of surprise, such that individuals viewing faces “through the eyes” of the most positive subject will show more positive ratings than those viewing faces “through the eyes” of the most negative subject.

Methods

Participants

An a priori power analysis using prior research (Neta, Cantelon et al., 2017) indicated a requisite sample size of 134 participants to replicate an effect size (r = .3) with 95% power and α = .05, when comparing first fixation to the mouth and more positive ratings of surprise. We tested 145 participants who were right-handed, had no history of psychological or neurological disorders, and were not taking any psychotropic medication. Additionally, all participants were Caucasian in order to control for any racial influences on responses to images of Caucasian faces. The local Institutional Review Board approved all research protocols, and participants gave written informed consent prior to testing. Seven participants with non-normative ratings (accuracy for happy and angry faces below 60%) were excluded. Nine were excluded because of missing data. The final sample included 128 subjects.

Behavioral Testing

Participants were initially given a task assessing valence bias, in which they viewed full images of happy, angry, and surprised faces and were asked to rate each image as positive or negative(i.e., forced choice) based on a gut reaction, as in previous work (Neta et al., 2009; Neta & Whalen, 2010; Tottenham, Phuong, Flannery, Gabard-Durnam, & Goff, 2013). Happy and angry expressions were included because, as in previous work, these expressions serve as anchors for clear positive and negative emotion. These trials also serve as a performance check (i.e., 7 participants were excluded for non-normative ratings of these expressions). In other words, because there is no “correct” answer for surprised faces, the ratings of both clearly positive and negative faces allow us to ensure that only participants who were able to follow the task directions were included. E-Prime software (Psychology Software Tools, Pittsburg, PA) was used for this task. Images included four male and four female faces from the NimStim (Tottenham et al., 2009), each with a happy, angry, or surprise expression. These images were presented in blocks of 24 (8 per expression) in a pseudorandom order, and blocks were counterbalanced between participants. Stimuli were presented for 500 ms with an interstimulus interval of 1500 ms. This relatively brief stimulus presentation was chosen in order to promote responses based on a gut reaction, consistent with previous work that has measured valence bias (e.g., Neta et al., 2009; Neta et al., 2013). We calculated valence bias for each participant using percent negative ratings of surprise faces, or the total number of trials rated as negative divided by the total number of trials per condition (ranging from 0%–100%).

Participants then completed the moving window task, in which they rated 174 surprised and fearful faces—which appeared either intact or high/low spatial frequency filtered—as being either positive or negative. Face identities were counterbalanced, such that each subject viewed a given face as either filtered (the HSF and LSF versions in a counterbalanced order) or intact (two presentations of the intact version), as in previous work (Neta & Whalen, 2010). Specifically, we avoided presenting the same identity in both intact and filtered versions to a given subject, so that the intact versions would not affect ratings of the filtered images (see Vuilleumier et al., 2003). The images and their order of presentation were all drawn from Neta, Cantelon et al. (2017), with the key manipulation being whether each participant viewed faces through the moving window of our previous participant who rated surprise the most positively (i.e., the individual from Neta, Cantelon et al. (2017) who most consistently rated surprised faces as positive; SEE positive group; SEE Pos) or our previous participant who rated surprise the most negatively (i.e., the individual from Neta, Cantelon et al., 2017 who most consistently rated surprised faces as negative; SEE negative group; SEE Neg). Thus, the difference between these groups is the perceptual input they received. Importantly, this task included filtered stimuli because the stimuli were taken from a previous study in which we collected eye movement data while participants viewed surprised and fearful faces that had been filtered (Neta, Cantelon et al., 2017). Thus, the stimuli presented here are naturalistic data from the participant that had the most negative valence bias from that study (SEE Neg), and from the participant that had the most positive valence bias from that study (SEE Pos). We wanted the present participants to have the exact same visual experience as those previous participants and, as such, included all trials from our previous experiment. Furthermore, the important effects relating eye movements to valence bias were evident for LSFs and HSFs of surprised and fearful faces (for example, individuals with a positive bias attended faster to the mouth [a feature that distinguishes surprise from fear] than those with a negative bias, but only for LSF faces; Neta, Cantelon et al., 2017). As such, we focused our analyses on these filtered images. Furthermore, we did not include the intact images in these analyses, based on methods from our original work combining degraded and intact images (Neta & Whalen, 2010) that argues the filtered stimuli are inherently different as they convey less information than an intact image. Stimuli were presented for 2000 ms with an interstimulus interval of 500 ms. The window (which was circular with a diameter of 200 pixels, such that it subtended approximately 4° of visual angle at a viewing distance of 45 cm) moved in a manner identical to how the previous participant(s) moved their eyes (same fixation locations/durations, and same timing). We used a longer stimulus presentation for this task (as in Neta, Cantelon et al., 2017), compared with the bias task, so that we could collect eye movement data while participants viewed each image. Indeed, although there can be slight variations as a function of task, individuals tend to make 3–4 eye movements per second. Brief presentations are problematic for eye movement research because the first fixation is always the location where the eyes were when the image appeared as opposed to being indicative of a processing choice that the individual has made. By extending viewing time to 2000 ms, it ensured that we would get approximately 6–8 eye movements per trial in our initial study, given our interest in linking visual input to emotion perception. This subsequently allowed us to create the moving windows in the present study, which also required multiple eye movements such that the participant had the perceptual experience of the eye moving through space across time. Again, participants were asked to rate each image as positive or negative (i.e., forced choice) based on a gut reaction.

Results

Bias Task

Participants rated clear faces accurately (mean ± standard error: angry = 90.7% ± 0.77, happy = 89.5% ± 0.88). However, there were individual differences in ratings for surprise (66.2% negative ± 1.9; Neta et al., 2009; Neta et al., 2013). Consistent with previous work (e.g., Neta et al., 2009), we used a median split of surprised ratings to divide participants into two groups: individuals with positive valence bias (Spos; mean percent negative ratings = 49.3%; 35 in SEE Neg group and 29 in SEE Pos group) and those with negative bias (Sneg; mean percent negative ratings = 82.2%; 33 in SEE Neg group and 31 in SEE Pos group). Importantly, in contrast to SEE Groups, which were defined by the perceptual input provided to each participant, valence bias groups were defined by the surprise ratings of the current participants in this initial task.

Simulated Eye-movement Experience; “SEE” Task

The dependent measure of interest is percent negative ratings of the filtered images (LSF and HSF), as in previous work (Neta & Whalen, 2010). We conducted an Expression (surprise, fear) × Spatial Frequency (LSF, HSF) × SEE Group (SEE Pos, SEE Neg) × Valence Bias (Spos, Sneg) repeated measures ANOVA. There was homogeneity in bias and SEE groups (ps > .1). A significant Expression × SEE Group interaction (F(1, 124) = 15.0, p < .001, partial η² = 0.11), revealed that the SEE Neg group exhibited more negative ratings of surprise than the SEE Pos group (p = .079), but there was no difference for fear (p = .125). A significant Expression × SEE Group × Valence Bias interaction (F(1, 124) = 6.15, p = .014, partial η² = 0.05) revealed that this effect for surprise was significant for the Spos group (p = .006), but not the Sneg group (p = .781), and there were no significant effects for fear (p = .434 and p = .164, respectively).

Importantly, we have previously argued that LSF and HSF images can be directly compared as they are both degraded, but these conditions cannot be compared to intact images, which are qualitatively different; Neta & Whalen, 2010). Having said that, we also conducted an analysis including intact images: an Expression (surprise, fear) × Spatial Frequency (LSF, HSF, intact) × SEE Group (SEE Pos, SEE Neg) × Valence Bias (Spos, Sneg) repeated measures ANOVA. All reported effects were the same as above.

It is worth noting that, during the moving window task, ratings of surprised faces were considerably more positive than the ratings during the bias task (Spos mean percent negative ratings = 43.7%, Sneg mean percent negative ratings = 58.9%). However, these values cannot be directly compared to the ratings in the constrained viewing task because of other important differences between the tasks. First, the valence bias task included presentations of angry and happy faces as anchors, consistent with much of our research examining valence bias (e.g., Neta et al., 2009; Neta et al., 2013), whereas the constrained viewing task included fear as the comparison to surprise (based on Neta, Cantelon et al., 2017). These comparison expressions are likely to have a dramatic effect on ratings of surprised faces, and may be the reason for the shift toward more positive ratings in the moving window task (i.e., the only comparison was fear, which is clearly negative). Taken together, the valence bias task serves as a better measure of valence bias because there are both positive (happy) and negative (angry) anchors, and because the participants were able to freely view the entire image when making a rating. Thus, caution is warranted when comparing ratings across tasks.

Discussion

We demonstrated that the visual input associated with a specific valence bias, as provided by the moving window, modulated the valence ratings of surprise, but not fear faces. Fearful expressions are rated as consistently negative across individuals, as they are clearly associated with negative outcomes. As such, the perceptual input provided by the moving windows did not have an effect on the valence ratings of these faces. In contrast, the moving window did result in differential ratings of surprised faces. In other words, surprised faces were malleable as a function of which window the participant received. This finding speaks both to the ambiguous nature of this expression, and the many factors that may impact emotional processing. Taken together, these data suggest that there are meaningful individual differences in how people process faces, and these differences impact our emotional perceptions.

This study builds on previous work demonstrating that a specific pattern of eye movements is associated with this bias. We used the visual input from the participants with the most positive and most negative valence bias in our prior work to create simulated eye-movement windows. We found that participants viewing surprised faces “through the eyes” of the most positive participant rated surprise more positively than those viewing surprise “through the eyes” of the most negative participant. Notably, the visual input was realistic data taken from individuals with a positive/negative bias, not from individuals with the most representative eye movements (e.g., looking faster to the mouth is associated with positivity).

Importantly, this effect was significant only in individuals with a baseline positive valence bias. Our ongoing work has shown that individuals with a positive bias are more malleable (e.g., Neta & Whalen, 2010), perhaps because they take longer to make a valence rating about surprise (Neta & Tong, 2016), and thus are more susceptible to manipulations that constrain perceptual input. Given that the window forces individuals to process facial features in a set and ordered manner, our participants may have perceived some regions of the face earlier and/or longer than they would have if allowed to process the face in any manner they choose. In contrast, individuals with a negative bias rate surprise based on a faster and more automatic response that is likely less susceptible to the same manipulation.

Related work has shown the participants with a more positive bias can be shifted in the negative direction in situation of stress (Brown, Raio, & Neta, 2017) or threat of shock (Neta, Cantelon, et al., in press). However, other work has shown that the negative participants might change their ratings following a simple manipulation that encourages them to deliberate longer before making a judgment (Neta & Tong, 2016), and following an intervention that relies on emotion regulation training (Neta, Tong, Brown, & Davis, 2017). Having said that, the interventions that allow negative people to see the glass half full are not the same as those that shift positive people in the more negative direction. The one exception to this includes a manipulation of time perspectives that has been examined in the context of age-related change in emotion. Specifically, the socioemotional selectivity theory posits that time perspectives influence goals such that an extended time perspective (associated with early life stages, e.g., young adulthood) prioritizes future-focused, preparatory goals such as gaining knowledge and exploring and experiencing novelty, whereas a limited time perspective (associated with later life stages, e.g., older adulthood) prioritizes present-focused goals aiming to achieve well-being and emotional gratification (Carstensen, 1992, 1995). Importantly, an extended time perspective is associated with more negative emotion, and a limited time perspective is associated with more positive emotion (e.g., Charles, Mather, & Carstensen, 2003). Numerous studies have found that when younger adults were primed with a limited time perspective, or when older adults were primed with an extended time perspective, the age-related positivity effect diminished (e.g., Cypryańska et al., 2014; Fung, Carstensen, & Lutz, 1999; Kellough & Knight, 2012). The time perspective manipulation has also been demonstrated within a population of college students by comparing college seniors, who naturally had a more limited time perspective with regards to their college experience, as compared to freshmen (Pruzan & Isaacowitz, 2006). Relative to freshmen, seniors demonstrated emotional responses consistent with the limited perspective in older adulthood; they spent less time viewing sad images and reported to have higher levels of positive affect. Consistent with these findings, we have recently demonstrated that priming college students with a limited time perspective (imagining graduation day) resulted in more positive ratings of surprised faces, whereas priming college students with an extended time perspective (imagining their first day on campus) resulted in more negative ratings of surprised faces (Neta, Tong, & Henley, in press). Importantly, this manipulation may work to shift ratings in both positive and negative individuals because they are related to a natural shift from negativity to positivity that takes place in all people over the course of our lifetime. In other words, these changes in emotion are related to effects of psychological maturity that have been linked with other bidirectional emotion-related changes in childhood (Tottenham et al., 2013), early life stress and mental health outcomes (Vantieghem et al., 2017), and healthy aging (Charles et al., 2003). Importantly, some of this previous work has argued that a more positive valence bias represents a compensatory adaptation that promotes resilience (Neta, Tong, Brown, & Davis, 2017), particularly following early life stress(i.e., institutional caregiving; Vantieghem et al., 2017).

Finally, we propose that our “SEE method” will be useful in examining individual differences in visual processing. In particular, compared to other methods that have restricted perceptual input (e.g., bubbles, gaze-contingent window, moving window), the “SEE method” is unique in creating a moving window that is entirely based on the eye movements of another individual, allowing these participants to view images “through their eyes.” This approach is likely to be useful in a variety of research domains, particularly those examining effects that are modulated by empathy or the experience is perceiving the world through the eyes of another person.