Seeing What You Feel: Affect Drives Visual Perception of Structurally Neutral Faces

Abstract

Affective realism, the phenomenon whereby affect is integrated into an individual’s experience of the world, is a normal consequence of how the brain processes sensory information from the external world in the context of sensations from the body. In the present investigation, we provided compelling empirical evidence that affective realism involves changes in visual perception (i.e., affect changes how participants see neutral stimuli). In two studies, we used an interocular suppression technique, continuous flash suppression, to present affective images outside of participants’ conscious awareness. We demonstrated that seen neutral faces are perceived as more smiling when paired with unseen affectively positive stimuli. Study 2 also demonstrated that seen neutral faces are perceived as more scowling when paired with unseen affectively negative stimuli. These findings have implications for real-world situations and challenge beliefs that affect is a distinct psychological phenomenon that can be separated from cognition and perception.

Affective realism refers to the idea that affective feelings help to construct your experience of the world (Anderson, Siegel, White, & Barrett, 2012; Barrett & Bar, 2009). Feelings do more than influence judgments of what you have seen; they influence the actual content of perception. Affective realism is consistent with neuroscientific evidence that the brain constructs experience by using past experience (i.e., memory) to anticipate sensory inputs and that these signals are corrected by sensory information from the world (Barrett, 2017; Chanes & Barrett, 2016; Clark, 2013). From this perspective, called predictive coding (Clark, 2013), active inference (Friston, 2010), or belief propagation (Deneve & Jardri, 2016), perceptions derive from the brain’s “predictions” about the causes of sensory events, based on past experience, with incoming sensory input, prediction error, serving to check those predictions. Anatomic, physiologic, and metabolic evidence (Chanes & Barrett, 2016; Kleckner et al., 2017) indicates that we do not see the world veridically, with cognition and emotion biasing perception in a top-down fashion. Instead, we see it as we predict it to be (i.e., consistent with our internal model of the body in the world), with sensory inputs confirming or adjusting that internal model.

Neuroscientific and behavioral studies suggest that affective feelings are integral to the brain’s internal model and, thus, perception. The cytoarchitecture of limbic regions puts affective feelings at the top of the brain’s predictive hierarchy, driving predictions throughout the brain as information cascades to primary sensory and motor regions (Barbas, 2015; Barrett & Simmons, 2015; Chanes & Barrett, 2016). These same regions control allostasis, the process of coordinating resources across physiologic systems to regulate metabolic energy (Kleckner et al., 2017), and the resulting internal sensations, called interoception (Craig, 2015). As a consequence, interoceptive sensations and their low-dimensional representations (Barrett, 2017; Barrett & Bliss-Moreau, 2009) are at the core of the brain’s internal model (Barrett, 2017; Chanes & Barrett, 2016; Craig, 2015) and, therefore, perception. The predictive structure of the brain and the driving role of limbic cortices help explain why affective properties of valence (pleasantness to unpleasantness) and arousal (Barrett & Russell, 1999) are basic features of consciousness, akin to loudness and brightness (Damasio, 1999; James, 1890/2007). Affect is not unique to instances of emotion but is present in every conscious moment, including during perceptions of the outside world.

In this article, we show that affective realism changes how people see one another, literally. We used an interocular suppression technique, continuous flash suppression (CFS; Tsuchiya & Koch, 2005), in which a neutral face is presented to one eye at full contrast while an affective face is presented to the other eye at low contrast. The neutral face is consciously perceived, whereas the affective face is suppressed from awareness but processed nonetheless. Research using CFS has revealed that affective information presented outside of awareness changes first impressions of neutral faces (Anderson et al., 2012). We demonstrated that affective realism extends beyond broad social judgments to the visual perception of neutral faces: Individuals perceive structurally neutral faces as more smiling or scowling when paired with unconscious, affective information.

Study 1

Method

Participants

Participants were undergraduate students recruited from Northeastern University with normal or corrected-to-normal vision without glasses. Forty-five participants (30 females, 15 males; age: M = 19.07 years, SD = 1.30) completed the experiment for credit toward the completion of their introductory psychology course. Sample size was determined by conducting a power analysis in G*Power (Faul, Erdfelder, Lang, & Buchner, 2007) using effect sizes from previous research in our laboratory that employed a similar experimental task (Anderson et al., 2012, Study 4). This power analysis revealed that for an effect size (η²) of 0.1 to be detected (80% chance) with significance at the 5% level, a sample of at least 40 participants would be required. Two participants were removed prior to analyses because of problems with calibrating the stereoscope during their experimental session, leaving a final sample of 43 participants.

Stimuli and apparatus

Instructions and stimuli were presented using E-Prime (Version 2; Schneider, Eschman, & Zuccolotto, 2012). Each participant viewed stimuli through a mirror stereoscope, a visual device that uses mirrors to simultaneously present different images, one to each eye, while leaning his or her chin and forehead on the rests of the device.

Stimuli included photographs of houses used for the contrast adjustment task, described below, and a series of high-contrast Mondrian-type images similar to those used by Tsuchiya and Koch (2005) that were “flashed” during CFS trials (an example trial can be seen in Fig. 1).

Fig. 1.

Trial structure for face perception task.

Additional stimuli included images of faces with smiling, scowling, and neutral expressions that were pulled from a normed set of facial stimuli developed in our laboratory (IASLab Face Set; http://www.affective-science.org/face-set.shtml). Images of faces had no visible teeth and were cropped to 150 (width) by 169 (height) pixels at 100 dpi. From these face images, we generated an additional set of morphed facial stimuli for use in response scales during the face perception task. The morphed facial stimuli represented a blend of affective expressions (i.e., smile and neutral expressions and scowl and neutral expressions). Using Abrosoft Fantamorph software (www.fantamorph.com), we morphed images of the same person displaying a neutral expression and smiling and displaying a neutral expression and scowling. Still images were selected at 20% smile (80% neutral), 10% smile (90% neutral), 10% scowl (90% neutral), and 20% scowl (80% neutral). Sample morphed facial stimuli can be seen in Figure 2. All stimuli were presented in gray scale on a 19-in. monitor.

Fig. 2.

Sample response screen. Participants indicated which of the five faces they just saw from a set of five faces that varied from slightly scowling to slightly smiling (1 = 20% scowl, 2 = 10% scowl, 3 = neutral, 4 = 10% smile, 5 = 20% smile).

Contrast adjustment task

We first established eye dominance for each participant using the hole-in-the-card test (Dolman, 1919) because suppression of images under CFS is more easily achieved when images are presented to the nondominant eye. Participants then completed a contrast adjustment task, during which the contrast level of images presented to the nondominant eye under CFS was adjusted to improve suppression on an individual basis. Each trial of the contrast adjustment task lasted 1,200 ms. On a given trial, participants were presented with a fixation point to both eyes for 500 ms. Then, the dominant eye was presented with six Mondrian-type images for 100 ms each; the alternating pattern of Mondrian images helped achieve CFS (Tsuchiya & Koch, 2005). Concurrently, the nondominant eye was presented with an empty frame for 100 ms and then with a low-contrast, low-luminance image of a house (either right-side up or upside down) for 200 ms. An empty frame was then presented in the nondominant eye for the remaining 300 ms. Following this sequence, a backward mask was presented to both eyes for 500 ms. Participants reported the orientation (upside down or right-side up) of the suppressed house image on each trial by clicking one of two keys on the keyboard. Participants also rated their subjective awareness of the suppressed house using the 4-point Perceptual Awareness Scale (Ramsøy & Overgaard, 2004), from 1, no experience, to 4, absolutely clear experience. Images of houses were presented at four discrete contrast levels, created by reducing the contrast and luminance levels of the original photographs to 75%, 50%, 25%, and 12.5%. For the first 20 trials of this task, all house images were presented at 75% contrast with half of the trials containing right-side-up images and half containing upside-down images. If any participant correctly guessed the orientation of the suppressed house on 70% of the trials or reported “no experience” on less than 75% of trials, the contrast level was reduced, and the participant completed another 20 trials of this task at the next lowest contrast level. This procedure was repeated until the participant correctly guessed the orientation on 13 or fewer trials and reported “no experience” on at least 15 trials, or until the 12.5% contrast level was reached. The contrast adjustment task determined the individualized contrast level at which all suppressed images would be presented for the remainder of the experimental tasks for each participant.

Face perception task

See Figure 1 for a visual representation of the trial structure for the face perception task. On each trial of the face perception task, perceivers were presented with a fixation point to both eyes for 500 ms. Following this, the dominant eye was presented with a Mondrian-type image for 100 ms, a face displaying a neutral facial expression (the target face) for 100 ms, another Mondrian-type image for 100 ms, the target face for 100 ms, and then a final Mondrian image for 100 ms. The alternating pattern of the target face and Mondrian images helped to achieve CFS. Concurrently, the nondominant eye was presented with an empty frame for 100 ms and then with a low-contrast, low-luminance face for 200 ms (the suppressed affective face); faces were smiling, were scowling, or displayed a neutral expression. Suppressed affective faces were the opposite gender of the target face. An empty frame was presented in the nondominant eye for the remaining 300 ms. Following this sequence, a backward mask was presented to both eyes for 500 ms.

We used 18 unique neutral target-face identities (9 male, 9 female), and each was matched with a unique identity of the opposite gender (to serve as a paired suppressed face). These identity pairings were consistent across all participants. The facial configurations portrayed by each suppressed identity (i.e., smiling, scowling, neutral) were counterbalanced, however, across participants (i.e., for all participants, Male A, posing a neutral expression, was paired with Female A, but Female A was smiling for some participants and scowling for others, etc.). For each participant, 6 of the suppressed identities (3 male, 3 female) portrayed each of the three expressions (i.e., smiling, scowling, neutral), and the expression of a given suppressed identity did not change throughout the course of the experimental session. For each participant, each neutral target and suppressed affective face pairing was shown 10 times. This resulted in a total of 180 trials (6 neutral target faces × 3 suppressed affective expression conditions × 10 repetitions). The task was divided into two blocks of 90 trials each, and participants were given a 2-minute break to rest their eyes between blocks.

At the conclusion of each trial, following the 500-ms backward mask, participants made two ratings on a standard keyboard. First, they indicated the gender of the face they saw by choosing “male,” “female,” or “don’t know.” They were instructed to choose “don’t know” if they had trouble determining the gender, saw more than one gender or face, or saw a blend of two genders or faces. Because the suppressed face was always the opposite gender of the seen neutral target face, this gender question was used as a trial-by-trial measure of subjective awareness of the suppressed face. All trials in which the suppressed face “broke through” the suppression effect to reach subjective awareness (i.e., where the participant selected the gender of the suppressed face or the “don’t know” option) were excluded from analyses (7.24% of all trials). Thus, we excluded every trial in which participants reported any subjective awareness of another face, not just those in which participants selected the gender of the suppressed face. Participants then completed a perceptual matching task (Witt & Proffitt, 2005), in which they identified the image that best matched their perception of the neutral target face. To do this, they were shown a set of five faces and asked to select which of the five faces they saw on that trial (see Fig. 2). All five images were of the seen neutral target face from the trial. However, the faces varied slightly in expression from 20% scowling to neutral to 20% smiling (see details on creation of morphed images in the Stimuli and Apparatus section). For analyses, images were numbered such that lower numbers indicated more scowling and higher numbers indicated more smiling (i.e., 1 = 20% scowl, 2 = 10% scowl, 3 = neutral, 4 = 10% smile, 5 = 20% smile).

Procedure

Each participant was greeted by a research assistant who confirmed that the participant had normal or corrected-to-normal vision without glasses. The participants then received a brief verbal description of the experiment and provided informed consent. Next, participants provided demographic information, including gender, race, age, and handedness. The researcher assessed the participant’s eye dominance and led the participant into an individual testing room with a computer and a mirror stereoscope. The researcher instructed the participant to sit with his or her head positioned on the chin and forehead rests of the stereoscope. The research assistant calibrated each participant to the stereoscope, adjusting the mirrors and rests as needed so that the stimuli being presented were aligned with the participant’s eyes. Before each of the experimental tasks, the researcher read instructions and watched while the participant completed five practice trials. The researcher left the participant alone in the testing room with the lights off while he or she completed each task. Participants first completed the contrast adjustment task and then the face perception task. At the end of the experimental session, researchers administered a debriefing questionnaire assessing participants’ awareness of the suppressed stimuli and the purpose of the experiment, as well as their comprehension of task instructions. They were then debriefed about the nature of the study and remunerated for their participation.

Results

As predicted, a repeated measures analysis of variance (ANOVA) revealed a significant effect of the suppressed affective stimuli on the visual perception of the seen neutral faces, F(2, 42) = 9.72, p < .001, η² = .32, 95% confidence interval (CI) = [0.08, 0.48] (see Fig. 3). The data were also examined by estimating a Bayes factor using Bayesian information criteria (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), comparing the fit of the data under the null hypothesis and the alternative hypothesis. A two-sided Bayesian repeated measures ANOVA (with a default Cauchy prior width of r = .707) revealed a Bayes factor (BF₁₀) of 62.9, indicating that the observed data are 62.9 times more likely under the alternative hypothesis (that suppressed affective information will influence perception) than under the null hypothesis. A Bayes factor of 62.9 is considered very strong evidence in favor of our hypothesis.

Fig. 3.

Violin plots showing mean ratings of seen expressions by suppressed-affective-information condition in Study 1. Individual dots represent each participant’s mean rating for each face type. Rectangular boxes represent the interquartile range of the distribution, with the line in the middle representing the mean. Density of the violin plots represents the density of the data at each value, with wider sections indicating higher density. Error bars represent ±2 SD. Lower ratings for seen expressions correspond to more intensely scowling morphs, whereas higher ratings correspond to more intensely smiling morphs.

Post hoc analyses revealed that seen neutral faces were perceived as having a more smiling expression when paired with a suppressed affectively positive stimulus (M = 3.18, SD = 0.33, 95% CI = [3.08, 3.28]) than when paired with either a suppressed affectively neutral stimulus (M = 3.08, SD = 0.28, 95% CI = [3.00, 3.16]), p = .03, or a suppressed affectively negative stimulus (M = 3.01, SD = 0.30, 95% CI = [2.92, 3.10]), p < .001. Neutral faces were, in turn, perceived as having a more smiling expression when paired with a suppressed affectively neutral stimulus than a suppressed affectively negative stimulus, p = .04. There were no differences in reaction time across the affective conditions, nor were there any differences in the number of breakthrough trials across affective conditions (during which participants reported conscious awareness of smiling, scowling, and neutral faces at similar rates), Fs < 1.04.

Study 2

Study 2 replicated Study 1 but included an additional objective detection task (detecting a stimulus behaviorally, regardless of subjective awareness; Cheesman & Merikle, 1984). When coupled with the trial-by-trial measure, this provided a more robust, converging assessment of awareness.

Method

Participants

Seventy-one participants (41 females, 29 males, 1 nonresponse; age: M = 23.56 years, SD = 8.09) were recruited from Northeastern University and the surrounding Boston community who had normal or corrected-to-normal vision without glasses. Sample size was determined on the basis of a power analysis calculated for Study 1, adjusted to accommodate expected data exclusions due to better-than-chance performance on the objective awareness task (Anderson et al., 2012). Participants either received credit toward the completion of their introductory psychology course requirements or received $5 per half hour of participation. Prior to analyses, we removed 4 participants who reported completing other experiments in the laboratory that used CFS. One additional participant was removed from analyses because of problems with calibrating the stereoscope during the experimental session, leaving a final sample of 66 participants.

Materials, tasks, and procedure

Materials and tasks in Study 2 were identical to those of Study 1, except that an additional task was completed at the end of the experiment as a second measure of awareness of the suppressed affective faces (to complement the trial-by-trial measure of awareness used in Study 1). With the exception of this additional task, the procedure was identical across Studies 1 and 2.

Objective awareness task

Trials in the objective awareness task were nearly identical to the experimental trials in the face perception task except that (a) suppressed affective faces were presented upside down on half of the trials, and (b) a scrambled image of a face (which was no longer identifiable as a face) was presented to the dominant eye (instead of a neutral target face). Participants completed 72 trials of this task; we presented each of the 18 unique suppressed affective faces from the face perception task four times, twice right-side up and twice upside down (rotated 180°). These suppressed affective faces were presented at the same contrast level as used in the face perception task for each participant. At the conclusion of each trial, participants were asked to guess the orientation of the face (upside down or right-side up) and then to rate the quality of their visual experience on the same 4-point Perceptual Awareness Scale (Ramsøy & Overgaard, 2004) used during the contrast adjustment task. If images presented to the nondominant eye were not successfully suppressed throughout the experiment, participants should have some conscious awareness of the faces and should report the correct orientation of the suppressed affective faces at better-than-chance level during this objective awareness task.

Eighteen of the 66 participants were able to correctly guess the orientation of the suppressed face on 62.5% or more of the trials (better than chance, p < .05, two-tailed). These participants were excluded from all further analyses (n = 48). Moreover, in Study 2, we again excluded individual trials of the face perception task on which breakthrough may have occurred: 10.78% of all face perception trials were removed prior to analyses because participants failed to accurately report the gender of the seen neutral target face.

Results

Replicating Study 1, a repeated measures ANOVA revealed a significant effect of suppressed affective stimuli on the visual perception of the seen neutral faces, F(2, 47) = 3.62, p = .03, η² = .07, 95% CI = [0.00, 0.18] (see Fig. 4). A two-sided Bayesian repeated measures ANOVA revealed a BF₁₀ of 1.4, indicating that the observed data are 1.4 times more likely under the alternative hypothesis (that suppressed affective information will influence perception) than under the null hypothesis. Whereas a Bayes factor of 1.4 is considered anecdotal evidence in favor of our hypothesis, these findings directly replicate those of Experiment 1 (which had a Bayes factor considered “very strong” evidence in favor of our hypothesis) and do so even with the implementation of highly conservative inclusion criteria based on the robust, converging assessments of awareness used in Experiment 2.

Fig. 4.

Violin plots showing mean ratings of seen expressions by suppressed-affective-information condition in Study 2. Individual dots represent each participant’s mean rating for each face type. Rectangular boxes represent the interquartile range of the distribution, with the line in the middle representing the mean. Density of the violin plots represents the density of the data at each value, with wider sections indicating higher density. Error bars represent ±2 SD. Lower ratings for seen expressions correspond to more intensely scowling morphs, whereas higher ratings correspond to more intensely smiling morphs.

Post hoc analyses revealed that seen neutral faces were perceived as having a more smiling expression when paired with a suppressed affectively positive stimulus (M = 3.10, SD = 0.38, 95% CI = [2.99, 3.21]) than when paired with either a suppressed affectively neutral stimulus (M = 3.02, SD = 0.37, 95% CI = [2.92, 3.13]), p = .04, or a suppressed affectively negative stimulus (M = 3.02, SD = 0.36, 95% CI = [2.92, 3.12]), p = .04. In this study, perceptions of seen neutral faces did not differ significantly when paired with a suppressed affectively neutral stimulus or a suppressed affectively negative stimulus, p = .97. Thus, the effect for negative stimuli was smaller in Study 2 than in Study 1, but in both studies, neutral faces were perceived as more smiling in trials with suppressed positive stimuli than in trials with suppressed negative stimuli. There were no differences in reaction time across the affective conditions, nor were there any differences in the number of breakthrough trials by suppressed affective condition (i.e., participants reported conscious awareness of smiling, scowling, and neutral faces at similar rates), Fs < 1.69.

General Discussion

Affective realism provides a novel framework for understanding affective misattribution effects (Clore, Gasper, & Garvin, 2001) and represents a critical extension of work on the role of affect in perception, which has largely focused on lower-level perceptual effects, such as sensitivity for contrast gradients and global versus local feature processing (for a review, see Zadra & Clore, 2011). To our knowledge, we are the first to demonstrate the role of affect in the perception of complex percepts that carry social meaning. The two studies reported here demonstrate that visual percepts are infused with affect. Previous research provided evidence that individuals experience neutral faces differently (i.e., as more likeable or trustworthy) depending on the affective feelings accompanying those faces (Anderson et al., 2012). Our data add to this literature, suggesting that individuals perceive faces differently depending on their affective feelings.

There is debate about whether perceptual matching tasks (such as ours) measure perception or memory (Philbeck & Witt, 2015), but accumulating empirical evidence indicates that the boundary between perception and memory is more phenomenological than physically real (see Fan, Hutchinson, & Turk-Browne, 2016). Visual perception and visual memory are associated with the same neural substrates and rely on shared processes (Slotnick & Schacter, 2004; Ungerleider, 1995). Predictive coding approaches provide further support by suggesting that perceptions are memories, constrained and corrected by sensory inputs from the outside world (Barrett, 2017; Clark, 2013; Summerfield et al., 2006). What a person consciously sees in the moment is a mental representation of the real world, not a direct reflection of it. In our studies, incidental affect was perceived as a property of seen faces in the same way that red is perceived as a property of a rose.

The present studies highlight several important avenues for future research. First, assessing participants’ confidence in their perceptual experience might offer insight into whether affective realism involves modulations in perceptual precision, particularly as affect has been shown to influence confidence ratings in other perceptual modalities (Allen et al., 2016). Future research should explore affective realism across different levels of awareness while exploring the extent of neural processing (e.g., Jiang & He, 2006). Furthermore, using affective stimuli other than faces (e.g., snakes) may also speak to the robustness of affective realism.

That we perceive others differently depending on how we feel may have important real-world implications. For instance, the affective-realism hypothesis may help to explain why police officers perceive targets as more or less threatening depending on the interoceptive information they receive (Azevedo, Garfinkel, Critchley, & Tsakiris, 2017). Research on affective realism stands to fundamentally alter our understanding of how perception influences decision making in real-world scenarios where errors can have costly, potentially deadly, consequences.