Perceiving facial affective ambiguity: A behavioral and neural comparison of adolescents and adults

Abstract

The current study examined perceptual differences between adults and youth in perceiving ambiguous facial expressions. We estimated individuals’ internal representation for facial expressions and compared it between age groups (adolescents: N=108, M_age=13.04 years, 43.52% female; adults: N=81, M_age=31.54, 65.43% female). We found that adolescents’ perceptual representation for facial emotion is broader than adults, such that adolescents experience more difficulty in identifying subtle configurational differences of facial expressions. At the neural level, perceptual uncertainty in face-selective regions (e.g., fusiform face area, occipital face area) were significantly higher for adolescents than for adults, suggesting that adolescents’ brain more similarly represents lower intensity emotional faces than adults. Our results provide evidence for age-related differences of psychophysical differences in perceptual representation of emotional faces at the neural and behavioral level.

Introduction

The ability to recognize and decode others’ facial expressions is an essential feature of social interaction (Adolphs, 2002). Emotion perception is incredibly complex, requiring the individual to both distinguish fine-grained differences in facial configuration and understand complicated, nuanced social context rules (Barrett, Lindquist, & Gendron, 2007). Although there is a robust connection between a confined set of prototypical facial configurations and emotional states (i.e., the “discrete emotions” perspective; Ekman, 1993), face emotion perception is not always determined by specific physical feature of facial configurations, such that various external and internal factors can change an observer’s emotion perception even for the same facial configuration (e.g., Kim et al., 2004; Lee, Choi, & Cho, 2012). Furthermore, emotional expressions are often subtle, ambiguous, and uncertain in everyday social interactions (Fridlund, 2014). Such ambiguity poses particular challenges to adolescents as they learn to identify and appropriately respond to seemingly ambiguous emotional states. Indeed, incorporation of various social cues to interpret others’ emotional states develops in conjunction with improvements in youths’ perceptual abilities (Barrett et al., 2007). Therefore, facial affect perception can be challenging for youth (Gross & Ballif, 1991; McClure, 2000) as perceptual learning of emotions is still developing (Pollak, Messner, Kistler, & Cohn, 2009).

Although evidence to date indicates that adolescents’ perception of others’ affect differs from adults’ perceptions, studies largely utilize overt facial affect recognition tasks that are not designed to capture the oft-ambiguous nature of real-world situations (i.e., ambiguous expressiveness; e.g., Batty & Taylor, 2006; Thomas et al., 2001). Furthermore, much of the research base focuses on clinical populations (e.g.,autism spectrum; Critchley et al., 2000). where affective processing is clearly sub-optimal. Studying the normative development of facial emotion perception is integral to improving our understanding of how affective processing normatively changes over the lifespan. In the only known study to date to examine developmental differences in ambiguous facial affect (Wiggins et al., 2015), adolescents recruited face-processing networks significantly less than adults when the emotional intensity of the face was unclear (i.e., ambiguous; e.g., 50% intensity of fearful face), indicating adolescents’ perceptions of subtle facial expressions may be comparatively underdeveloped. This study suggests that activation in the ventral stream is a likely neural candidate reflecting the maturation of systems for perceiving facial affect.

Building upon this work (Wiggins et al., 2015), we sought to examine the internal representation of perceptual uncertainty for emotional faces between adolescents (N=108) and adults (N=81) by fitting behavioral and neural data to psychophysics model (Fig 1A; Calder, Jenkins, Cassel, & Clifford, 2008; Clifford, Mareschal, Otsuka, & Watson, 2015; Lynn et al., 2016; Mareschal, Calder, Dadds, & Clifford, 2013; Wang et al., 2017). In the present study, we focused on neural pattern similarities between emotional faces as a form of multi-voxel pattern approach (MVPA) to directly fit neural patterns to a psychophysics model. To generate emotionally ambiguous facial expressions, we used happy and angry faces morphed with neutral faces ranging from 15% to 75% intensity levels (Fig 1B). We hypothesized that adolescents would be less sensitive to ambiguous facial emotions. In other words, adolescents would be more likely to perceive ambiguous facial expressions as non-emotional or “neutral” compared with adults, thereby demonstrating broader representations of non-emotional faces.

Fig 1.

(A) The schematic psychophysical model, showing a perceptual representation for emotion perception and perceptual criteria between perceiving emotion (either happy or angry) and non-emotion (neutral) as two perception change points (red and blue line) are closer, observer has more keen criteria in perceiving emotionality from subtle facial expressions as the uncertainty boundary gets smaller (B) An example of face stimuli used in the current study. (C) Neural pattern similarity estimation within the ROI as a function of emotion intensity. Using neural pattern anchor averaged across lowest emotion intensities in both happy and angry, we computed pattern similarities between neural pattern anchor and each intensity using Pearson-r (Fisher-z transformed), then fitted them into the emotion perception model. The matrices (4×4) are just for schematic illustration of pattern within the ROI mask. (D) Group activation map responding to all face stimuli versus baseline during the “observe” round. The stimuli robustly activated regions along the ventral visual pathway. The bar plots show activation strength in those regions on the “Affect label” round as a function of emotion intensity across participants. Note that there was no significant difference for happy and angry stimuli at corresponding stimulus levels (e.g., 75% happy and angry; all Ps>.09) (E) Representative subject’s fitted curves for behavioral and neural data, showing the perceptual boundary representation as a function of facial emotion intensity. The fitting values on average were 0.87 and 0.56 for behavioral and neural data respectively.

Method and Analysis¹

Participants.

An emotional labeling task was presented to 189 participants during an fMRI scan; 108 adolescents (M_age=13.04 years, SD=0.90, range:12–15, 43.52% female) and 81 adults (M_age=31.54, SD=12.47, range:19–54, 65.43% female) participated. The adult sample included younger adults (N=39, college students) as well as older adults, some of whom were the parent of an adolescent in the sample (N=33). Data from eight individuals were excluded due to motion (three adolescents; mean FD=1.10 mm, DVARS=51.52) and technical failure (four adolescents and one adult). The remaining participants for fMRI data analysis (N=181) did not have any motion issues (mean FD=0.11mm, DVARS=29.69; adolescents: FD=0.14mm, DVARS=30.27; adults: FD=0.08mm, DVARS=29.10). All participants provided informed consent and were remunerated for their participation. The study was approved by the Institutional Review Board (IRB) of University of Illinois at Urbana-Champaign (UIUC).

Task and stimuli.

Face stimuli consisted of angry, happy, and neutral expressions. To vary emotional intensity parametrically, we morphed happy and angry faces with neutral faces in 15% increments (e.g., 15%, 30%, 45%, 60%, and 75%, where the percentage indicates the emotional intensity [happy or angry] of each category). Eighty total stimuli comprised these emotion intensity categories (40 happy and angry faces with intensity variations). Participants completed two different variants of the task: “Affect Label” and “Observe” rounds. During the “Affect Label” round, participants were instructed to match the facial emotion of the stimuli displayed with one of three labels (“Happy,” “Neutral,” and “Angry”), displayed across the bottom of the screen, using their index, middle, and ring fingers respectively. During the “Observe” rounds, participants were asked to press their thumb for each face instead of making an effort to label the emotion of face. This “Observe” was designed to serve as a main-task independent localizer for face-selective voxels (Fig S1) with the assumption that it reflects simple face perception without recruiting affective resources explicitly (see RT results in the online supplement; Fig S2.

Data acquisition and preprocessing.

T1-MPRAGE and T2*-weighted echoplanar images (EPI) were collected using a 3T-Siemens Trio MRI scanner with a 32-channel matrix coil. Preprocessing was carried out using FSL 5.0.10 (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki).

Analysis of behavioral response.

We defined perceptual uncertainty for the proportion of trials labeled as “neutral’ as a function of facial emotion intensity (Fig 1A). The more neutral judgements across face emotion intensities represented more perceptual uncertainty for the face emotion. To quantify this perceptual uncertainty level, we computed the proportions of “neutral’ responses (i.e., indicating no emotion perception for a given emotional intensity) for each intensity of face stimuli, and fitted them into the psychophysics model using a Gaussian function representing the perceptual uncertainty boundary in sensory representation (Fig 1A; Clifford et al., 2015; Jun, Mareschal, Clifford, & Dadds, 2013; Mareschal et al., 2013):

$$f(x)=\alpha e^{\frac{-{(x-\mu )}^2}{2{\sigma }^2}}$$ (Equation 1)

where, α represents peak amplitude of responses (i.e., the height of the curve’s peak), μ specifies the position of the center of the peak (i.e., face emotion intensity in which faces were judged as neutral), and σ is the bandwidth (i.e., standard deviation of the curve). The bandwidth parameter, σ, was used as the primary metric for the degree of perceptual uncertainty (Calder et al., 2008; Clifford et al., 2015; Mareschal et al., 2013) as wider curves (larger σ) suggest participants had greater neutral responses in emotion judgement to changes in emotional intensity. That is, participants were less perceptively sensitive to subtle emotional changes in the faces and vice versa for narrower curves (smaller σ). The fitting values (r-square) on average were 0.87 and 0.56 for behavioral and neural data respectively.

Analysis of neural pattern

To fit the neural data on the psychophysics model depicted in Fig 1A, we performed a neural pattern similarity analysis (e.g., Kriegeskorte, Mur, & Bandettini, 2008; Lee, Qu, & Telzer, 2017) by estimating single-trial activation patterns for each emotional intensity based on least-squares-single methods (LSS; Mumford, Turner, Ashby, & Poldrack, 2012). We then extracted standardized voxel-wise pattern activity (i.e., z-map) for each emotion intensity within the ROI on each individual space, and computed the similarity values (i.e., Fisher’s z- transformed Pearson correlation coefficients) across each vector between the pattern anchor (Wang et al., 2017) and the other vectors in each emotional intensity (Fig 1C). The neural anchor was created by averaging the neural patterns of 15% angry and 15% happy faces, and thus the anchor pattern should show very high similarity with both neural patterns of 15% happy and angry faces respectively. Finally, we fitted pattern similarity metrics of each intensity into the psychophysical model. Higher pattern similarities for the anchor indicate neural encoding for a given face is more likely to be perceived as neutral.

ROI selection

For the face-sensitive-ROI selection, we performed a standard two-stage univariate GLM analysis for the “Observe” rounds as an orthogonal functional localizer (Poldrack, 2007). An individual-level GLM estimated brain activation for faces regardless of their intensities contrasted to the baseline (e.g., Bishop, Aguirre, Nunez-Elizalde, & Toker, 2015; Thielscher & Pessoa, 2007), and then group-level random effects were estimated (clusters-corrected Z>2.3, p=0.05; one-tailed; FLAME1+2; Table S1). Finally, we selected voxels that fell within the previously defined functional parcels (http://web.mit.edu/bcs/nklab/GSS.shtml) for face-sensitive-voxels (Julian, Fedorenko, Webster, & Kanwisher, 2012). No clear STS cluster activation was observed and this may be due to our current approach (contrasted with baseline instead of face minus other categorical stimulus such as places). However, it does not suggest that the STS is not a face-selective region; hence our final ROI mask included the FFA and OFA (k=2104 voxels; Fig 1D). Given that previous studies that the amygdala also plays a role in encoding emotion parametrically (Wang et al., 2017), we also selected voxels (k=432) within the bilateral amygdala atlas (Harvard-Oxford, 50%-threshold).

Results²

Each participant’s neutral responses were fitted to the psychophysics model to estimate the uncertainty boundary (i.e., σ). An independent-samples t-test indicated that perceptual uncertainty levels were significantly higher for adolescents (M=45.82, SD=8.25, SE=0.82) than adults (M=43.37, SD=7.49, SE=0.83), t(177)=2.09,p=0.037, 95% CI=[0.14,4.68], Cohen’s d=0.31 (Fig 2A). This indicates that adolescents’ face emotion perception is less sensitive to changes in expression intensities compared to adults, and therefore adolescents are more likely to perceive subtle expressions as neutral or not indicative of increasing emotional intensity. In contrast, adults’ perceptual ability is more finely tuned, enabling them to recognize subtler expressions with only minor observed affective changes³.

Fig 2.

Averaged perceptual uncertainty parameter (σ) based on (A) behavioral response and (B) neural pattern similarity as a function of age. Error bars represent ± SEM. * denotes statistical significance at 95% CI level based on bootstrapping resampling (n=9999).

Consistent with the behavioral findings, an independent-samples t-test on the neural parameter indicated that perceptual uncertainty levels in face-selective regions (see ROI selection) were significantly higher for adolescents (M=56.74, 50=28.88, SD=2.87) than for adults (M=45.31, SD=19.68, SE=2.18), t(175)=3.18,p=0.002, 95% CI=[4.71,18.61], Cohen’s d=0.46 (Fig 2B), suggesting that adolescents’ more similarly represent lower intensity emotional faces than adults. In other words, subtle intensities in facial expression are less finely represented in adolescents at the neural level, and therefore, adolescents need more intense-emotional facial expressions to perceive facial emotion at the neural level, whereas adults perceived more emotionality even from subtle facial expressions. The bandwidth parameter from the neural data showed a modest yet significant positive correlation with the bandwidth from the behavioral data across participants, $\underset{\_}{r}(179)=0.195$, p=0.022, 95% CI=[0.02, 0.32]. Additional correlation analyses separately for each group, however, did not reveal significant relationships between the behavioral and neural parameters (for teens, p=.792; for adults, p=.139), implying that there was no explicit convergent evidence between behavioral and neural measures within each age group. Lastly, we estimated the bandwidth metric with the amygdala voxels identified from the same ROI contrast, but no age-related differences in the bandwidth parameter emerged, t(180)=1.46, p=270, 95% CT=[0.56, 25.51].

Discussion

Youth have less experience with emotion as a function of age, with some difficulty recognizing and interpreting others’ facial affect, particularly when expressed in subtle or ambiguous ways. The current study was designed to provide a more nuanced analysis of the perceptual differences between adults and youth by comparing internal representations of emotional faces between the age groups. We provide evidence for age-related differences in perceptual representation of emotional faces by fitting the behavioral and neural data to a psychophysics model of emotion perception.

Our work expands upon previous findings (Wiggins et al., 2015) that the ventral stream system may provide a neural index for the ability of perceiving ambiguous facial expressions and maturation of fine-tuned internal perceptual representations for ambiguity in developing youth. More specifically, our results suggest that adolescents show less perceptual sensitivity in the ventral stream system to perceive changes of facial expression, such that adolescents’ perceptual representation for neutral expression is broader than adults. In other words, adolescents have more uncertainty for emotion than adults, leading adolescents to be more likely to perceive subtle facial expressions of emotion as non-emotional, consistent with previous interpretations of the broader curve in the perception model (Calder et al., 2008; Clifford et al., 2015; Mareschal et al., 2013)

Our work provides support that adolescents perceive ambiguous facial affect as being less emotionally salient than their adult counterparts. However, some limitations exist in our design. Given our recruitment of teens and adults specifically, we are not able to speak to how this facial affect processing develops in early childhood, a critical developmental period for learning about affect (Sroufe, 1997). Additionally, given the cross-sectional design, we are unable to examine these changes in vivo. Future work is necessary to study the progression of affect-processing across development, as this will provide greater insight into how these processes are shaped normatively and how they may be impacted by life experiences. Another constraint on generalizability may be the lack of attention paid towards how adolescents express emotions relative to adults (McLaughlin, Garrad, & Somerville, 2015). It may be that adolescents are generally less expressive, perhaps complicating the interpretations of the current study. Finally, we did not address individual differences, such as anxiety (e.g,. Bishop et al., 2015), or physiological reactivity (e.g., McManis, Bradley, Berg, Cuthbert, & Lang, 2001), which may play an important modulatory role in affect processing. For example, social-emotional competency may moderate how well one perceives or attributes emotional states particularly in subtle or ambiguous presentations (e.g., Mayer & Geher, 1996). Future examinations should test whether individual differences, such as arousal reactivity, moderate perceptual differences in developing populations, or if the same individual differences that predict adult perception can be linked to adolescents’ affect perception. Lastly, we used relatively short ISIs between faces (range: 3.17–4.54s, based on gaussian distribution), which may be suboptimal compared to fully- stimulus-spaced design with long SOAs (e.g, 12s). Thus, it is possible the neural estimation for each trial may be less specific and more influenced by a close trial as model fitting for neural data was not as high as behavior-based-values. Although, we found that there is a consistency in findings across age groups for both neural and behavioral data as we hypothesized, future work is necessary to have more optimal parameters in the design to increase the specificity of neural estimation.

Extending previous work (Batty & Taylor, 2006; Gross & Ballif, 1991; McClure, 2000; Thomas et al., 2001; Wiggins et al., 2015), the present study adds to our knowledge about age-related differences in facial emotion perception. Our findings provide direct evidence that internal perceptual criteria in representing others’ emotional expressions is still developing during adolescence. Compared with adults, adolescents exhibited a broader bandwidth for neutral face perception indicating that they may be less sensitive to subtle features of emotional expression and are more likely to perceive others’ subtle expressions as non-emotional or neutral.