Skip to main content
PMC home page
Social Cognitive and Affective Neuroscience logoLink to Social Cognitive and Affective Neuroscience
. 2010 May 11;6(3):321–329. doi: 10.1093/scan/nsq039

Stop looking angry and smile, please: start and stop of the very same facial expression differentially activate threat- and reward-related brain networks

Andreas Mühlberger 1,*,✉, Matthias J Wieser 1,*, Antje BM Gerdes 1, Monika CM Frey 1, Peter Weyers 1, Paul Pauli 1,2
PMCID: PMC3110429  PMID: 20460301

Abstract

Static pictures of emotional facial expressions have been found to activate brain structures involved in the processing of emotional stimuli. However, in everyday live, emotional expressions are changing rapidly, and the processing of the onset vs the offset of the very same emotional expression might rely on different brain networks, presumably leading to different behavioral and physiological reactions (e.g. approach or avoidance). Using functional magnetic resonance imaging, this was examined by presenting video clips depicting onsets and offsets of happy and angry facial expressions. Subjective valence and threat ratings clearly depended on the direction of change. Blood oxygen level dependent responses indicate both reward- and threat-related activations for the offset of angry expressions. Comparing onsets and offsets, angry offsets were associated with stronger ventral striatum activation than angry onsets. Additionally, the offset of happy and the onset of angry expressions showed strong common activity in the lateral orbitofrontal cortex bilaterally, the left amygdala and the left insula, whereas the onset of happy and the offset of angry expressions induced significant activation in the left dorsal striatum. In sum, the results confirm different activity in motivation-related brain areas in response to the onset and offset of the same emotional expression and highlight the importance of temporal characteristics of facial expressions for social communication.

Keywords: Functional magnetic resonance imaging, face, facial expression, striatum, amygdala

Human daily behavior is guided by social signals. Among the most important signals in non-verbal communication are facial expressions, because from these expressions we can glean information about the internal emotional state and the intentions of others. Consequently, these signals have a deep impact on our motivational systems. In our everyday live, facial expressions are normally changing continuously (e.g. from a smile to a frown). Such changes carry diverging information about the emotional status of the observed person and are of motivational relevance for the observer. The neural correlates of such changes in facial expressions, however, are not very well investigated since until now research has been mostly concerned with static facial expressions.

Static pictures of emotional facial expressions have consistently been found to activate brain structures involved in emotional processing (for review see Phan et al., 2002). Pictures of happy faces have been shown to activate reward-related areas like the basal ganglia, including the ventral striatum and the putamen (Morris et al., 1996; Phillips et al., 1998; Whalen et al., 1998). In contrast, threatening faces are expected to activate the fight–flight system, and accordingly it has been found that fearful as well as angry faces activate the amygdala (Morris et al., 1996; Whalen et al., 1998) and other limbic areas like the insula (e.g. Schienle et al., 2002).

However, amygdala responses have also been confirmed for happy, sad and neutral facial expressions (e.g. Yang et al., 2002; Fitzgerald et al., 2006); thus, the amgydala might respond to meaningful stimuli in general (see Phan et al., 2002; Sergerie et al., 2008). Similarly, Adolphs (2008) states that the amygdala might be responsible for detecting salience and biological relevance which is thought to serve a fundamental role to facilitate biologically relevant learning (Whalen, 2009). Additionally, meta-analytic approaches have pointed to hemispheric differences in amygdala reactivity to emotional stimuli. While the right amygdala seems to be indifferent with respect to the emotional valence of the stimulus and to be activated by emotionally salient (facial) stimuli in general, the left amygdala seems to be involved in ‘more detailed, cognitive perceptual emotional information processing’ (p. 266), to depend on the emotional valence of the stimulus and to be only responsive to negative facial expressions (Fusar-Poli et al., 2009).

Comparing static and dynamic facial expressions, it was shown that dynamic facial expressions elicit enhanced ratings of arousal (Weyers et al., 2006; Sato and Yoshikawa, 2007), enhanced facial mimicry (Weyers et al., 2006; Sato et al., 2008), and stronger amygdala activity (e.g. LaBar et al., 2003; Sato et al., 2004; van der Gaag et al., 2007). Yet, all these studies have examined the dynamic onset of emotional facial expressions; thus, the effect of the dynamic offset of an emotional facial expression is still unknown. In addition, the onset of a positive facial expression may have comparable effects to the offset of a negative expression on threat- or reward-associated systems and vice versa because in each case both dynamic changes carry comparable emotional information. Specifically, the onset of a happy expression and the offset of an angry expression share a positive valence, while the offset of a happy expression and the onset of an angry expression share a negative valence.

So far, effects of the onset and the offset of stimuli have been examined in the context of pain processing and conditioning. Even if this research did not involve dynamic changes during stimulus presentation, but a discrete onset and offset of stimuli, important information might be drawn for the onset and offset of emotional facial expressions. The first line of research was based on opponent process theories (Solomon, 1980) and investigated temporal aspects of processing painful stimuli under the assumption that pain onset and pain offset are opposite ends of the reward-aversion continuum (Becerra and Borsook, 2008). Indeed, brain imaging studies indicate that pain onset (aversive) elicits deactivation in the ventral striatum (nucleus accumbens), whereas pain offset (rewarding) elicits its activation (Becerra and Borsook, 2008). Interestingly, recent research confirmed that physical pain and pleasure include the same networks as social pain and pleasure (Lieberman, 2009).

The other line of research investigated neural structures associated with conditioned stimuli. After fear conditioning, activity of the amygdala and the insula have been confirmed (e.g. LaBar et al., 1998; LeDoux, 2000; Maren, 2001; Craig, 2002, for review see Delgado et al., 2008). Contrary to fear conditioning, in appetitive conditioning the striatum (e.g. nucleus caudate, putamen, nucleus accumbens) has been found to be a key structure activated by the conditioned stimulus associated with either a primary (e.g. O'Doherty et al., 2001; Pagnoni et al., 2002; Gottfried et al., 2003) or a secondary reinforcer (e.g. Delgado et al., 2000; Kirsch et al., 2003; Delgado et al., 2008, for review see Delgado et al., 2008). However, recent studies indicate that the dichotomy of fear conditioning as primarily relying on the amygdala and appetitive conditioning as primarily relying on the striatum is too simple. On the contrary, it seems that the striatum and the amygdala are functionally interconnected and both involved in appetitive and aversive conditioning (e.g. Setlow et al., 2002; Seymour et al., 2007, for review see Delgado et al., 2008). Accordingly, it has been proposed that the lateral nucleus, the central nucleus, and the basal nucleus of the amygdala are associated with the processing of aversive conditioning, while the basolateral nucleus is associated with appetitive conditioning (Delgado et al., 2008). Regarding the striatum both the ventral and the dorsal striatum have been associated with a prediction error in appetitive conditioning. Additionally, the ventral striatum, mostly the nucleus accumbens, but even the dorsal striatum has been associated with aversive conditioning. However, Delgado et al. (2008) stated that up to now the different contributions of striatum subdivisions (e.g. dorsal/ventral) for aversive conditioning have not been clearly identified. Nevertheless, some authors suggest that the ventral striatum might be solely involved in the processing of appetitive stimuli (Knutson et al., 2001a).

The orbitofrontal cortex (OFC) has been investigated in the processing of reinforcement, often applying operant conditioning paradigms. It has been found that medial regions of the OFC are more associated with the reward value of reinforcers, and lateral regions of the OFC are more associated with the evaluation of punishers (e.g. Kringelbach, 2005). However, it has been assumed that not only the hedonic valence of the reinforcement, but also the choice of a response is important for the activation of the OFC (e.g. Kringelbach, 2005). A recent study by Elliot et al. (2010) now shows that the medial OFC response is specific to positive outcomes and is independent of the behavioral significance (Elliot et al., 2010). Additionally, they confirmed that the lateral OFC is activated by negative outcomes. Whereas emotional facial expressions have been confirmed to act as reinforcers using the presentation of static pictures (e.g. Vrticka et al., 2008; Spreckelmeyer, 2009), to our knowledge no study has investigated changes of these expressions as reinforcers. To investigate the reinforcing value of changes of facial expressions, however, onset and offset of the same expression have to be differentiated. The onset of a smile might be a reinforcer activating the reward circuitry, whereas the offset of a smile might serve as a threat or punishing stimulus consequently activating systems involved in fear. In contrast, the onset of an angry facial expression might predict threat or punishment and thus activate systems involved in fear, whereas the offset of an angry expression might serve as negative reinforcer: watching an angry person calming down might activate reward circuitries.

According to the reasoning outlined above, the aim of the current study was to investigate whether the dynamic onset and the dynamic offset of angry and happy facial expressions each have opposite effects on reward-aversion related neural structures. We focused on a subset of brain areas involved in processing of pleasure and pain. Our hypotheses were that all emotional expressions will activate the amygdala bilaterally, but that angry facial expressions will activate the left amygdala as well as the insula, whereas happy facial expressions will activate regions within the ventral striatum. More importantly, we further assumed that this effect will be modulated by the temporal change of the expressions (onset, offset). The dynamic onset of happy facial expressions and the dynamic offset of angry facial expressions should lead to stronger activations of reward-related regions within the striatum (all sub-regions, but especially the ventral striatum, e.g. the nucleus accumbens) and in the medial OFC, whereas the onset of angry facial expressions and the offset of happy facial expressions should lead to stronger lateral OFC, left amygdala, and insula activations. To examine these questions we measured neural activation by functional magnetic resonance imaging (fMRI) in response to short video clips displaying the onset and offset of angry and happy facial expressions.

METHODS

Participants

After giving informed written consent 16 volunteers (seven females; age M = 22.9, s.d. = 2.3) participated in the pilot study, and 18 in the fMRI study (eight females, age M = 22.4, s.d. = 2.6). None of the participants reported any psychiatric or neurological history; furthermore, they reported normal or corrected-to-normal vision. Additionally, all participants in the fMRI study were right-handed (Edinburgh Handedness Inventory, Oldfield, 1971). The study was approved by the local ethics committee.

Design, stimuli and task

The present studies employed a 2 × 2 factorial design with the factors emotion (angry vs happy expression) and dynamic (onset vs offset of emotion). The stimuli consisted of computer-generated (virtual) male faces (Poser, Curious Labs, Santa Cruz, CA) depicting the onset or offset of either angry or happy facial expressions (see examples in Figure 1). Virtual faces provide useful research tools because they allow complete control over the facial expression and its dynamics. Compared to photos of real persons they were found to elicit similar brain responses (e.g. Moser et al., 2007; Mühlberger et al., 2009).

Fig. 1.

Fig. 1

Open in a new tab

Examples of the four experimental conditions. In each column, one experimental condition is depicted with the start and the end frame of the video clip.

We used clips of 1.67 s length and a frame rate of 30 frames per second. Altogether, six male characters were created, which differed in hairstyle and hair color. For the onset conditions the clips linearly changed from a neutral into a happy or an angry face. For the offset conditions, the clips linearly turned from the fully expressed emotion into a neutral face (Figure 1). Linear changes were used to secure the same physical features for the presentation of onset and offset. Each character displayed all four experimental conditions.

In the pilot study, participants were asked to rate the clips with respect to valence, arousal, and threat. The clips were presented in a randomized order, and after each clip, three different 9-point Likert rating scales were presented (valence: 1 = very unpleasant to 9 = very pleasant; arousal: 1 = very arousing to 9 = not arousing at all; threat: 1 = not threatening at all to 9 = very threatening).

The fMRI experiment was conducted as a block design (8 min in total). The four experimental conditions were repeated twice per session resulting in eight experimental blocks. Two baseline blocks were inserted (fixation cross for 25 s), after the first and the seventh experimental block, while the order of experimental blocks was randomized across participants. Each block contained 12 stimuli each displayed for 1670 ms followed by a 750 ms inter-stimulus interval. Participants were instructed to watch the clips or the fixation cross without any further task. The stimuli were presented on a light gray background via MRI-compatible goggles (VisuaStim; Magnetic Resonance Technologies, Northridge, CA) using Presentation (Version 9.13, Neurobehavioral Systems, Albany, CA, USA).

Image acquisition

Functional and structural MRI was performed with a Siemens 1.5 T MRI whole body scanner (SIEMENS Avanto) using a standard head coil and a custom-built head holder. Functional images were obtained using a T2*-weighted single-shot gradient echoplanar imaging (EPI) sequence (TR: 2500 ms, TE: 30 ms, 90° flip angle, FOV: 200 mm, matrix: 64 × 64, voxel size: 3.1 × 3.1 × 3 mm3). Each EPI volume contained 25 axial slices (thickness 5 mm, 1 mm gap), acquired in interleaved order, covering the whole brain. The orientation of the axial slices was parallel to the AC–PC line. Each session contained 190 functional images. The first nine volumes of each session were discarded to allow for T1 equilibration. In addition, a high-resolution T1-weighted magnetization-prepared rapid gradient-echo imaging (MP-RAGE) 3D MRI sequence was obtained from each subject (TR: 2250 ms, TE: 3.93 ms, 98 flip angle, FOV: 256 mm, matrix: 256 × 256, voxel size: 1 × 1 × 1 mm3).

Image preprocessing and analyses

Data were analyzed by using Statistical Parametric Mapping software (SPM5; Wellcome Department of Imaging Neuroscience, London, UK) implemented in Matlab 7.0 (Mathworks Inc., Sherborn, MA, USA). Functional images were slice-time corrected and realigned by an affine registration. The mean functional image was subsequently normalized to the Montreal Neurological Institute (MNI) single-subject template (Evans et al., 1992). Normalization parameters were then applied to the functional images and coregistered to the T1-image. Images were re-sampled at a 2 × 2 × 2 mm3 voxel size and spatially smoothed using an 8 mm full width half maximum Gaussian kernel, and temporally filtered with a high-pass filter (cutoff 128 s). Each experimental condition was modeled using a boxcar reference vector convolved with a canonical hemodynamic response function. Parameter estimates were subsequently calculated for each voxel using weighted least squares to provide maximum likelihood estimates based on the non-sphericity assumption of the data in order to get identical and independently distributed error terms. Realignment parameters for each session were included to account for residual movement related variance. Parameter estimation was corrected for temporal autocorrelations using a first-order autoregressive model. For each subject, main effects were computed by applying appropriate baseline contrasts (simple effects).

Afterwards these first-level individual contrasts were fed into a second-level group analysis using an ANOVA (factors: emotion, dynamic, and blocking factor subject), thus employing a random effects model (Penny and Holmes, 2003). The subject factor models subject constants that absorb much of the inter-subject variability present in most imaging data, which in turn leads to more sensitivity for the experimental effects (including group differences). First, we analyzed simple effects of each condition vs baseline and simple effects between onset and offset of both emotional expressions. Second, because we were especially interested in investigating the interaction between the factors emotion and dynamic, and in formally testing differences in simple effects, the following interaction contrast was calculated [(angry-onset – angry-offset) vs (happy-offset – happy-onset)]. Third, common activations of angry-offset and happy-onset and of angry-onset and happy-offset were identified by means of a conjunction analysis using the Global Null, a less conservative approach testing the combined null hypothesis for both contrasts of interest. Thus, a significant conjunction does not mean all the contrasts are individually significant, but that the contrasts are consistently high and jointly significant (cf. Friston et al., 2005). These analyses were performed using the small volume correction of SPM 5, a height threshold of P < 0.05, family-wise correction (FWE), and an extent threshold of k = 5 contiguous voxels. Resulting activation peaks were superimposed on standard high-resolution anatomical images. For a priori expected activations, ROI analyses were carried out in the amygdala, the insula, the ventral and dorsal part of the striatum (nucleus accumbens and head of caudate, body of caudate and putamen, respectively), and the medial and lateral part of the OFC based on masks from the WFU Pick Atlas (Maldjian et al., 2003) as implemented in SPM5.

RESULTS

Pilot study

Valence, arousal and threat ratings are shown in Table 1. Concerning valence, a significant main effect of emotion, F(1,15) = 64.7, P < 0.01, η2p = 0.81, and a significant Emotion × Dynamic interaction, F(1,15) = 148.3, P < 0.01, η2p = 0.91, was observed. Post hoc comparisons revealed that the onset of angry expressions was rated as more negative than the offset of angry expressions, t(15) = 8.52, P < 0.01, and the onset of happy expressions was rated as more positive than the offset of happy expressions, t(15) = 11.28, P < 0.01. Additionally, the onset of happy expressions was rated as more positive than the onset of angry expressions, t(15) = 11.26, P < 0.01, while the rating of offset between happy and angry expressions did not differ, P > 0.16.

Table 1.

Arousal, valence and threat ratings for angry and happy expressions as well as their onset and offset

Variable
 Valence
 Arousal
 Threat
Expression Dynamic M s.d. M s.d. M s.d.
Angry Onset 2.90 1.12 4.51 1.15 6.51 1.05
Offset 5.05 0.71 5.52 1.06 3.36 1.57
Happy Onset 7.48 1.03 4.49 0.97 1.92 0.96
Offset 4.71 0.42 5.30 1.10 3.84 1.98
Open in a new tab

For arousal ratings there was only a main effect of dynamic, F(1,15) = 64.7, P < 0.01, η2p = 0.81, indicating that the onsets of both emotional expressions were rated as more arousing than the offsets.

Concerning threat ratings, there were significant main effects for emotion, F(1,15) = 34.1, P < 0.01, η2p = 0.70, and dynamic, F(1,15) = 5.4, P = 0.03, η2p = 0.27, furthermore, the interaction Emotion × Dynamic was significant, F(1,15) = 108.1, P < 0.01, η2p = 0.88. The onset of an angry expression was perceived as more threatening than the offset of an angry expression, t(15) = 9.3, P < 0.01, whereas the offset of a happy expression was perceived as more threatening than the onset of a happy expression, t(15) = 5.15, P < 0.01. Additionally, the onset of happy expressions was perceived as less threatening than the onset of angry expressions, t(15) = 14.25, p < 0.01, while the threat ratings of the offsets did not differ between happy and angry expressions, P > 0.36.

In sum, valence and threat ratings revealed that the onset of an angry facial expression is perceived as more negative and threatening than the offset of the same emotion, whereas for happy facial expressions, we observed the opposite effect.

fMRI study

The regions of interest (ROI) analyses (amygdala, insula, ventral and dorsal striatum, OFC) for the simple effects revealed for the onset of a happy facial expression significant activations (P < 0.05, FWE-corrected) in the right putamen (see Table 2 for all main effects of the ROI analyses). For the offset of happy faces, significant activations were found in the left and right amygdala, the left and right insula, the left and right putamen, and the left and right lateral OFC. Both onset and offset of angry facial expressions revealed significant activations in the right and left amygdala, the right and left insula, and the left and right lateral OFC. The offset of angry facial expressions furthermore showed significant activations in the left and right head of caudate, the right and left body of the caudate, the right and left nucleus accumbens, and the left and right medial OFC.

Table 2.

Significant activations as revealed by ROI analysis for the main effects

Contrast x y z Z k Brain region
Angry onset
−22 0 −16 3.17 156 Amygdala L
22 0 −18 3.00 130 Amygdala R
−34 24 −4 4.38 409 Insula L
36 24 −4 3.97 164 Insula R
24 22 0 3.35 184 Putamen R
−48 34 −4 5.34 342 Lateral OFC L
54 34 −4 7.23 302 Lateral OFC R
Angry offset
−20 −8 −16 3.42 139 Amygdala L
20 0 −12 3.71 107 Amygdala R
−30 24 −4 4.53 1089 Insula L
32 22 −2 4.57 854 Insula R
−6 10 2 3.61 170 Caudate head L
14 14 6 3.84 187 Caudate head R
−6 10 −2 2.84 40 N. accumbens L
16 16 −4 3.42 123 N. accumbens R
−34 −16 −8 4.14 743 Putamen L
24 22 0 5.07 863 Putamen R
−50 22 −4 4.37 99 Lateral OFC L
54 30 −4 5.36 133 Lateral OFC R
−24 12 −14 4.17 23 Medial OFC L
14 42 −10 2.95 17 Medial OFC R
Happy onset
−34 −16 −8 3.53 169 Putamen L
Happy offset
−20 2 −16 2.84 117 Amygdala L
32 4 −20 3.09 89 Amygdala R
−34 24 −2 4.64 717 Insula L
36 28 −4 4.54 279 Insula R
−32 −14 −6 4.49 476 Putamen L
24 6 2 3.43 454 Putamen R
−52 24 −4 5.24 554 Lateral OFC L
52 32 −4 5.08 308 Lateral OFC R
Open in a new tab

Alpha = 0.05 (FWE-corrected) for ROI analyses with a minimum cluster size of k = 5. L = left, R = right hemisphere, OFC = orbitofrontal cortex. aThe cluster with the largest number of significant voxels within each region is reported. Coordinates x, y and z are given in MNI space.

The direct comparison of the onset and offset of happy facial expressions was not significant. In contrast, the same comparison for angry facial expressions revealed significantly stronger activation to the offset compared to the onset in the left and right head of the caudate (x = −12, y = 22, z = 0; Z = 3.54; FWE P = 0.005; 150 voxel; and x = 12, y = 16, z = 0; Z = 3.17; FWE P = 0.015; 141 voxel, respectively), and the right nucleus accumbens (x = 14, y = 16, z = −2; Z = 2.83; FWE P = 0.049; 94 voxel). To a minor extent using a lowered statistical threshold also stronger activation in right medial OFC was detected (x = 16, y = 48, z = −4; Z = 3.35; P < 0.001, uncorrected; 8 voxel).

To directly evaluate our hypothesis of common mechanisms of the onset of happy and the offset of angry faces and vice versa we conducted conjunction analyses for the related conditions (e.g. happy-offset and angry-onset) in the specified ROIs. The analysis of common activations between the onset of an angry expression and the offset of a happy expression revealed overlapping activation patterns in the left amygdala and left insula (Figure 2A and B). In contrast, the onset of a happy facial expression and the offset of an angry facial expression significantly activated the left putamen (Figure 2C). Thus, these results partially confirm our hypotheses.

Fig. 2.

Fig. 2

Open in a new tab

Statistical parametric maps for the conjunction analyses of angry-onset and happy-offset, and of angry-offset and happy-onset, respectively. (A) Common activation of angry-onset and happy-offset in the left amygdala, x = −20, y = 2, z = −16; Z = 2.84; P (FWE-corrected) = 0.035; k = 114 voxel. (B) Common activation of angry-onset and happy-offset in the left insula, (x = −34, y = 24, z = −4; Z = 4.38; P (FWE-corrected) = 0.002; k = 264 voxel. (C) Common activation of angry-offset and happy-onset in the left putamen, (x = −34, y = −16, z = −8; Z = 2.84; P (FWE-corrected) = 0.018; k = 163 voxel.

Regarding the OFC, the conjunction analysis of angry-onset and happy-offset revealed strong common activations in bilateral clusters in the lateral OFC (left: x = −52, y = 36, z = −4; Z = 6.72; P (FWE-corrected) < 0.001; k = 524 voxel; right: x = 52, y = 32, z = −4; Z = 7.69; P (FWE-corrected) < 0.001; k = 436 voxel). A small cluster in the left OFC was also found to be commonly activated by happy-onset and angry-offset (x = −46, y = 28, z = −4; Z = 4.64; P (FWE-corrected) = 0.002; k = 26 voxel).

To further evaluate the idea that the onset of happy and the offset of angry faces compared to the onset of angry and the offset of happy faces would activate reward-associated brain regions stronger than threat-associated brain regions, respectively, ROI interaction analyses were computed. The ROI analysis of the interaction between onset and offset of angry vs happy facial expressions [(angry-offset vs angry-onset) vs (happy-onset vs happy-offset)] partially supported our hypothesis of an interaction within the striatum (left head of caudate, but not in the nucleus accumbens, see Figure 3). However, there was no significant activation for the reversed interaction term [(angry-onset vs angry-offset) vs (happy-offset vs happy-onset)].

Fig. 3.

Fig. 3

Open in a new tab

Statistical parametric maps for the Emotion × Dynamic interaction analysis, revealing selective activation in the left caudate head, x = −14, y = 22, z = 6; Z = 3.29; P (FWE-corrected) = 0.011; k = 56 voxel.

DISCUSSION

To our knowledge, this is the first study investigating the neural effects of the dynamic onset and offset of emotional facial expressions. Subjective ratings confirmed that valence and threat of the investigated facial expressions depend on their temporal change. While the onset of happy facial expressions was rated as highly positive and least threatening, the onset of angry facial expressions was rated as highly negative and highly threatening. Interestingly, the offsets of happy and angry expressions were rated as neutral in valence and only medium threatening, respectively, thus eliciting attenuated responses compared to the onset of the same facial expressions. Arousal ratings for all presentations were at an average level.

Interestingly, angry offset, angry onset, and happy offset activated the amygdala bilaterally. This result confirms the assumption that the amygdala is involved in the processing of emotionally salient stimuli in general (Adolphs, 2008). The non-significant results for the happy onset might mean that this condition is emotionally less salient. This interpretation is supported by the fact that also no significant activation of the ventral striatum was found in response to the onset of happy facial expressions. Another explanation could be that the happy faces have not been recognized as ‘real’ Duchenne smiles, but as social (fake) smiles. However, the valence ratings of our pilot study and the fact that fake smiles are characterized by their relatively short onsets (<0.4 s, see Krumhuber, 2007), seem to make this explanation unlikely. Furthermore, as humans are keen to detect cheaters (i.e. humans pretending to be friendly to take personal advantage), one would assume even higher salience and thus higher amgydala activations in response to fake smiles (Cosmides, 2005). Recently, it has been argued that the amygdala responds to facial expressions on the basis of their predictive value as a CS (Whalen, 2009). According to this line of reasoning, the amygdala will always track the stimulus, which shows the most promise for learning. Probably, in the context of angry offsets and onsets, and also happy offsets, the onset of a happy face is therefore least interesting and salient.

The insula was activated bilaterally by the same conditions as the amygdala. The activation of the insula in the angry onset and the happy offset condition is easily explained by the role in the (social) pain network (Lieberman, 2009) and is in line with earlier results investigating facial expressions. Nevertheless, the activation in the angry offset condition as well might point at the insula having a similar functionality as the amygdala in detecting salience for biological learning.

Furthermore, not only the insula, but also the lateral OFC was activated bilaterally by the same conditions as the amygdala. The activation of the lateral OFC in the angry onset and the happy offset condition is easily explained by the role of the processing of negative reinforcers (Elliot et al., 2010). Nevertheless, the lateral OFC activation in the angry offset condition as well might indicate that this condition has positive and negative aspects or point at the notion that activations in this region are arousal-related, which has been found in passive viewing of emotional pictures and emotional imagery (Sabatinelli et al., 2007; Costa et al., 2010). Furthermore, the distinction between lateral and medial OFC as distinct regions for either processing punishment or reward cues is mainly derived from totally different paradigms.

Regarding activations within the ventral striatum, it is very interesting that only the angry offset condition was associated with significant activations (nucleus accumbens and head of the caudate bilaterally). Additionally, the anger offset condition was the only one to activate the medial OFC. These results indicate that the offset of an angry facial expression is an important trigger to activate reward- or pleasure-related structures. Thus, while an angry face is supposed to be a prototypical stimulus activating the human fear system (Öhman, 1986), the offset of this ‘prepared’ signal seems to be likewise of evolutionary significance to activate the reward system.

A direct comparison confirmed a higher activation for the offset than the onset of angry expressions in the head of the caudate bilaterally and in the right nucleus accumbens. Therefore and according to our hypothesis, the offset of angry facial expressions activates reward-associated brain areas within the ventral striatum; these results confirm our findings in the main contrasts that the offset of an angry face is of evolutionary significance by presumably providing information about the end of a threat. Remarkably, these findings are in line with results of a recent study investigating general emotional stimuli that appeared either to approach (‘onset’) or to recede (‘offset’) from the observer (Mühlberger et al., 2008). Interestingly, approaching unpleasant pictures elicited enhanced startle responses compared to receding unpleasant pictures. Thus, depending on movement the same unpleasant picture triggered differential neural responses.

Furthermore, conjunction analyses confirmed strong common activations in the left amygdala and the left insula as well as the lateral OFC induced by the onset of an angry expression and the offset of a happy expression. This fits to the assumption that the left amygdala is specifically involved in the processing of negative information (Fusar-Poli et al., 2009), and we might assume that such laterality is even true for the insula. However this is speculative because such insula laterality has not yet been described in the literature. Additionally, the lateral OFC activation fits well to the literature on the role of this region in the evaluation of unpleasant stimuli (e.g. Kringelbach, 2005; Elliot et al., 2010). Furthermore, the onset of happy expressions and the offset of angry expressions induced significant activations in the left dorsal striatum (putamen). However, the role of the putamen in reward-processing is still unclear although it is known that this area is involved in motivational processes and has also been related to reward prediction (Breiter et al., 2001; Knutson et al., 2001b; Bjork et al., 2004). It has to be noted however that putamen activations were also present in our study for the main effects of angry onset and happy offset, pointing at a more general involvement of the putamen in the processing of dynamic facial expressions.

The interaction analysis additionally revealed that the offset of angry facial expressions and the onset of happy facial expressions are associated with activity in the left ventral striatum (head of caudate). This last result confirms the reward association of angry offset, since we speculate that the angry offset contributed more to this effect than the happy onset (see main contrasts).

To sum up, this study is the first that investigated onset and offset of happy and angry facial expressions. Our results confirm that the angry offset condition is associated with reward-processing (e.g. nucleus accumbens, head of caudate, and to a fewer extent, medial OFC). Furthermore, angry-onset as well as happy-offset were found to activate regions involved in the processing of negative facial expressions (e.g. left amygdala). This finding confirms the laterality of the amygdala in processing of negative facial stimuli.

Our results also relate to both the literature on pain processing and on conditioning. There is evidence that onset and offset of a painful (unconditioned) stimulus activate threat- and reward-related brain areas in a different way. First, our findings fit well with the results of Becerra and Borsook (2008) indicating that pain offset activates the nucleus accumbens whereas pain onset leads to its deactivation; similarly we found that the offset of angry facial expressions elicits a bilateral nucleus accumbens activation. We conclude that both the offset of an angry facial expression and the offset of pain signify relief and therefore are associated with the activity in reward-processing brain areas. Second, there are hints from the conditioning literature that the onset and the offset of an unconditioned stimulus have opposite effects on approach/avoidance behavior (e.g. Tanimoto et al., 2004). Our observation of a different activation of reward- and threat-related brain areas by the onset and the offset of an emotional face are well in line with these results. Furthermore, the left amygdala and left insula responses to the onset of angry faces and the offset of happy faces perfectly fit with the assumption that these dynamic facial expressions both act as (conditioned) aversive stimuli.

Some caveats, however, have to be discussed. First, because we used only male faces, we have to limit the results for those stimuli. Additionally, it might well be that there are gender differences in the processing of faces, which could not be fully addressed in the present study. However, exploratory analyses did not reveal any gender differences in the processing of the faces in our sample. Further research might include female and male faces and a larger sample to scrutinize gender-related differences. Next, it could be that clips depicting emotion offsets lead to a bidirectional activation of motivational systems. This points to a methodological challenge, namely that the beginning of the stimulus (clip) is always a confounding variable. Dynamically presented offsets have to start with the full emotional expressions, as for example is the case for our angry offset clips which start with a full blown angry expression; thus we cannot disentangle the blood oxygen level dependent responses to the onset of the clip and to the dynamic change. Furthermore, we can not disentangle whether the observed effects relate to the specific event of an emotion appearing or disappearing or to a more sustained psychological state created through the repeated onset or offset of the emotional facial expressions. Further research should try to disentangle these two components by using event-related measures. A further limitation of our study is that only two emotions were investigated. Further research should investigate whether only the offset of angry facial expressions activates the reward system, or whether this is a more general phenomenon apparent for other negative emotional facial expressions as well. Last, although subjective ratings confirmed the positive value of our happy onset clips, brain activations in response to these stimuli were not as strong as expected. It might be that the happy expression could be improved by enhancing the amount of the smile or by confirming that the smile is seen as a Duchenne smile.

In sum, our results clearly indicate that the onset and offset of the same emotional facial expression are not only rated as being differently threatening, but that they also activate reward- and threat-related brain areas in a different way. As a consequence it seems necessary to take the temporal dynamics of facial expressions into account to understand their impact on social interaction as well as learning in social contexts.

Acknowledgments

The authors thank Felix Breuer for his excellent assistance in fMRI data acquisition. This research was supported by German Research Foundation (FOR 605 and SFB TRR 58 B01).

REFERENCES

  1. Adolphs R. Fear, faces, and the human amygdala. Current Opinion in Neurobiology. 2008;18:166–72. doi: 10.1016/j.conb.2008.06.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Becerra L, Borsook D. Signal valence in the nucleus accumbens to pain onset and offset. European Journal of Pain. 2008;12:866–9. doi: 10.1016/j.ejpain.2007.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bjork JM, Knutson B, Fong GW, Caggiano DM, Bennett SM, Hommer DW. Incentive-elicited brain activation in adolescents: similarities and differences from young adults. The Journal of Neuroscience. 2004;24:1793–802. doi: 10.1523/JNEUROSCI.4862-03.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron. 2001;30:619–39. doi: 10.1016/s0896-6273(01)00303-8. [DOI] [PubMed] [Google Scholar]
  5. Cosmides L, Tooby J. Neurocognitive adaptions designed for social exchange. In: Buss DM, editor. The handbook of evolutionary psychology. Hoboken, NJ: John Wiley & Sons Inc; 2005. pp. 584–627. [Google Scholar]
  6. Costa VD, Lang PJ, Sabatinelli D, Bradley MM, Versace F. Emotional imagery: assessing pleasure and arousal in the brain's reward circuitry. Human Brain Mapping. 2010;31:1446–57. doi: 10.1002/hbm.20948. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Craig AD. How do you feel? Interoception: the sense of the physiological condition of the body. Nature Review Neuroscience. 2002;3:655–66. doi: 10.1038/nrn894. [DOI] [PubMed] [Google Scholar]
  8. Delgado MR, Gillis MM, Phelps EA. Regulating the expectation of reward via cognitive strategies. Nature Neuroscience. 2008;11:880–1. doi: 10.1038/nn.2141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA. Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology. 2000;84:3072–7. doi: 10.1152/jn.2000.84.6.3072. [DOI] [PubMed] [Google Scholar]
  10. Elliot R, Agnew Z, Deakin J.FW. Hedonic and informational functions of the human orbitofrontal cortex. Cerebral Cortex. 2010;20:198–204. doi: 10.1093/cercor/bhp092. [DOI] [PubMed] [Google Scholar]
  11. Evans AC, Marrett S, Neelin P, et al. Anatomical mapping of functional activation in stereotactic coordinate space. Neuroimage. 1992;1:43–53. doi: 10.1016/1053-8119(92)90006-9. [DOI] [PubMed] [Google Scholar]
  12. Fitzgerald DA, Angstadt M, Jelsone LM, Nathan PJ, Phan KL. Beyond threat: amygdala reactivity across multiple expressions of facial affect. Neuroimage. 2006;30:1441–8. doi: 10.1016/j.neuroimage.2005.11.003. [DOI] [PubMed] [Google Scholar]
  13. Friston KJ, Penny WD, Glaser DE. Conjunction revisited. Neuroimage. 2005;25:661–7. doi: 10.1016/j.neuroimage.2005.01.013. [DOI] [PubMed] [Google Scholar]
  14. Fusar-Poli P, Placentino A, Carletti F, et al. Laterality effect on emotional faces processing: ALE meta-analysis of evidence. Neuroscience Letters. 2009;452:262–7. doi: 10.1016/j.neulet.2009.01.065. [DOI] [PubMed] [Google Scholar]
  15. Gottfried JA, O'Doherty J, Dolan RJ. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science. 2003;301:1104–7. doi: 10.1126/science.1087919. [DOI] [PubMed] [Google Scholar]
  16. Kirsch P, Schienle A, Stark R, et al. Anticipation of reward in a nonaversive differential conditioning paradigm and the brain reward system: an event-related fMRI study. Neuroimage. 2003;20:1086–95. doi: 10.1016/S1053-8119(03)00381-1. [DOI] [PubMed] [Google Scholar]
  17. Knutson B, Adams CM, Fong GW, Hommer D. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. Journal of Neuroscience. 2001a;21:RC159. doi: 10.1523/JNEUROSCI.21-16-j0002.2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Knutson B, Fong GW, Adams CM, Varner JL, Hommer D. Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport. 2001b;12:3683–7. doi: 10.1097/00001756-200112040-00016. [DOI] [PubMed] [Google Scholar]
  19. Kringelbach ML. The human orbitofrontal cortex: Linking reward to hedonic experience. Nature Reviews Neuroscience. 2005;6:691–702. doi: 10.1038/nrn1747. [DOI] [PubMed] [Google Scholar]
  20. Krumhuber E, Manstead A.SR, Kappas A. Temporal aspects of facial displays in person and expression perception: the effects of smile dynamics, head-tilt, gender. Journal of Nonverbal Behaviour. 2007;31:39–56. [Google Scholar]
  21. LaBar KS, Crupain MJ, Voyvodic JT, McCarthy G. Dynamic perception of facial affect and identity in the human brain. Cerebral Cortex. 2003;13:1023–33. doi: 10.1093/cercor/13.10.1023. [DOI] [PubMed] [Google Scholar]
  22. LaBar KS, Gatenby JC, Gore JC, LeDoux JE, Phelps EA. Human amygdala activation during conditioned fear acquisition and extinction: a mixed-trial fMRI study. Neuron. 1998;20:937–45. doi: 10.1016/s0896-6273(00)80475-4. [DOI] [PubMed] [Google Scholar]
  23. LeDoux JE. Emotion circuits in the brain. Annual Review of Neuroscience. 2000;23:155–84. doi: 10.1146/annurev.neuro.23.1.155. [DOI] [PubMed] [Google Scholar]
  24. Lieberman MD, Eisenberger NI. Pains and pleasures of social life. Science. 2009;323:890–1. doi: 10.1126/science.1170008. [DOI] [PubMed] [Google Scholar]
  25. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. Neuroimage. 2003;19:1233–9. doi: 10.1016/s1053-8119(03)00169-1. [DOI] [PubMed] [Google Scholar]
  26. Maren S. Neurobiology of Pavlovian fear conditioning. Annual Review of Neuroscience. 2001;24:897–931. doi: 10.1146/annurev.neuro.24.1.897. [DOI] [PubMed] [Google Scholar]
  27. Morris JS, Frith CD, Perrett DI, et al. A differential neural response in the human amygdala to fearful and happy facial expressions. Nature. 1996;383:812–5. doi: 10.1038/383812a0. [DOI] [PubMed] [Google Scholar]
  28. Moser E, Derntl B, Robinson S, Fink B, Gur RC, Grammer K. Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods. 2007;161:126–33. doi: 10.1016/j.jneumeth.2006.10.016. [DOI] [PubMed] [Google Scholar]
  29. Mühlberger A, Neumann R, Wieser MJ, Pauli P. The impact of changes in spatial distance on emotional responses. Emotion. 2008;8:192–8. doi: 10.1037/1528-3542.8.2.192. [DOI] [PubMed] [Google Scholar]
  30. Mühlberger A, Wieser MJ, Herrmann MJ, Weyers P, Tröger C, Pauli P. Early cortical processing of natural and artificial emotional faces differs between lower and higher social anxious persons. Journal of Neural Transmission. 2009;116:735–46. doi: 10.1007/s00702-008-0108-6. [DOI] [PubMed] [Google Scholar]
  31. O'Doherty J, Kringelbach ML, Rolls ET, Hornak J, Andrews C. Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neuroscience. 2001;4:95–102. doi: 10.1038/82959. [DOI] [PubMed] [Google Scholar]
  32. Öhman A. Face the beast and fear the face: animal and social fears as prototypes for evolutionary analyses of emotion. Psychophysiology. 1986;23:123–45. doi: 10.1111/j.1469-8986.1986.tb00608.x. [DOI] [PubMed] [Google Scholar]
  33. Pagnoni G, Zink CF, Montague PR, Berns GS. Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci. 2002;5:97–8. doi: 10.1038/nn802. [DOI] [PubMed] [Google Scholar]
  34. Penny WD, Holmes AP. Random effects analysis. In: Frackowiak RS, Friston KJ, Frith CD, editors. Human Brain Function. London, UK: Academic Press; 2003. pp. 843–50. [Google Scholar]
  35. Phan KL, Wager T, Taylor SF, Liberzon I. Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage. 2002;16:331–48. doi: 10.1006/nimg.2002.1087. [DOI] [PubMed] [Google Scholar]
  36. Phillips ML, Young AW, Scott SK, et al. Neural responses to facial and vocal expressions of fear and disgust. Proceedings of the Royal Society B: Biological Science. 1998;265:1809–17. doi: 10.1098/rspb.1998.0506. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Sabatinelli D, Bradley MM, Lang PJ, Costa VD, Versace F. Pleasure rather than salience activates human nucleus accumbens and medial prefrontal cortex. Journal of Neurophysiology. 2007;98:1374–9. doi: 10.1152/jn.00230.2007. [DOI] [PubMed] [Google Scholar]
  38. Sato W, Fujimura T, Suzuki N. Enhanced facial EMG activity in response to dynamic facial expressions. International Journal of Psychophysiology. 2008;70:70–4. doi: 10.1016/j.ijpsycho.2008.06.001. [DOI] [PubMed] [Google Scholar]
  39. Sato W, Kochiyama T, Yoshikawa S, Naito E, Matsumura M. Enhanced neural activity in response to dynamic facial expressions of emotion: an fMRI study. Brain Research: Cognitive Brain Research. 2004;20:81–91. doi: 10.1016/j.cogbrainres.2004.01.008. [DOI] [PubMed] [Google Scholar]
  40. Sato W, Yoshikawa S. Enhanced experience of emotional arousal in response to dynamic facial expressions. Journal of Nonverbal Behaviour. 2007;31:119–35. [Google Scholar]
  41. Schienle A, Stark R, Walter B, et al. The insula is not specifically involved in disgust processing: an fMRI study. Neuroreport. 2002;13:2023–6. doi: 10.1097/00001756-200211150-00006. [DOI] [PubMed] [Google Scholar]
  42. Sergerie K, Chochol C, Armony JL. The role of the amygdala in emotional processing: a quantitative meta-analysis of functional neuroimaging studies. Neuroscience and Biobehavioral Reviews. 2008;32:811–30. doi: 10.1016/j.neubiorev.2007.12.002. [DOI] [PubMed] [Google Scholar]
  43. Setlow B, Holland PC, Gallagher M. Disconnection of the basolateral amygdala complex and nucleus accumbens impairs appetitive pavlovian second-order conditioned responses. Behavioral Neuroscience. 2002;116:267–75. doi: 10.1037//0735-7044.116.2.267. [DOI] [PubMed] [Google Scholar]
  44. Seymour B, Daw N, Dayan P, Singer T, Dolan R. Differential encoding of losses and gains in the human striatum. Journal of Neuroscience. 2007;27:4826–31. doi: 10.1523/JNEUROSCI.0400-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Solomon RL. The opponent-process theory of acquired motivation. The cost of pleasure and the benefit of pain. American Psychologist. 1980;35:691–712. doi: 10.1037//0003-066x.35.8.691. [DOI] [PubMed] [Google Scholar]
  46. Spreckelmeyer KN, Krach S, Kohls G, et al. Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women. Social Cognitive and Affective Neuroscience. 2009;4:158–65. doi: 10.1093/scan/nsn051. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Tanimoto H, Heisenberg M, Gerber B. Experimental psychology: event timing turns punishment to reward. Nature. 2004;430:983. doi: 10.1038/430983a. [DOI] [PubMed] [Google Scholar]
  48. van der Gaag C, Minderaa RB, Keysers C. The BOLD signal in the amygdala does not differentiate between dynamic facial expressions. Social Cognitive and Affective Neuroscience. 2007;2:93–103. doi: 10.1093/scan/nsm002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Vrticka P, Andersson F, Grandjean D, Sander D, Vuilleumier P. Individual attachment style modulates human amygdala and striatum activation during social appraisal. Plos One. 2008;3:e2868. doi: 10.1371/journal.pone.0002868. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Weyers P, Mühlberger A, Hefele C, Pauli P. Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology. 2006;43:450–3. doi: 10.1111/j.1469-8986.2006.00451.x. [DOI] [PubMed] [Google Scholar]
  51. Whalen P, Davis FC, Oler JA, Kim MJ, Netta M. Human amygdala responses to facial expressions of emotion. In: Whalen P.J.P.EA, editor. The human amygdala. New York, NY: Guilford Press; 2009. pp. 265–88. [Google Scholar]
  52. Whalen PJ, Rauch SL, Etcoff NL, McInerney SC, Lee MB, Jenike MA. Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. Journal of Neuroscience. 1998;18:411–8. doi: 10.1523/JNEUROSCI.18-01-00411.1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Yang TT, Menon V, Eliez S, et al. Amygdalar activation associated with positive and negative facial expressions. Neuroreport. 2002;13:1737–41. doi: 10.1097/00001756-200210070-00009. [DOI] [PubMed] [Google Scholar]

Articles from Social Cognitive and Affective Neuroscience are provided here courtesy of Oxford University Press

RESOURCES