Shared Multisensory Stimulation between Faces Facilitates Recognition of Fearful Facial Expressions.

Abstract

Embodied simulation accounts of emotion recognition claim that we vicariously activate somatosensory representations to simulate, and eventually understand, how others feel. Interestingly, Mirror-Touch Synaesthetes, who experience touch when observing others being touched, show both enhanced somatosensory simulation and superior recognition of emotional facial expressions. We employed synchronous visuotactile stimulation to experimentally induce a similar experience of ‘mirror touch’ in non-synesthetic participants. Seeing someone else’s face being touched at the same time as one’s own face results in the ‘enfacement illusion’, which has been previously shown to blur self-other boundaries. We demonstrate that the enfacement illusion also facilitates emotion recognition, and, importantly, this facilitatory effect is specific to fearful facial expressions. Shared synchronous multisensory experiences may experimentally facilitate somatosensory simulation mechanisms involved in the recognition of fearful emotional expressions.

An important aspect of successful social interaction is the ability to detect and understand the emotional states of others. Often, the only immediate source of information available to us regarding the emotional state of another individual is from their facial expression. Embodied simulation theories of emotion recognition argue that we reactivate the body states associated with the observed emotional expression in ourselves in order to recognize the emotional expression of others (e.g. Niedenthal, 2007). This mechanism relies on the activation of somatosensory, visceral and motoric representations to simulate how another person feels when making a facial expression. This resonant mapping between the bodies of self and other may give us a unique experiential understanding of the other’s emotions (Gallese, Keysers, & Rizzolatti, 2004).

This theory has received support from a wide range of different studies (for a review, see Goldman & Sripada, 2005). For example, reliable ‘mirror-like’ activation of somatosensory and premotor areas are observed both when observing emotional facial expressions of others as well as when producing the same expressions oneself (Carr, Iacoboni, Dubeau, Mazziotta & Lenzi, 2003). In addition, patients with damage to right primary and secondary somatosensory cortices were significantly impaired at recognizing emotional facial expressions (Adolphs, Damasio, Tranel, Cooper, & Damasio, 2000). In accordance with these findings, transcranial magnetic stimulation (TMS) delivered to the face region of the right somatosensory cortex disrupted recognition of emotional facial expressions (Pitcher, Garrido, Walsh, & Duchaine, 2008).

Further support for the somatosensory simulation account comes from a recent study into individuals with a rare type of synaesthesia known as ‘mirror-touch’ synaesthesia (MTS: Banissy, Cohen Kadosh, Maus, Walsh, & Ward, 2009). These individuals report touch sensations on their own bodies when they observe other people being touched. Congruent with their reported experiences, MTS individuals show increased vicarious activation of sensorimotor areas when observing other-experienced tactile stimulation (Blakemore, Bristow, Bird, Frith, & Ward, 2005). Intriguingly, MTS individuals also show superior emotion recognition when compared to non-synesthetic participants (Banissy et al. 2011). Taken together, these findings suggest that the way MTS individuals share others’ somatosensory experiences, also known as ‘interpersonal somatosensory resonance’, serves to facilitate their recognition of others’ emotional expressions. A prediction that stems out from research into MTS is that when this somatosensory resonance between the bodies of self and other is enhanced, emotion recognition is facilitated. The way in which we map others’ bodily experiences onto our own bodily experiences may be an important part of successful emotion recognition.

It has recently been shown that the relationship between our own bodies and the bodies of others is flexible, dynamic and sensitive to experimental manipulations (Tsakiris, 2010). One method used to manipulate self-other bodily representations is the ‘enfacement illusion’. A participant watches a video showing the face of an unfamiliar other being stroked with a cotton bud on the cheek, whilst the participant receives identical stroking on their own cheek in synchrony with the touch they see. Synchronous, but not asynchronous, shared visuotactile stimulation between the participant’s own face and another person’s face produces a measurable bias in self-face recognition (Tsakiris, 2008; Sforza, Bufalari, Haggard, & Aglioti, 2010). Participants accept images with a larger percentage of the other’s facial features blended with their own as their own face (Tajadura, Grehl & Tsakiris, in press), and they also rate the other’s face as more similar to theirs. Interestingly, synchronous visuotactile stimulation applied to the face also influences sociocognitive processes such as conformity behaviour and self-other fusion (Paladino, Mazzurega, Pavani, & Schubert, 2010). Overall, this ‘enfacement’ leads the participant to incorporate features of the other’s face into their self-face representation, decreasing the perceptual distance between self and other. By manipulating the way in which we represent the bodies of self and other, we may be able to modulate interpersonal somatosensory resonance, and thus modulate emotion recognition. However, no study has yet investigated this intriguing possibility. We predicted that enfacement may temporarily enhance somatosensory resonance with the individual with whom tactile stimulation is shared. Given that somatosensory simulation plays an important role in emotion recognition, shared visuotactile stimulation in the enfacement illusion should facilitate emotion recognition via a similar mechanism.

To test this hypothesis, we measured emotion recognition before and after a period of shared visuotactile stimulation between individuals. Accuracy of participants’ emotion recognition was compared before and after synchronous or asynchronous multisensory stimulation, to assess whether enfacement of the other had modulated their ability to recognize the emotions of that other. A ‘No-Touch’ control condition was also included, in which no multisensory stimulation was delivered, to allow us to assess the effect of mere visual familiarity with the other’s face. This was deemed an important consideration, as some studies have shown that expression judgements can be modulated by face identity and familiarity (e.g. Baudouin, Sansone & Tiberghien, 2000; Herba et al., 2008; Schweinberger & Soukup, 1998; Zhang & Parmley, 2011). Controlling for the effect of the No-Touch condition on emotion recognition allowed us to investigate the true effects of both synchronous and asynchronous stimulation, over and above that of mere exposure to the face.

Method

Participants

Fifteen Caucasian female volunteers (Mean Age = 19.8 years, SD = 0.9) participated in the study. All participants gave their signed, informed consent and were paid for participation.

Stimuli

Preparation of emotion stimuli

Three Caucasian female models (Mean Age = 19.5 years, SD = 1.3) were photographed making fearful, disgusted, happy and neutral facial expressions, after a brief instruction period for each expression using a mirror. The photographic set-up was kept constant between models. Each model had their hair tied back, removed distracting makeup and jewellery, and wore a black gown to cover any visible clothing on their shoulders. Fearful, happy and disgusted emotions were chosen as each has clear empirical evidence to suggest that their recognition is at least partially ‘embodied’ (e.g. Hennenlotter et al., 2005; Oberman et al., 2007; Pitcher et al., 2008; but also see Hussey & Safford, 2009, for discussion). Anger and Surprise were also initially included in a pilot study1, but excluded from final stimulus selection due to poor recognition levels.

Three sets of stimuli per model were generated by morphing an emotional expression with the model’s neutral facial expression. This provided us with three sets of morphed photos ranging from 0% emotional strength (the neutral expression) to 100% emotional strength (the pure emotional expression) for each of the happy, fearful and disgusted expressions. Seven strengths of each emotion were selected from these sets, comprising 20%, 30%, 40%, 50%, 60%, 70% and 80%¹. This provided us with a range of stimulus difficulty.

Preparation of multisensory stimuli

To develop the stimuli used for the multisensory stimulation, videos were also recorded of each model. Their right cheek was stroked with a cotton bud every three seconds for two minutes, whilst they looked straight at the camera with a neutral expression. The models also sat for a two-minute video without tactile stimulation, for use in the ‘No-Touch’ control condition.

Tasks

Emotion recognition task

The task consisted of 42 trials, each of which displayed one of three emotional expressions (fear, happiness or disgust), at one of seven intensity levels (20%, 30%, 40%, 50%, 60%, 70%, or 80%), shown by the same model. Each stimulus was presented twice during the emotion recognition session. Order of trials was randomized. Participants had to choose which of the three emotions (fear, disgust or happiness) was displayed, making a 3-alternative forced choice via labelled keys on the keyboard. The emotional stimulus remained on the screen until the participant’s response, whereby a 500ms inter-stimulus interval was presented before the next trial. Before the task, participants completed six practice trials with a different model to learn the position of the response keys and trial structure. The task was carried out twice per experimental block, once before and once after a period of multisensory stimulation.

Multisensory stimulation

For the sessions of multisensory stimulation, participants sat in front of a computer screen and were asked to keep their head and face as still as possible. In the Synchronous condition, they then watched a 2-minute video of the model being stroked on their face with a cotton bud every three seconds, during which the experimenter stroked the participant’s face in a specularly-congruent location in synchrony with the touch seen in the video. In the Asynchronous condition, the participant’s face was instead stroked in asynchrony with the touch in the video. In the No-Touch condition, participants watched the model in the video for two minutes, but neither the model nor the participant received touch on their faces. The identity of the model in the video always matched the identity of the model featuring in the pre- and post-stimulation emotion recognition tasks.

Enfacement questions

Participants were asked a series of 10 questions about their experience during the videos, to measure the extent to which they experienced ‘enfacement’. Questions were taken from Tajadura et al. (in press). Each question required a response ranging from ‘strongly disagree’ (−3) to ‘strongly agree’ (+3) on a 7-point Likert scale. Responses were given via the keyboard.

Procedure

Each participant performed the experiment in three blocks. Each block featured a different female model. All blocks began with a pre-stimulation emotion recognition task. Participants then viewed a two-minute video of the model, during which they either received a period of multisensory stimulation (Synchronous and Asynchronous conditions) or did not (No-Touch condition). They then completed a post-stimulation emotion recognition task featuring the same model. This procedure is illustrated in Figure 1A. Each block ended with the participants answering the ten enfacement questions. The blocks differed from each other in two ways. First, they differed in the type of stimulation (i.e. synchronous, asynchronous or no touch) given during the multisensory stimulation phase. Second, each block featured a different model. Order of blocks, and the identity of the model in each block, was counterbalanced between participants.

Figure 1.

Panel A: The design of the experiment. Each participant completed three experimental blocks, for Synchronous, Asynchronous and No-Touch conditions. Panel B: Graph showing the differential effects of synchronous vs. asynchronous visuotactile stimulation on accuracy of fear recognition, measured using D’. Error bars reflect standard error of the mean, and asterisk indicates p-value < .05, two-tailed.

Results

To analyse accuracy of emotion recognition, a signal detection analysis was used to calculate D’ scores for each emotion, condition and model from the number of ‘hits’ (i.e. emotion was correctly identified) and ‘false alarms’, following the 3-AFC Signal Detection procedure (see Frijters, 1979 for discussion; also Dessirier & Mahoney, 1998; Stewart-Knox et al., 2005; Stillman, 1993). Descriptive data detailing performance before the stimulation manipulation can be found in Table 1.

Table 1. Table showing details of ‘baseline’ performance on the Emotion Recognition Task before stimulation was delivered. Mean D’ scores, proportion of ‘hit’ responses, and proportion of ‘false alarm’ responses are given, for each emotional expression in each experimental condition.

condition	emotional expression	D’ scores M(SD)	proportion of ‘hits’	proportion of ‘false alarms’
synchronous	fear	2.79 (0.47)	.86	.05
	happiness	3.19 (0.20)	.96	.07
	disgust	2.88 (0.34)	.83	.04
asynchronous	fear	2.75 (0.46)	.86	.04
	happiness	3.17 (0.21)	.95	.05
	disgust	2.87 (0.31)	.84	.05
no-touch	fear	2.75 (0.60)	.84	.07
	happiness	3.10 (0.27)	.92	.05
	disgust	2.89 (0.44)	.84	.05

First, responses to trials in the No-Touch condition were analysed to assess the effect of familiarity on emotion recognition. Scores from the No-Touch condition were entered into a 2(Testing Phase: pre-stimulation vs. post-stimulation) × 3(Emotion: fear vs. happiness vs. disgust) repeated-measures ANOVA, which revealed a main effect of Emotion, F(1,14) = 4.98, p = .035, whereby D’ scores on happiness trials (M = 3.12, SD = 0.25) were numerically higher than both disgust trials, M = 2.91 (SD = 0.46), and fear trials, M = 2.82 (SD = 0.54). The difference with fear trials reached significance, p = .043. Performance on fear and disgust trials did not significantly differ from each other, p = .469. There were no main effects of Testing Phase, F(1,14) = 1.09, p =.314, nor a Testing Phase * Emotion interaction, F(1,14) = 0.52, p = .485. This analysis thus confirmed that in the absence of stimulation delivered to the participant’s face, there was no significant increase in D’ scores between testing phases. This suggests that the effect of familiarity on emotion recognition was negligible at the group level. However, to ensure than individual differences in the effect of familiarity and practice on emotion recognition were properly controlled for, D’ change between pre- and post-stimulation sessions for the No-Touch condition was used as a covariate in all further analyses, The covariate (D’change_noto) was obtained by calculating the mean change in D’ between pre- and post-stimulation across all emotions.

The main analysis was carried out using a repeated-measures ANCOVA with Emotion (fear vs. happiness vs. disgust), Condition (Synchronous vs. Asynchronous) and (Testing Phase: pre-stimulation vs. post-stimulation) as factors, and D’change_noto as a covariate. All assumptions for ANCOVA were met. The results revealed a main effect of Emotion, F(1,13) = 16.48, p = .001, whereby D’ scores on happiness trials, M = 2.91 (SD = 0.24), were significantly higher than scores on fear trials, M = 2.58 (SD = 0.31), t(14) = 6.33, p < .001, and scores on disgust trials, M = 2.57 (SD = 0.23), t(14) = 4.88, p < .001. D’ scores for fear and disgust trials did not significantly differ, t(14) = 0.13, p = .902. There was no main effect of Testing Phase, F(1,13) = 1.85, p = .197, nor of Condition, F(1,13) = 1.17, p = .300. There was, however, a significant three-way interaction between Testing Phase, Condition and Emotion, F(1,13) = 5.56, p = .035. No other interactions were significant.

To investigate this interaction, a repeated-measures ANCOVA with Testing Phase and Condition entered as factors was performed on the D’ scores for each emotion separately. A significant interaction between Testing Phase and Condition was found only for responses to fear trials, F(1,13) = 4.65, p = .050, and not for happiness or disgust trials, p-values > .05. No other main effects or interactions were significant. Paired t-tests revealed that there was no significant difference between conditions in D’ scores for fear in the pre-stimulation phase, t(14) = 0.37, p = .717. However, in the post-stimulation phase, D’ was significantly higher after synchronous stimulation than asynchronous stimulation, t(14) = 2.63, p = .020 (see Figure 1B).

To investigate this effect further, separate analyses were performed on the proportion of ‘hits’ and the proportion of ‘false alarms’ for fear responses. An ANCOVA on false alarms with Testing Phase and Condition as factors revealed no main effects or interaction, p-values > .05. However, the same ANCOVA repeated on ‘hit’ responses revealed a significant interaction between Testing Phase and Condition, F(1,13) = 5.11, p = .042, whereby the proportion of ‘hit’ responses to fearful expressions after synchronous stimulation, M = .91 (SD = .10), was significantly higher than after asynchronous stimulation, M = .81 (SD = .14), t(14) = 2.43, p = .029. There was no significant difference between conditions in the pre- stimulation responses, t(14) < .001, p > .99.

We then investigated whether the effect of synchronous stimulation on sensitivity to fearful expressions was modulated by the intensity of the expression. We repeated the final ANCOVA on D’ scores, with Intensity (20%, vs. 30% vs. 40% vs. 50% vs. 60% vs. 70% vs. 80%) included as an additional factor. There was an expected main effect of Intensity, F(1,5) = 38.62, p < .001, which reflected a positive linear relationship between increasing intensity and increasing D’ scores, r(7) = .89, p = .007 (Spearman’s coefficient), from a mean D’ of 1.80 at 20% intensity to a mean of 3.05 at 80% intensity. However, no interactions between Intensity, Condition and/or Testing Phase were significant, all p-values > .05.

Finally, responses to the Enfacement questions were analysed to check that our synchronous stimulation was successful in eliciting a stronger subjective experience of enfacement than our asynchronous condition. Responses to each question, which were given on a scale from -3 (strongly disagree) to +3 (strongly agree), were averaged to provide an ‘enfacement score’ in which higher values indicated a stronger experience of enfacement. Synchronous multisensory stimulation induced a significantly higher enfacement score than did asynchronous stimulation (see Table 2).

Table 2. Table showing mean Likert responses to each Enfacement question ranging from −3 (strongly disagree) to +3 (strongly agree), for Synchronous and Asynchronous conditions. Independent t-tests give statistical significance of differences in responses between conditions.

Enfacement question	Synchronous M(SD)	Asynchronous M(SD)	t(14)	P
“I felt like the other’s face was my face”	0.47 (2.17)	−1.93 (1.44)	3.80	.002**
“It seemed like the other’s face belonged to me”	−0.33 (2.22)	−1.53 (1.85)	1.77	.098
“It seemed like I was looking at my own mirror reflection”	0.40 (2.02)	−1.13 (2.07)	2.30	.038*
“It seemed like the other’s face began to resemble my own face”	0.60 (1.68)	−1.33 (1.76)	3.28	.005**
“It seemed like my own face began to resemble the other person’s face”	0.53 (2.07)	−0.93 (2.05)	2.09	.056
“It seemed like my own face was out of my control”	0.27 (1.83)	−1.67 (2.29)	4.49	.001**
“It seemed like the experience of my face was less vivid than normal”	−0.20 (2.04)	−0.93 (1.75)	0.99	.338
“I felt that I was imitating the other person”	0.93 (1.67)	−0.20 (2.57)	1.48	.162
“The touch I felt was caused by the cotton bud touching the other’s face”	0.47 (2.13)	−1.47 (2.39)	3.19	.006**
“The touch I saw on the other’s face was caused by the cotton bud touching my own face”	−0.27 (2.25)	−1.33 (2.26)	1.30	.214
Total Mean Response	0.29 (1.48)	−1.25 (1.67)	2.93	.011*

^*p < .05.

^**p < .01.

Discussion

Embodied accounts of emotion recognition argue that we recognize emotional facial expressions via a process of somatosensory simulation. In support of this argument, individuals with Mirror-Touch Synaesthesia (MTS), who experience touch when they see others being touched, have both facilitated somatosensory simulation and enhanced emotion recognition. In the Enfacement Illusion, we delivered touch to non-synesthetic participants’ faces whilst they viewed another individual’s face being touched, eliciting an experience of ‘shared touch’ that bears some similarity to MTS. This study tested whether this shared synchronous visuotactile stimulation, previously shown to manipulate self-other boundaries, could facilitate the recognition of emotional facial expressions. Synchronous, but not asynchronous, visuotactile stimulation did indeed facilitate emotion recognition, and this effect was specific to expressions of fear. We suggest that synchronous visuotactile stimulation may temporarily enhance somatosensory resonance with the ‘enfaced’ other, facilitating fearful emotion recognition via a similar mechanism to that enhanced in MTS.

To ensure that any effect of visuotactile stimulation on emotion recognition was attributable to the synchrony of stimulation rather than visuotactile stimulation per se, we compared the effect of synchronous stimulation to the effect of asynchronous stimulation. Thus, only facilitation of emotion recognition after synchronous stimulation could be taken as evidence supporting our prediction. We also controlled for general facilitatory effects of familiarity of the other’s face on emotion recognition, by including a third condition in which the participant viewed the model’s face for two minutes, but in which no tactile stimulation was delivered. In this way, we could ensure that the facilitatory effect of synchronous stimulation was over and above any facilitation due to mere familiarity with the face of the model, or of effects of practice.

Our results demonstrated a facilitatory effect of synchronous stimulation on recognition of fear only, and did not affect recognition of happiness or disgust. This finding was not due to differences in difficulty between the emotions. Although sensitivity to happy facial expressions was found to be significantly higher than sensitivity to fear or disgust (replicating several other studies; e.g. Kirita & Endo, 1995; Kirouac & Doré, 1983), sensitivity to fear and disgust did not differ, and thus differences in task difficulty for each emotion is unable to explain the specificity of the effect to fear. Similarly, the effect was not modulated by the intensity of the stimuli; multisensory stimulation modulated fear recognition equally at both weak and strong intensities of expression. Several previous studies suggest that the recognition of fearful expressions is more heavily reliant on somatosensory representations than are other emotions. For example, Pourtois and colleagues (Pourtois et al., 2004) demonstrated that TMS over the right somatosensory cortex disrupted recognition of fearful, but not happy, facial expressions, and suggested that the recognition of fearful faces might require a stronger activation of somatosensory representations than the recognition of other emotions. Cardini, Bertini, Serino, and Ladavas (2012) found that the ‘visual remapping of touch’ (VRT: an effect whereby the perception of touch on one’s face is modulated by seeing another’s face being touched) is enhanced when the other’s face is showing fear, but not when showing happiness or anger. As VRT is thought to reflect a process of somatosensory resonance, this suggests that the perception of fearful facial expressions may be more strongly reliant on this process than other emotions. Indeed, in order to recognise fear in others, it may be evolutionarily adaptive to rely heavily on somatosensory mechanisms. Expressions of fear communicate biologically salient information about threats in the environment, and thus rapid, automatic recognition and somatic behavioural preparation may be particularly useful in the recognition of fear signals over other emotions.

The experience of synchronous shared touch provided by the enfacement illusion has previously been reported to experimentally change the way in which we represent self and other, affecting several aspects of social cognition. Importantly, we now show that these effects of shared multisensory stimulation can be extended to the domain of emotion processing. By modulating self-other boundaries, shared multisensory stimulation may temporarily enhance somatosensory resonance with the other, and facilitate the interpretation of their fearful emotional expressions. We speculate that enhanced somatosensory resonance with other’s fearful expressions may give a rapid, salient ‘input’ to an embodied simulation mechanism (Goldman & Sripada, 2005), allowing fast and accurate identification of fear signals. More broadly, our results suggest that the way we represent the relationships between the bodies of self and other is an important factor in the somatosensory simulation of emotions, and furthermore, demonstrate that such a process is sensitive to multisensory intervention.