Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence

Abstract

Historically, research on emotion perception has focused on facial expressions, and findings from this modality have come to dominate our thinking about other modalities. Here, we examine emotion perception through a wider lens by comparing facial with vocal and tactile processing. We review stimulus characteristics and ensuing behavioral and brain responses, and show that audition and touch do not simply duplicate visual mechanisms. Each modality provides a distinct input channel and engages partly non-overlapping neuroanatomical systems with different processing specializations (e.g., specific emotions versus affect). Moreover, processing of signals across the different modalities converges, first into multi- and later into amodal representations that enable holistic emotion judgments.

Nonverbal Emotions – Moving from a Unimodal to a Multimodal Perspective

Emotion perception plays a ubiquitous role in human interactions and is hence of interest to a range of disciplines, including psychology, psychiatry, and social neuroscience. Yet, its study has been dominated by facial emotions, with other modalities explored less frequently and often anchored within a framework derived from what we know about vision. Here, we take a step back and put facial emotions on a par with vocal and tactile emotions. Like a frightened face, a shaking voice or a cold grip can be meaningfully interpreted as emotional by us. We will first explore these three modalities in terms of signal properties and brain processes underpinning unimodal perception (see Glossary). We will then examine how signals from different channels converge into a holistic understanding of another’s feelings. Throughout, we will address the question whether emotion perception is the same or different across modalities.

Sensory Modalities for Emotion Expression

There is vigorous debate about what exactly individuals can express nonverbally. There are hotly debated open questions about whether people are in an objective sense “expressing emotions” as opposed to engaging in more strategic social communication, whether they express discrete emotions and which ones, and whether their expressions are culturally universal. For practical reasons, we ignore these debates here and simply assume that people do express and perceive what we usually call emotions (Box 1).

Box 1. How Are Emotion Expression and Perception Defined?

Emotional expressions are motivated behaviors co-opted to be perceived as communicative signals, but that may or may not be intended as such. Darwin was the first to propose three causal origins for expressions [85]. One of them involves an immediate benefit to the expressive individual; for example, increasing one’s apparent body size serves to intimidate an opponent. Other expressions, Darwin suggested, effectively communicate the polar opposite; for example, lowering one’s body-frame to signal submission instead of aggression. Lastly, Darwin held that some expressions, such as the trembling in fear, are mere vestigial by-products that may not serve a useful role. Although many details of Darwin’s proposal are debatable, there is evidence that some facial expressions exhibit useful functions (e.g., widening of the eyes to maximize the visual field during fear) [86] and/or to effectively manipulate the behavior of perceivers [87].

Perception, recognition, and categorization are probed with distinct tasks. Emotion perception includes early detection and discrimination, and is often conceptualized as providing input to the richer inferences that we make about people’s internal states and the retrieval of associated conceptual knowledge (“recognition”), the sorting into emotion categories often defined by words in a particular language (“categorization”), as well as our own emotional responses that can be evoked (e.g., empathy) [88]. Building on earlier work suggesting that perceptual representations feed into systems for judging emotions [89], neuroimaging findings point to specific nodes in regions such as the posterior STS that may mediate between early emotion perception and mental state inference [90]. Similarly, regions of right somatosensory cortex within which lesions [20] or brain stimulation [91] can alter multimodal emotion recognition [92,93], show multivoxel activations in fMRI that correlate with self-reported emotional experiences of the perceiver [30]. There are thus rich connections between early perceptual processing, later mental state inference, and emotion induction in the perceiver. Most studies do not distinguish between these stages, an issue that also confounds the interpretation of putative top-down effects of cognition on perception [94]. A fuller picture of the flow of information from perceptual to inferential processes will also benefit from techniques like magnetoencephalography (MEG), which combines good spatial resolution with excellent temporal resolution [95,96].

Facial expressions have been studied in the most detail, possibly because they seem most apparent in everyday life and their controlled presentation in an experimental setting is relatively easy. Facial expressions in humans depend on 17 facial muscle pairs that we share fully with great apes and partially with some other species (Box 2). The neural control of these muscles depends on a network of cortical and subcortical structures with neuroanatomical and functional specialization [1]. For instance, the medial subdivision of the facial nucleus, which moves our ears, is relatively underdeveloped in humans compared to mammals that can move their ears, whereas the lateral subdivision, which moves our mouth, is exceptionally well developed [2].

Box 2. Emotion Expression and Perception Exist in Non-Human Animals.

Following Darwin’s [85] description of homologues in the emotional behaviors of mammals, there are now quantitative methods similar to the human FACS for describing the facial expressions of other species. The application of these methods has revealed expression similarities. For example, humans share the zygomaticus major with other primates and dogs who, like us, use it to retract mouth-corners and expose teeth [97,98]. Moreover, across these species zygomaticus activity together with the dropping of the jaw contribute to a joyful display termed the “play face” (Figure I).

Importantly, species similarities extend to other expressive channels including the voice and touch. Human fearful screams, as an example, share many acoustic properties (e.g., deterministic chaos, high intensity, high pitch) with the screams of other primates and mammals in general [99]. Additionally, in humans as in many other social species, affiliative emotions trigger close physical contact and gentle touch [14].

Species similarities in expression are matched by similarities in perception as evidenced by both behavioral and brain responses. In monkeys encountering a novel object, the concurrent display of positive and negative facial expressions bias approach and avoidance behaviors, respectively [100]. Tree shrews respond differentially to the same call produced with high and low affective intensity [101]. Zebrafish show a faster reduction in fear behaviors and cortisol following stress when a water current stroked their lateral lines during recovery [102].

Corresponding to these perceptual effects are neural systems that bear resemblance to those identified in humans. In nonhuman primates, inferior and superior temporal regions are likely homologues to the human face and voice areas, respectively, and are linked to a network for crossmodal convergence including prefrontal cortex and STS [103,104]. Dogs, like primates, show face sensitivity in their inferior temporal lobes [105]. Moreover, their brains respond differently to neutral (e.g., “such”) and praise words (e.g., “clever”) depending on the speaker’s voice: only a positive word in combination with a positive voice activates reward-related brain regions [106]. Lastly, emotion perception seems relatively lateralized to the right hemisphere in a wide range of species [107]. In sum, available evidence speaks for homology in the neural systems subserving emotion expression and perception, at least across mammals. Nonetheless, figuring out exactly how these signals influence social behaviour will require substantially more ethological research.

Figure I.

Positive facial affect in humans, apes and dogs. Adapted with permission from Schirmer and colleagues (2013) as well as Palagi and Mancini [110].

Quantification of facial muscle movements and their composition into emotional expressions is provided by the Facial Action Coding System (FACS) [3]. Different expressions can be characterized by linear combinations of action units (basic sets of muscle movements), whereby a single unit can participate in multiple emotional expressions (e.g., brow lowerer, Figure 1). Displays that are typically categorized above chance include happiness, sadness, fear, anger and disgust [4]. However, defining accuracy for this as well as other expressive modalities depends on methodological choices, including the posers’ acting ability or expressive disposition, the set of emotion categories to which the stimulus is typically matched in the task, as well as cultural points of reference [5,6]. Although we do not cover it here, emotion perception from body postures is another relevant visual channel, perhaps more important in certain animals like cats and dogs rather than humans, and one which interacts with other modes of expression [7].

Figure 1.

Nonverbal emotion expression. Illustrated are the three modalities explored in this manuscript. (Left) The emotional facial expression differs from the neutral facial expression in action unit 4 (brow lowerer, corrugator supercilii, depressor supercilii), which forms part of anger, fear, and sadness displays. (Middle) The syllable “ah” spoken in a neutral and a happy tone are illustrated by oscillogram (top) and spectrogram (bottom). Vocalizing duration is longer, and pitch (blue) and intensity (yellow) are higher and more varied when the speaker is happy as compared to neutral. (Right) Compared to no-touch interactions, tactile interactions are perceived as more positive and arousing. Images taken from the Social Touch Picture Set [108].

In addition to faces, voices are an important modality for emotion communication. They emerge from the vocal system via complex movements that involve the abdomen, larynx, tongue, as well as cavities along the vocal tract that filter airwaves and produce resonance effects. Voices convey inborn and learned emotion signals (e.g., screams, sobs, laughs) and their sound may be affected by emotion-induced physiological changes of the vocal system including changes in breathing and muscle tone. Vocal expressions in the context of speech are referred to as speech melody (i.e., prosody) and can be dissociated from verbal content [8].

Sound characteristics can be quantified with a range of acoustic parameters including loudness, fundamental frequency (i.e., melody), and voice quality (e.g., roughness, Figure 1). Some of these parameters show a robust relationship with perceived speaker affect. For instance, loudness and fundamental frequency each correlate positively with arousal [9,10]. The relationship between vocal acoustics and specific emotions is a bit more tenuous, but some evidence suggests that listeners rely on voice quality, individually and in combination with other acoustic parameters [11]. Although emotion recognition can be fairly accurate when listeners chose from a limited set of emotion categories, agreement drops significantly as more categories become available. Moreover, fewer emotions can be perceived from the voice than from the face [12].

By contrast to the perception of faces and voices, the perception of touch requires direct physical contact. In the context of benign human interaction, touch is typically an affectively positive stimulus perceived as pleasurable and promoting bonding and trust [13,14]. Contributing to these effects is a special tactile receptor, the C-tactile (CT) afferent, found in non-glabrous skin. CT-afferents fire maximally to skin-temperature touch [15] and to light-pressure stroking of 3 to 10 cm per second [16]. Because their firing rate correlates positively with the subjective pleasure that people experience from touch, they are thought to play a central role in the social and affective aspects of somatosensation [16].

In addition to the intrinsically rewarding aspects of touch, its specific forms can communicate individual emotions. Based on the body part that is being stimulated, the tactile action (e.g., pushing, stroking), its attributes (e.g., force, duration), and the context (e.g., touch from a lover or a stranger) people can infer a range of emotions. For instance, American college students can correctly categorize anger, fear, disgust, love, gratitude, sympathy, happiness, and sadness from such stimuli when presented in a controlled setting [17]. While relatively unexplored, such emotion categorization very likely involves higher-order inferences and culturally acquired knowledge about social rules and metaphors.

Neural Systems for Perceiving Emotions

The neural systems underpinning the perception of emotions have been studied with a range of tasks and techniques. The following review emphasizes approaches that were used frequently and consistently across modalities and, hence, support a systematic comparison between face, voice and touch. With respect to tasks, those contrasting nonverbal expressions with control stimuli (e.g., face-house) as well emotional with neutral expressions were selected as most relevant in our review. With respect to techniques, lesion studies, simple contrasts or pattern classifications of fMRI data, as well as event-related potentials (ERPs) of the EEG were selected.

Facial Emotions

The cortical system for primate face processing encompasses a mosaic of patches in temporal, occipital, parietal, and frontal cortices that collectively process particular features of faces and generate a distributed perceptual representation [18]. Of particular prominence in human fMRI studies is an area in the inferior temporal cortex that is more strongly activated by faces than non-face objects, the fusiform face area (FFA, Figure 2) [19]. The cortical face network is complemented by a key subcortical structure, the amygdala.

Figure 2.

Key brain regions involved in nonverbal emotion processing. Regions typically more active for emotional as compared to neutral stimuli are marked in green, blue, and red for voice, face, and touch, respectively. Regions typically more active for emotional multimodal as compared to unimodal stimulation are marked in beige.

Insight into neural components that might be more specific for judging emotions from faces, rather than subserving face processing in general, has come from several approaches. Earlier lesion studies pointed to regions in right ventro-parietal and posterior-temporal cortices, including the angular and supramarginal gyri, and even components of somatosensory cortex [20]. However, these regions may be more specifically involved in fine-grained inferences about the emotion from faces that depend on a particular embodiment mechanism, such as representing how the emotion would feel through real or imagined mimicry [21]. Neuroimaging revealed that facial emotions modulate amygdala activity as well as activity in other components of the face processing system [22]. Single amygdala neurons respond to facial emotions [23], in particular expressions of the eye region [24]. Whereas cortical face processing [25] and its emotion modulation [26] appear preferentially lateralized to the right hemisphere, the lateralization of subcortical emotion effects is still unclear.

There is some evidence that expressions of different emotions recruit somewhat segregated face processing circuits. This was demonstrated with multi-voxel pattern analysis (MVPA) within FFA and primary visual cortex [27], medial prefrontal cortex (mPFC) [28], and superior temporal sulcus (STS) [29]. Additionally, an intriguing recent study found that multivoxel decoding of individual emotions could be achieved from somatosensory cortex. The accuracy of such decoding showed a somatotopic pattern: for example, the perception of fear, which is distinguished by wide eyes, could be decoded best from the region of somatosensory cortex that represents the eyes [30]. Case studies of rare lesion patients have provided evidence for emotion differentiation in subcortical structures as well. For example, recognition of fear and disgust can be impaired by lesions to the amygdala [31,32] and the basal ganglia [33], respectively. However, neither of these findings is well corroborated by fMRI studies and neither clearly separates perception from other post-perceptual biases.

Complementing the excellent anatomical resolution of fMRI are techniques, such as the EEG, that yield excellent temporal resolution. ERPs, which are derived from the EEG by time-locked averaging in a particular task, are characterized by a series of positive and negative deflections of which some are modulated by emotion. Of particular relevance in the context of faces is the N170, a negativity peaking about 170 ms following stimulus onset over temporal electrodes that is specific to faces and for which sources lie in the FFA [34]. This negativity [35], but also a later positive potential that is nonspecific for faces [36], are greater to emotional as compared to neutral expressions, especially if they are highly arousing (Figure 3). Furthermore, some electrophysiological responses to facial emotions occur intracranially with latencies barely exceeding 100 ms even in traditionally “high-level” regions such as prefrontal cortex [37], arguing for multiple streams of visual processing that contribute to emotion perception.

Figure 3.

ERP correlates of facial and vocal emotion processing. (Left) The facial emotion effect occurs for a negative deflection termed the N170 [109]. (Right) The vocal emotion effect occurs for a positive deflection termed the P200 [57]. Relevant components are marked by a gray box. (Middle) The scalp topography of the ERP peak maxima are illustrated on the cartoon head.

In particular, there is a long-standing debate about the possibility that the amygdala receives coarse (low spatial frequency) information about emotional faces through a rapid route via the superior colliculus [38]. One recent intracranial recording study found evidence supporting this idea from field potentials recorded from the amygdala of neurosurgical patients [39], even though action potentials of amygdala neurons to faces have very long latencies inconsistent with this view. Some information about facial emotions may thus be represented already very early in the visual system, and may bias subsequent processing [40].

Vocal Emotions

Similar to the better known visual system, the auditory system comprises two pathways originating from different sections within the respective sensory nucleus of the thalamus, the medial geniculate thalamus. One pathway, referred to as lemniscal, has a strong preference for auditory signals, is tonotopically organized, relays fast signal aspects and targets primary auditory cortex (BA41, core). The second, non-lemniscal pathway is multimodal, weakly tonotopic, prefers slow signal aspects, and targets predominantly secondary auditory cortex (BA42, belt and parabelt) [41].

Voices recruit both lemniscal and non-lemniscal pathways [41]. When compared with non-vocal sounds, they activate bilateral secondary auditory cortex together with surrounding superior temporal cortex more strongly [42,43, Figure 2]. These mid-temporal regions are often referred to as “voice areas” by analogy to the FFA. Because they are also recruited by other complex sounds (e.g., music, environmental sounds), they may not be voice-specific. As for faces [44], some of the differential responses to vocalizations may simply result from them being more relevant, practiced, or attention capturing [45].

Emotional vocalizations produce greater activation in voice areas than do neutral vocalizations and this difference is relatively lateralized to the right hemisphere [46,47]. Additionally, there are reports of amygdala lesions impairing vocal emotion recognition and altering associated cortical processes [48,49]. However, PET or fMRI studies in healthy populations provide only weak support for the amygdala’s role: greater amygdala activation to emotional than neutral voices is found typically only with more liberal statistical thresholding. Possibly, multimodal neurons in the amygdala establish associations with face processing [50,51] but contribute to unimodal voice perception in a fairly subtle way.

As for facial emotions, there is some indication that brain representations diverge for different vocal emotions. Using MVPA, multiple vocal emotions could be categorized based on their specific activation patterns in the temporal voice areas [52] and within mPFC and posterior superior temporal cortex [28]. Additionally, brain lesions have been reported to produce emotion-specific deficits. Similar to facial disgust, vocal disgust may be perceived less accurately following insular damage [53], whereas vocal fear and anger may be perceived less accurately following amygdala damage [49], although these lesion studies will require larger samples for corroboration.

The timecourse of voice perception has been investigated with ERPs. Compared to non-vocal sounds, vocalizations enhance a positivity around 200 ms following stimulus onset with sources in the temporal voice areas [54]. Moreover, within that same time range, ERPs differentiate between emotional and neutral voices [55,56, Figure 3] with amplitude modulations predicting changes in the affective connotation of concurrent verbal content [57]. Specifically, neutral words heard in an emotional voice are subsequently rated as more emotional the larger their positive amplitude in the ERP. Together, this evidence implies that affective information from the voice is available within 200 ms following voice onset. Furthermore, as for faces, this temporally coincides with the processing of other voice information, potentially providing an early bias for emotions.

Emotional Touch

Touch constitutes one amongst several somatosensory channels that include the sensation of temperature, pain, and the position of one’s body in space [58]. CT-afferents as well as a variety of mechanoreceptors collectively referred to as Aβ-fibers convey mechanical stimulation from the skin to the brain. Whereas CT signals travel slowly, via unmyelinated fibers primarily along the spinothalamic pathway, Aβ signals travel quickly via myelinated fibers primarily along the posterior-column medial lemniscus pathway. Moreover, whereas CT projections reach the posterior insula directly from the somatosensory thalamus, Aβ projections travel from the thalamus first to primary and secondary somatosensory cortex. Compared to Aβ processing, CT processing has more in common with interoception – it is more affective and less spatially discriminative [58]. Rare peripheral demyelinating diseases can abolish Aβ touch while sparing CT touch. Such patients are impaired at localizing touch on their body, but can still perceive it as pleasant, and selectively activate the insula [59].

To date, neuroimaging studies on tactile emotions have focused on perceived pleasure rather than communicative function. A recent meta-analysis identified the right posterior insula as a structure that is more frequently activated by pleasurable as compared to neutral touch [60, Figure 2]. Studies focusing specifically on the CT system additionally highlight the right posterior STS [61–63] and somewhat less consistently the right orbito-frontal [64–66] and pre-motor/motor areas [62,65,67]. There is some evidence that primary somatosensory cortex contributes to the pleasure from touch: a caress activates this region differentially depending on whether participants believe it to come from a person of the same or the opposite sex [68]. This finding may be an example of higher-order attributional feedback, since a combined fMRI/TMS study using a mechanical touch device found that pleasure ratings of CT-appropriate stroking correlated with activity outside the primary somatosensory cortex and that TMS stimulation to this area altered the perception of touch intensity but not pleasure [67].

Again, insights into the timecourse of emotional touch perception have been sought with electrophysiological measures. Several decades of research have established a somatosensory ERP to tactile stimulation that resembles the ERP for stimuli from other modalities and shows prominent deflections within the first 100 ms following touch onset [69,70]. So far, however, efforts have focused on the effects of benign or painful stimulation to the finger tips, which are dense with ordinary Aβ-fibers but lack CT-afferents. Only one study explored CT appropriate touch and failed to identify a clear somatosensory potential [71]. The ERP associated with light pressure to the forearm caused by an inflating pressure cuff or the hand of a friend showed no visible deflections. Perhaps the gradual increase in pressure could not be picked up with the ERP because it was insufficiently time-locked. Additionally, cortical processing – the main contributor to the ERP – may be limited for CT touch, especially if it is task-irrelevant.

Modality Similarities, Differences and Convergence

Modality Similarities and Differences

To date, research on emotion perception has emphasized the face. Moreover, insights from the face have served as a general guide to search for and establish analogies with other modalities [72,73]. And indeed such analogies exist. Vision, audition, and touch each have slow and fast processing pathways that project to specialized regions of sensory cortex, with upstream regions generally responding more robustly to social over non-social signals, especially if they are emotional (Table 1). Additionally, across these three modalities, contrasts between social and non-social stimuli on the one hand, and between emotional and neutral expressions on the other hand, activate overlapping brain regions with a right hemisphere bias and a similar timecourse.

Table 1.

Comparing the processing characteristics of vision, audition, and touch for perceiving emotions.

Processing Characteristic	Face	Voice	Touch
Topographic maps	Yes	Yes	Yes
Slow and fast processing streams	Yes	Yes	Yes
Enhanced response to social vs non-social stimuli	Yes	Yes	Yes
Enhanced response to emotional vs. neutral expression	Yes	Yes	Yes
Peripheral receptor numbers	100M	16,000	~100,000
ERP markers within the first 200 ms	Yes	Yes	?
Perception requires temporal integration	No	Yes	Yes
Coordination with other modality	Voice	Face	?
Extent of processing before primary sensory cortex	Low	High	Low
Processing streams that bypass primary sensory cortex	Probably	Minimal?	Yes
Does stimulus perception induce feelings?	Sometimes	Yes	Yes

Although we have emphasized central mechanisms of perception, there are also important perceptual differences arising from the nature of the stimulus itself, and from the way that stimulus is produced. One example is the temporal dimension of nonverbal expression [74]. Although all expressions extend in time, faces and body postures can be discerned from a single snapshot, a fact that is being leveraged in most visual studies by presenting still images. In contrast, the investigation of vocalizations and touch necessitates a dynamic stimulus for which receivers must integrate information over time.

Second, the different modalities have different physical constraints imposing different demand characteristics on emotion perception. The face with its many muscles can be minutely controlled to produce a wide range of expressions that perceivers must differentiate. Moreover, the results of this process map onto fairly specific emotion categories rather than simple affect [75]. By comparison, typical interpersonal touch is more limited in variance with regards to temperature, speed, and pressure. As such associated processing demands are comparatively low and perceptual representation more likely to carry simple affective qualities as opposed to information about more differentiated emotions. Exceptions occur when the significance of touch is shaped by additional factors, such as where it is placed on the body or its cultural significance. This may result in context-dependent associations with specific emotions.

Third, emotional expressions show interdependencies among some modalities but not others. For example, facial movements inadvertently shape the mouth region and thereby impact vocal acoustics. In turn, vocalizing by engaging the mouth alters concurrent facial displays. As a consequence, facial and vocal expressions are not independent and influence perceptual processes in a coordinated manner. Yet both have little or no impact on tactile modes of communication and vice versa.

Last, there is evidence for modality-specificity in the neural pathways underpinning perception. Signal representations in primary auditory cortex are significantly more processed than those in primary visual cortex [73]. Pleasurable human touch is known to bypass primary sensory cortex to produce emotions independently of a conscious tactile sensation [59]. In contrast, similar subcortical routes for sound and sight are still being debated. Additionally, to the extent that body representations are involved when we make emotion judgments, touch may evoke emotions fairly immediately due to its interoceptive quality, whereas the voice and face may do so more indirectly via further inferences and embodiment.

Given all these differences, it should come as no surprise that the visual, auditory, and tactile senses each offer particular strengths and limitations when it comes to emotion perception. As a consequence, emerging unimodal representations are only partially redundant, necessitating modality convergence for a holistic and ecological understanding of emotion.

Convergence of the Senses

The mechanisms of modality convergence have long been investigated through behavioral, fMRI, and ERP studies [111]. A clear behavioral finding is that compared to unimodal presentations (e.g., face only), multimodal presentations (e.g., face and voice) yield faster and more accurate emotion judgments [75]. Similarly, fMRI contrasts show greater activation to multi- as compared to unimodal expressions in a range of brain regions including thalamus, STS, FFA, insula and amygdala as well as higher order association areas like lateral and medial PFC and the posterior cingulate [75, Figure 3]. Increased activation in the pSTS, a known convergence zone for expressions from the different senses, is associated with increased functional connectivity between this region and unimodal visual and auditory cortex [76].

Corroborating these results, MVPA successfully categorized emotions across different modalities in a subset of the areas that were found to be more activated for multi- as compared to unimodal stimuli. For example, emotional faces, voices, and body postures could all be classified from activation in STS and medial PFC [28]. Emotional faces and fractals for which participants had learned to associate an emotion could be classified from activation in medial PFC and posterior cingulate [77].

Lastly, ERP research revealed that, compared to unimodal stimulation, multimodal stimulation produces larger amplitude differences between emotional and neutral conditions. Moreover, such multi-modal effects have been reported within 100 ms following stimulus onset [78], whereas comparable unimodal effects emerge typically at 170 ms (faces) or 200 ms (voices) only.

These data indicate that multimodal integration is not simply a late process that occurs after the individual modalities have been analyzed [111]. Instead, it is supported by multimodal neurons at some of the earliest processing stages, including the superior colliculus (SC) [79,80] and the thalamus [81], and increases in strength along cortical pathways [40]. In more detail, the SC, a midbrain structure, divides into superficial layers that are exclusively visual, and deeper layers that combine visual, auditory, and somatosensory input. Reciprocal connections between these layers and their onward projections to the thalamus likely support early aspects of multimodal processing such as temporal binding and cross-modal enhancement (e.g., reducing a unimodal detection threshold) [79,80].

Multimodal processing at the telencephalic level may be two-fold. Regions like the FFA, amygdala, middle STS, and posterior insula may support largely perceptual representations. Here, different physical cues may be “assembled” and mapped onto a stored template previously referred to as an “emotional gestalt” [82]. At a more advanced stage of processing, regions like the PFC, the posterior cingulate, and posterior STS may support amodal, conceptual representations that involve a modality-nonspecific abstract code [82,83]. Although further research is needed to better specify the computations at subcortical and cortical levels, available evidence suggests that multimodal representations emerge incrementally (Figure 4).

Figure 4.

Multimodal convergence of expressive signals in the brain. Beige color indicates the mixing of modalities which begins during early perceptual stages in the brainstem (Superior Colliculus, SC) and thalamus, and progresses at the level of the cerebrum, where physical cues (e.g., facial action units, vocal acoustics) are integrated and mapped onto existing emotion templates (i.e., emotional gestalt). At a late conceptual stage, individuals represent emotional meaning amodally. Higher-level representations can feed back and modulate lower-level representations.

Resultant benefits of convergence for the individual are manifold. As mentioned above, multimodal convergence ensures that distributed emotion signals are being integrated into a holistic percept, which is also generally more reliable. Additionally, convergence underpins temporal binding across unimodal processing streams and the formation of temporal predictions concerning the behavior of interaction partners [74]. Affectively congruous expressions are processed more readily than incongruous ones [84], an effect perhaps subserved by Hebbian associations. Furthermore, multimodal incongruity may provide clues to deception, as when a smiling face belies a more aggressive tone of voice. Learned multimodal associations may also help interpret the personal expressive styles of particular individuals.

Concluding Remarks

Social interactions elicit spontaneous and strategic expressions that we commonly classify as emotions, and that profoundly influence a perceiver’s mental state and ongoing behavior [13,57,71]. Neuroscience investigations have been uneven, with a historical overemphasis on facial signals. Although many questions remain (see Outstanding Questions), enough evidence has accumulated to characterize mechanisms both unique and shared across modalities. Furthermore, although each channel engages modality-specific sensory systems, the perceptual representations emerging from these systems are integrated very early. Uni- and multimodal streams feed back to earlier levels as well as project onward to perceptual and conceptual nodes distributed throughout the cortex. To ultimately understand how emotion signals guide our social interactions, future studies should adopt a rich approach that cuts across methods and temporal scales, that acknowledges the ecological validity of dynamic stimuli, and that involves not one but multiple sensory modalities.

Outstanding Questions.

Do individual emotion categories activate distinct neural regions?

While there is good evidence that MVPA within regions can distinguish emotion categories, the role of entire neural structures is less clear. For instance, while lesion studies suggest the amygdala may be necessary specifically for fear recognition, this is not corroborated by fMRI studies.

What are the relations between emotion expression, perception, and experience?

It is still open whether emotion recognition necessitates an emotional experience and whether such experience is causally involved in the recognition. Similarly, emotion perception often causes mimicry in the perceiver, which is in turn perceived; such coupling likely plays an important role in how emotion communication unfolds in time.

What are good tasks for studying spontaneous and intentional emotion recognition?

The tasks used to study emotion recognition are typically highly artificial (e.g., change detection, forced-choice categorizing). It will be important to develop paradigms that more closely approximate how we engage with emotion expressions in real life.

Do current insights hold when stimuli are made more ecologically valid?

In an effort to avoid confounds, many studies use impoverished stimuli that are devoid of context. For example, many visual studies use still images and many tactile studies use a tactile device rather than actual human touch.

How prominent are individual differences?

Most studies to date have explored average responses across all study participants. However, there is accumulating evidence that individual factors like sex, age, personality and culture influence both emotion expression and perception.

What social signals qualify as expressions of emotions?

We make a lot of nonverbal sounds and move our faces a lot. Are all of these “emotional?” We would suggest reserving that term for only a subset, but considerably more theoretical work is needed to clarify where to draw the boundary.

Trends.

• Facial expression and perception have long been the primary emphasis in research. However, there is a growing interest in other channels like the voice and touch.

• Facial, vocal, and tactile emotion processing have been explored with a range of techniques including behavioral judgements, EEG/ERP, fMRI contrast studies and muti-voxel pattern analyses.

• Results point to similarities (e.g., increased responses to social and emotional signals) as well as differences (e.g., differentiation of individual emotions versus encoding of affect) between communication channels.

• Channel similarities and differences enable holistic emotion recognition – a process that depends on multisensory integration during early, perceptual and later, conceptual stages.

Glossary

Affect

often assumed to be a precursor or prerequisite for a more differentiated emotion, affect is typically described as two-dimensional, varying in valence (pleasant to unpleasant) and arousal (relaxed to excited). While applied primarily to the structure of feelings, it can also be applied to behavioural consequences of emotion states (e.g., approach-withdrawal).

Amygdala

an almond-shaped nucleus situated in the anterior aspect of the medial temporal lobe. It comprises several sub-nuclei and contributes to a diverse set of functions including sexuality, memory, and emotions. In the context of emotion perception, the amygdala has been implicated particularly in the recognition of fear from facial expressions.

CT-afferent

a tactile receptor found in non-glabrous (i.e., hairy) skin that conveys pressure, speed and temperature information via slowly conducting C fibers – a class of fibers that typically enables pain. However, CT responses are tuned to the characteristics of benign human touch.

FMRI

functional magnetic resonance imaging detects small changes in blood oxygenation in the brain to infer bulk changes in neuronal activity. It enables the investigation of stimulus- or task-dependent regional changes in brain activation, as well as the investigation of functional connectivity between regions.

EEG

electroencephalogram, a technique that records the bulk electrical activity of cortical neurons with electrodes placed on the scalp. It provides much better temporal resolution than fMRI (milliseconds compared to seconds) but poorer spatial resolution.

Embodiment

refers to the engagement of one’s own body, or representations thereof, in cognition. For instance, embodiment of observed actions is hypothesized to facilitate their comprehension: facial expressions are recognized more accurately when individuals can spontaneously mimic the expression they see.

Emotions

in this review, emotions are taken to be complex neurobiological states elicited by situations relevant for an individual’s current or prospective needs. They motivate and coordinate cognitions and behaviors that help fulfil these needs.

ERP

event-related potentials are derived by time-locked averaging of the EEG, typically in response to a specific stimulus. They comprise a series of positive and negative deflections referred to as ERP components.

Feelings

the conscious experiences of emotions, typically measured with verbal report in humans.

Interoception

the perception of one’s internal bodily state (e.g., heart rate).

Multi-voxel pattern analysis (MVPA)

a multivariate machine learning approach used for fMRI data analysis. It analyzes the response pattern of multiple individual voxels within a specified brain region. In a first step, a classifier learns to distinguish the repertoire of patterns from two conditions. In a second step, the trained classifier is applied to novel data (a novel pattern) and its accuracy in distinguishing between the two conditions is established. The approach capitalizes on the assumption that information in the brain is represented in a high-dimensional format – in variance over patterns, rather than mere amplitude of response.

Perception

initial receptive processing encompassing abilities such as detection and discrimination, often thought of as providing the input representations for subsequent recognition and categorization.

Tonotopy

the spatial organization of neurons into maps (topography) so that they are organized according to their tuning to sound frequency.

Transcranial magnetic stimulation (TMS)

experimental manipulation of neuronal activity over a region of cortex achieved with magnetic field pulses that induce local currents in the brain.

Voice quality

a composite of vocal characteristics that are shaped by vocal tract configuration and laryngeal anatomy. They arise in part from resonance properties of the vocal system and create sound impressions such as hoarse, nasal or breathy. (http://www.ncvs.org/ncvs/tutorials/voiceprod/tutorial/quality.html).