Tuning the developing brain to social signals of emotions

PREFACE

Humans in diverse cultures develop a similar capacity to recognize the emotional signals of different facial expressions. This capacity is mediated by a brain network that involves emotion-related brain circuits and higher-level visual representation areas. Recent studies suggest that the key components of this network begin to emerge early in life. The studies also suggest that initial biases in emotion-related brain circuits and the early coupling of these circuits and cortical perceptual areas provides a foundation for a rapid acquisition of representations of those facial features that denote specific emotions.

INTRODUCTION

Most humans are particularly skilled at recognizing affectively-relevant information displayed in faces. Scientific interest in the recognition of emotional expressions was reawakened nearly 40 years ago by the discovery that facial expressions are universal^{1, 2}. With the advent of neuroimaging, the brain systems that underlie the ubiquitous human capacity to recognize emotions from facial expressions and other types of social cues have become a burgeoning area of research in cognitive and affective neuroscience. A closely related field of research has examined these processes and their developmental foundations in human children and experimental animals. In this article, we review and synthesize recent findings from these interrelated areas of research. These findings reveal that emotion-related brain circuits (which include amygdala and orbitofrontal cortex) and the influence of these circuits on higher-level visual areas underlies rapid and prioritized processing of emotional signals from faces. The findings also suggest that the key components of the emotion-processing network and emotion-attention interactions begin to emerge early in postnatal life at the time that infants’ visual discrimination abilities undergo substantial experience-driven refinement. Collectively, there follows the suggestion that the ability to mentally represent facial expressions of emotion might be a paradigmatic case of how emotional brain systems (which are biased to respond to certain biologically salient cues) and interconnected perceptual representation areas attune to species-typical and salient signals of emotions in the social environment. We will also discuss how genetic and environmental factors may bias this developmental process and give rise to individual differences in sensitivity to signals of certain (negative) emotions.

NEURAL BASES OF FACIAL EMOTION PROCESSING

An important function of the emotional brain systems is to scan incoming sensory information for the presence of biologically relevant features (e.g., stimuli that represent a threat to well-being) and grant them priority in access to attention and awareness^{3, 4}. For humans, the most salient signals of emotion are often social in nature, such as facial expressions of fear (which are indicative of a threatening stimulus in the environment) or facial expressions of anger (which are indicative of potential aggressive behavior). Consistent with the view that such signals are rapidly detected and subjected to enhanced processing, behavioral studies in adults have shown preferential attention to fearful facial expressions relative to simultaneously presented neutral or happy facial expressions⁵, better detection of fearful than neutral facial expressions in studies in which the likelihood of stimulus detection is reduced by using rapidly changing visual displays^{6, 7}, and delayed disengagement of attention from fearful as compared to neutral or happy facial expressions⁸.

Electrophysiological^{5, 9, 10} and functional Magnetic Resonance Imaging (fMRI)^11–13 studies have further shown that activity in face-sensitive cortical areas such as the fusiform gyrus and the superior temporal sulcus is enhanced in response to fearful as compared to neutral facial expressions. Although similar enhanced activation in these areas is observed to attended relative to unattended facial stimuli, there is evidence that attentional and emotional modulation of perceptual processing are mediated by a distinct neural network, the former reflecting a distal influence of frontoparietal attention networks and the latter reflecting the influence of emotion-related brain structures such as the amygdala and orbitofrontal cortex on perceptual processing⁴ (FIG. 1).

Figure 1. Emotion processing network in the brain.

Neural systems involved in processing emotional signals from faces (Based on models presented in REFS. ⁴, ¹⁴, ¹²⁷). Emotion-related neural systems (amygdala and orbitofrontal cortex) receive visual information from cortical regions that are involved in the visual analysis of invariant and changeable aspects of faces (fusiform gyrus and superior temporal sulcus) and possibly also via a faster magnocellular pathway directly from the early visual cortex³³ or via a subcortical collicular-pulvinar pathway to the amygdala⁴, ⁴¹. The amygdala and orbitofrontal cortex send feedback projections to widespread visual areas, including the fusiform gyrus and superior temporal sulcus. AMY, Amygdala: OFC, Orbitofrontal Cortex: FG, Fusiform Gyrus: STS, Superior Temporal Sulcus

The importance of the amygdala for emotion recognition is well established¹⁴ but only recently have studies begun to shed light on the mechanisms by which the amygdala enhances the processing of emotional stimuli⁴. Findings from these studies are consistent with a model in which the amygdala responds to coarse, low-spatial frequency information about facial expressions (i.e., the global shape and configuration of facial expressions) in the very early stages of information processing (possibly as rapidly as 30 ms after stimulus onset)¹⁵ and subsequently enhances more detailed perceptual processing in cortical face-sensitive areas such as the fusiform gyrus and the superior temporal sulcus^16–18. The amygdala may enhance cortical activity via direct feedback projections to visual representation areas^19–21 or via connections to basal forebrain cholinergic neurons that transiently increase cortical excitability^22–24.

The exact stimulus features to which the amygdala is responsive are unknown. The amygdala was initially associated with the processing of fearful facial expressions but more recent findings point to a broader role in processing biological relevance (either reward- or threat-related)²⁵, and in evaluating and acquiring information about associations between stimuli and emotional significance^{13, 26}. Such processes may be more reliably engaged in response to fearful than, for example, happy expressions, explaining why enhanced amygdala activity is more consistently observed to fearful as compared to other facial expressions.

The orbitofrontal cortex has also been implicated in the recognition of emotions from facial expressions and in top-down modulation of perceptual processing. Patients with localized brain damage to the orbitofrontal cortex exhibit impaired recognition of a range of facial expressions²⁷, and this region is activated in fMRI and PET studies when neurologically normal adults view positive or negative facial expressions^{28, 29}. Activity in the orbitofrontal cortex is increased when observers learn object-emotion associations from stimuli that show facial expressions paired with novel objects, which is consistent with the putative role of this region in representing positive and negative reinforcement value of stimuli^{30, 31}.

The orbitofrontal cortex has reciprocal connections with the amygdala and widespread cortical areas, including face-sensitive regions in the inferotemporal cortex and the superior temporal sulcus (FIG. 1)³². As is the case with the amygdala, the orbitofrontal cortex may receive low-spatial frequency information via a rapid magnocellular pathway and exert a top-down facilitation effect on more detailed perceptual processing in perceptual representation areas³³. Consistent with such a neuromodulatory role, recent studies have provided evidence for an early response in orbitofrontal cortex (130 ms after stimulus onset) that precedes activity in occipitotemporal perceptual representation areas (165 ms post-stimulus)³⁴.

Individual differences in facial emotion processing

Although a common neural network is generally engaged in response to salient facial expressions, the strength of activity in this network and sensitivity to signals of certain emotions can vary substantially across individuals³⁵. For example, stable individual differences in anxiety-related traits predict sensitivity to facial expressions of threat so that individuals with high trait-anxiety show relatively enhanced orienting of attention to threat-related facial cues and are relatively less efficient in disengaging their attention from fearful facial expressions^{8, 36}. Consistent with these behavioral findings, fMRI studies have shown that high trait-anxiety is correlated with elevated activity in the amygdala in response to fearful and angry facial expressions^{37, 38}, and that individuals with higher trait anxiety show less habituation in the amygdala over repeated presentation of facial expressions³⁹. The elevated activity in the amygdala may partly reflect less efficient emotion regulation processes arising from reduced functional connectivity between the amygdala and regions in the prefrontal cortex (anterior cingulated cortex)³⁹.

THE ONTOGENY OF FACIAL EMOTION PROCESSING

Prior to the onset of language, the primary means infants have in communicating with others in their environment, including caregivers, is through “reading” faces. Thus, it is important for an infant not only to discriminate familiar from unfamiliar individuals, but as importantly, to derive information about the individual’s feelings and intentions; for example, whether the caregiver is pleased or displeased, afraid or angry. After the onset of locomotion, infants also use others’ facial expressions to acquire knowledge about objects in the physical environment - that is, about objects that are safe and can be approached and objects that are potentially harmful and should be avoided⁴⁰. Thus, the accurate decoding of facial signals, particularly facial expressions, is absolutely fundamental in early interpersonal communication. Further, even after language develops, the accurate decoding of facial emotion continues to play a prominent role in face to face interactions.

The human infant’s ability to discriminate and recognize facial emotion has received extensive study over the past 20 years. Such work has recently been complemented by electrophysiological and optical imaging studies as well as developmental work in other species. Evidence from these studies converges to suggest that the key components of the adult emotion-processing network emerge early in postnatal life. It has been suggested that some of these brain systems (the amygdala) are functional at birth and have a role in orienting newborn infants’ attention towards faces and in enhancing activity in certain cortical areas to faces⁴¹. Consistent with their role in adult facial emotion processing, the evidence reviewed below suggests that the amygdala and associated brain regions also participate in facial emotion processing and emotion-attention interactions in infants, although these functions likely do not emerge until the second half of the first year of life.

Behavioral studies

Because of infants’ limited visual acuity, contrast sensitivity and ability to resolve high spatial frequency information at birth⁴², and limited attention to internal features of faces during the first two months of life^{43, 44}, it is unlikely that infants are able to visually discriminate facial expressions at birth or during the first months of life, except when highly salient facial features change (such as open vs. closed mouth; see REF. 45). Consistent with this view, several studies have shown that the reliable perception of facial expressions, such as attention to configural rather than featural information in faces⁴⁶, and the ability to recognize facial expressions across variations in identity or intensity^47–49, are not present until the age of 5 to 7 months. It also seems that instead of visual information from facial expressions, infants may initially use more salient multimodal cues (e.g., synchronous facial and vocal stimuli) to detect and discriminate emotional expressions, and only later acquire representations of the relevant unimodal cues^50–52. Supporting this view, a recent study⁵¹ demonstrated that the ability to discriminate emotional expressions in audiovisual stimuli emerged between 3 and 4 months of age, earlier than discrimination of emotions in unimodal auditory (at 5 months of age) or visual stimuli (at 7 months of age, see FIG. 2).

Figure 2. Development of facial emotion discrimination in infancy.

a) An illustration of the habituation visual-paired test paradigm in infants. Presentation of facial expressions (either right side up or upside down) from a specific category (“happy”) is continued until the infant habituates (e.g., until their looking time declines to half of what it was when the stimulus was first presented). After habituation, the stimulus from the familiar category is paired with a stimulus from a novel category (“fearful’). Discrimination is inferred from a preference (that is, an increased looking time) for the novel stimulus. b) Results showing that discrimination of emotional expressions in bimodal (audiovisual) stimuli emerges earlier than discrimination of emotional expressions in unimodal auditory or visual stimuli. (adapted from REF. 51). c) After habituation to happy expressions on different faces, 7-month-old infants were able to discriminate this expression from fearful and angry expressions when the stimuli were presented upright but not when they were inverted. These findings show that, similar to adults, infants attend to orientation-specific configural cues to categorize facial expressions (adapted from REF. 46). * p < .05, ** p < .01

At around the same age as infants start to exhibit stable visual discrimination of facial expressions, infants begin to exhibit adult-like attentional preference towards fearful over neutral or happy facial expressions: for example, when exposed to face pairs, 7-month-old infants look longer at a fearful over a happy facial expression^{53, 54}. More detailed investigation into this looking-time bias has shown that fearful facial expressions affect the ability to disengage attention⁵⁵. Specifically, infants are less likely to disengage their attention from a centrally presented target face towards a suddenly appearing peripheral target when the face displays a fearful expression as compared to neutral or happy expressions (FIG. 3). The finding that novel non-emotional grimaces⁵⁵ or neutral faces with large eyes (M. J. Peltola, J. M. Leppänen, V. K. Vogel-Farley, J. K Hietanen, & C. A. Nelson, unpublished manuscript) fail to exert similar effects on attention disengagement indicates that the effect of fearful faces is not simply attributable to their novelty in infants’ environment or to their distinctively large eyes (a feature that is also known to affect infants’ attention). It is also of note that the effect of fear on attention in infants is very similar to that observed in adults⁸, suggesting a similar underlying mechanism.

Figure 3. Emotional influences on attention and perception in infants.

Adults show enhanced perceptual processing and attention to stimuli that are associated with emotional significance¹³⁸, which probably reflects a modulatory effect of emotion-related brain structures on cognitive processing. Recent data suggest that similar effects of emotion on attention and perception occur in infants. a) In a behavioral attention task, 7-month-old infants were less likely to move their gaze from a centrally presented fearful face to a peripheral target than from a non-face control stimulus, suggesting enhanced attention to fearful faces. This effect is not explained by the low-level features of fearful facial expressions (e.g., salient eyes) or the novelty of these expressions in infants’ rearing environment because control stimuli with these characteristics failed to produce similar effects (adapted from REF. 55). b) Recordings of event-related potentials from posterior scalp regions reveal augmented ERP activity over the semi-medial occipito-temporal scalp in response to fearful as compared to neutral facial expressions, suggesting a modulatory effect of fear on early cortical face processing (adapted from REF. 66).

Electrophysiological and optical imaging studies

There is a substantial body of literature concerning the neural correlates of face perception in infants. An extensive review of this literature has been published elsewhere⁵⁶. In brief, studies that have recorded event-related potentials (ERPs) to measure brain activity have shown that neural activity over the occipitotemporal part of the scalp differs for faces as compared to various non-face objects in 3–12 month-old infants^57–60. Evidence from other sources such as a rare PET study in 2-month-old infants⁶¹ and results of recent optical imaging studies⁶² further suggests that the fusiform gyrus and the superior temporal sulcus are functional in infants at this age and exhibit some degree of tuning to faces. It seems, however, that infants’ face-processing mechanisms are activated by a broader range of stimuli than those of adults^{58, 60}, suggesting that the underlying neuronal populations become more tuned to human faces over the course of development.

Reciprocal connections between visual representation areas and the amygdala⁶³ and the orbitofrontal cortex⁶⁴ are observed soon after birth in anatomical tracing studies in monkeys (BOX 1). This suggests that emotion-related brain structures are possibly functional at the time when infants start to exhibit behavioral discrimination of facial expressions. To date, the evidence for this hypothesis has accrued from investigations of the neural correlates of infants’ processing of happy and fearful facial expressions. A recent study examined the neural bases of perceiving neutral and smiling faces in 9–13-month-old infants and their mothers⁶⁵. An extensive adult literature has shown that regions in the orbitofrontal cortex are activated in responses to positive affective cues, and may have a role in representing the reward value of such cues^{28, 31}. To study whether the same regions are active in infants, Minagawa-Kawai et al.⁶⁵ used Near Infra-Red Spectroscopy (NIRS) to measure changes in activity in fontal brain regions in response to neutral and smiling faces. The results revealed an increase in brain activity in response to smiling as compared to neutral faces, with the peak of the activity observed in anterior parts of the orbitofrontal cortex. This increase in activity was particularly pronounced when infants were viewing their mother’s face as compared to an unfamiliar adult’s smiling face, although it was present in both conditions. A similar increase in activity in the same brain regions was observed in mothers while they viewed happy expressions of their own infant. Mothers showed no response to an unfamiliar infant’s happy expressions, suggesting that the same regions are activated in infants and adults but in adults, the activation of the orbitofrontal cortex is more selective to happy expressions of a specific individual.

Box 1. Anatomical development of emotion-related brain structures.

Most of the information regarding the anatomical maturation of the amygdala and the orbitofrontal cortex comes from neuroanatomical studies in macaque monkeys (reviewed in REF. 64). Both structures seem to reach anatomical maturity relatively early in development. Neurogenesis of the amygdala is completed by birth^109–111, reciprocal connections to various cortical regions are established in 2-week old monkeys⁶³, and the distribution of opiate receptors as well as the density and distribution of serotonergic fibers seem adult-like at birth or soon after^{64, 112–114}. There is no information regarding the neurogenesis of the orbitofrontal cortex⁶⁴ but connections between orbitofrontal cortex and temporal cortical areas¹¹⁵ and adult-like dopamine innervation^{64, 113} are established by one week of age in the monkey. Although these pieces of information point to very early anatomical maturation of structures implicated in emotion recognition, there is also evidence that notable anatomical changes in these structures and their connectivity with other brain regions occur during a relatively protracted period of postnatal life. For example, there are connections from inferotemporal cortex area TEO to the amygdala in infant monkeys that do not exist in adults¹¹⁶, and, although feedback projections from the orbitofrontal cortex to temporal cortical areas emerge early, they continue to mature until the end of the first year⁶⁴. Also, myelination of axons in the amygdala, the orbitofrontal cortex and their connections with other brain regions, begins in the first months of life but continues for several years ^{64, 117}. Together, these latter findings suggest that, although the key components of the emotion-processing networks and their interconnectivity are established soon after birth, the wiring pattern becomes more refined over the course of postnatal development.

Because ERP and optical imaging tools are generally insensitive to activity in subcortical brain structures, it has not been possible to demonstrate directly the role of the amygdala in infants’ emotion processing. However, some ERP findings are consistent with the existence of adult-like neural circuitry that is specifically engaged by fearful facial expressions and modulates activity in cortical perceptual and attention networks. In 7-month-old infants, a positive component that occurs ~400 ms after stimulus onset over medial occipito-temporal scalp and relates to visual processing of faces^57–60 is enhanced when infants are viewing a fearful as compared to happy or neutral facial expressions (FIG. 3)⁶⁶. Similar effects are well-documented in the adult literature and are thought to reflect an effect of affective significance on cortical processing^{9, 10}. Besides visual processing, fearful facial expressions also enhance activity in cortical attention networks, which is consistent with behavioral indications of enhanced attention towards fearful facial expressions. In 7-month-old infants, the negative central (Nc) component over the frontocentral scalp is larger to fearful as compared to happy facial expressions^{66, 67}. The Nc is known to relate to orienting of attentional resources in response to salient, meaningful, or infrequently occurring stimuli^{68, 69}. The cortical sources of the Nc have been localized to the anterior cingulate region⁷⁰, which is consistent with the role of this region in the regulation of attention⁷¹.

Recent studies have further shown that the augmented Nc in response to fearful expressions is more pronounced when infants view a person who expresses fear and gazes at a novel object (implying that the object possesses an attribute of which the infant should be fearful) as compared to a situation in which a fearful looking person directs eye gaze at the infant (suggesting that the person looking at the infant feels afraid)⁷². It also seems that infants attend more to a novel object after they have seen an adult expressing fear towards the object⁷³. These findings are remarkable as they not only suggest that the neural circuitry that underlies the modulatory effect of affective significance on perceptual and attention networks are functional in infants but also that the stimulus conditions that engage these circuits resemble those that are optimal for engaging emotion-related brain circuitry in adults (i.e., situations that involve stimulus-emotion learning)¹³.

Evidence from other species

Studies in monkeys provide further evidence for an important role of the amygdala in mediating early-emerging affective behaviors. The strongest evidence for this comes from experimental lesion studies in monkeys, showing that amygdala lesions in neonate monkeys result in abnormal affiliation and fear-related behaviors, possibly due to underlying impairments in the evaluation and discrimination of safe and potentially threatening physical and social stimuli⁷⁴. Another important finding that has emerged from recent work in rats shows that the neural circuitries for learning stimulus-reward associations (preferences) and stimulus-shock associations (aversion) have distinct developmental time-courses in early infancy. In rats, the ability to form preferences to cues associated with positive reinforcement is present from birth whereas the ability to avoid cues associated with negative stimuli (footshock) is not observed until postnatal day 10 when the pup is ready to leave the nest⁷⁵. Other experiments have shown that the delayed onset of learning to avoid aversive stimuli reflects immature GABAergic function and amygdala plasticity during the first postnatal days⁷⁶. These findings are of interest as they may shed light on the observation that the differential responsiveness to happy and fearful emotional expressions (a preference for fear) is not observed in humans infants until several months after birth^{66, 77}.

MECHANISMS OF DEVELOPMENT

The early emergence of some components of the emotion processing network begs a more fundamental question concerning the mechanisms that govern the development of this brain network. In the sections that follow, we discuss the possibility that these early foundations reflect a functional emergence of an experience-expectant mechanism⁷⁸ that is sensitive to and shaped by exposure to species-typical aspects of emotional expressions. We will also discuss how these early foundations are further modified by individual-specific experiences, reflecting an additional experience-dependent⁷⁸ component of the development of emotion processing networks.

Experience-expectant mechanisms

The universal nature of some facial expressions and the presence of these expressions throughout the evolutionary history of humans raises the possibility that the species has come to “expect” the occurrence of these expressions in different environments at a particular time in development (cf. REF. 78). The species might have evolved brain mechanisms that are to some extent biased from the beginning for processing biologically salient signals displayed in the face. The evidence for the early maturation of emotion-related brain circuits, functional coupling of these structures with visual-representation areas, and the behavioral indices of a bias to attend more to emotionally salient than to neutral facial expressions is consistent with the existence of such an experience-expectant foundation for the development of emotion recognition. What is known so far about the attentional biases in infants is more consistent with a prewired readiness to attend and incorporate information about some salient cues rather than a bias towards a stimulus that infants have learned signals a specific meaning. That is, infants “prefer” to attend to fearful faces⁵⁴, show enhanced visual and attention-related ERPs to fearful faces^{66, 67} and have difficulty in disengaging from fearful faces⁵⁵, but there is no evidence that they feel afraid when they are exposed to fearful faces. Thus, infants exhibit a seemingly obligatory attentional bias towards fearful facial expressions and find them perceptually salient even though they do not seem to derive meaning from fearful facial expressions or understand why they do so. It also seems that the bias to attend to fearful expressions emerges at the developmental time point at which such expressions are most likely to occur in the infant’s environment; that is, around 6 to 7 months, when infants start to crawl and actively explore the environment (and hence place themselves at risk for harm unless there is an attentive caregiver in proximity). Although the evidence is consistent with the existence of limited preparation to attend to biologically salient cues that are displayed in the face, the exact stimulus features to which the infant is sensitive are currently not known. The bias to attend to fearful facial expressions might reflect a bias towards some visual features of fearful facial expressions, a bias towards expressions of fear more generally, or a more abstract and broadly tuned bias towards certain feature configuration of which fearful facial expressions are only a good example.

As is the case with other experience-expectant mechanisms⁷⁸, the preparedness to process facial expressions is likely to involve a coarsely specified but slightly biased neural circuitry that requires exposure to species-typical emotional expressions in order to be refined and develop towards more mature form. This developmental process may involve preservation and stabilization of some initially existing synaptic connections and pruning of others, resulting in a more refined pattern of connectivity in the network (see BOX 2), and a narrowing of the range of stimuli to which the network is responsive. Recent studies in infants illustrate how the perceptual mechanisms that underlie face processing are initially broadly tuned, and become more specialized for specific types of perceptual discriminations with experience^79–82. In a study that first demonstrated this phenomenon, 6-month-old infants were shown to be able to discriminate two monkey faces from one another as easily as two human faces, whereas 9-month-old infants and adults were able to discriminate only human faces⁷⁹. It was subsequently reported that 6-month-old infants who were given 3 months of experience viewing monkey faces retained the ability to discriminate novel monkey faces (at 9 months) whereas infants who lacked such experience could not⁸⁰. Similar phenomena have now been demonstrated to occur in the perception of intersensory emotion-related cues (i.e., in the ability to match a heard vocalization with the facial expression producing that vocalization)⁸¹ and in the perception of lip movements that accompany speech⁸². Four to 6 month-old infants can, for example, discriminate silent lip movements that accompany their native speech as well as lip movements that accompany non-native speech, but only the native-language discrimination is maintained at the age of 8 months⁸². Collectively, these findings suggest that experience-driven fine-tuning of perceptual mechanisms reflects a general principle of the development of different aspects of face processing⁸³.

Box 2. Multiple functions of the superior temporal sulcus.

Evidence from single-cell studies in the macaque monkey^118–121 and functional MRI studies in humans^122–125 show that regions in occipito-temporal cortex, which in humans include the inferior occipital gyri, the fusiform gyrus, and the superior temporal sulcus, are critical for perceptual processing of information from faces. Of these face-responsive regions, the superior temporal sulcus (STS) seems to have a key role in the perception of “changeable” aspects of faces such as facial expression, eye gaze, and lip movements^125–127. The importance of the STS in the perception of facial expressions may also reflect its role in the integration of separate sources of information¹²⁸. People typically use and integrate information from several sources to recognize emotional expressions, including the spatial relations of key facial features¹²⁹, dynamic cues related to temporal changes in expression¹³⁰, gaze direction¹³¹, and concurrent expressive cues in other sensory modalities such as vocal emotional expressions^{132, 133}.

The STS has been implicated not only in the perception of changeable aspects of faces and audiovisual integration¹³³ but also in several other domains such as the perception of biological motion and social stimuli¹³⁴ and speech perception¹³⁵ (See REF. 136 for a review). Although it is possible that these functions are mediated by distinct subregions of the STS, a recent review of fMRI studies in humans identified only two distinct clusters of activation, one in the anterior STS that was systematically associated with speech perception, and another in the posterior STS that was associated with several functions, including face perception, biological motion processing, and audiovisual integration¹³⁶. The similar activation of the STS in different contexts may be explained by a common cognitive process across different domains¹³⁷. Alternatively, differential patterns of co-activation and interactions with other brain regions may explain how the same region can participate in different functions^{127, 136}.

A recent study in monkeys⁸⁴ has further demonstrated that face-processing mechanisms remain in the immature state if the expected experiences do not occur and also that experiences occurring in a sensitive period (i.e., a period when the animal is first exposed to faces) may have irreversible influences on the developing face-processing system. Here, infant monkeys were reared with no exposure to faces for 6–24 months and were then selectively exposed to either monkey or human faces for one month. Upon termination of the deprivation period (and prior to exposure to any faces), the monkeys exhibited a capacity to discriminate between individual monkey face identities as well as human face identities, suggesting that such discrimination abilities require little if any visual experience in order to develop. This initial capacity was, however, changed after the short exposure period so that monkeys maintained the ability to discriminate face identities of the exposed species but had considerable difficulties in discriminating face identities of the non-exposed species.

Experience-dependent development

Although experience-expectant mechanisms and exposure to species-typical facial expression may provide a foundation for a rapid acquisition of perceptual representation of the universal features of facial expressions, these representations are likely to be further shaped by individual-specific experiences and the frequency and intensity of certain facial expressions in the rearing environment. The strongest evidence that emotion-recognition mechanisms are shaped by individual experience comes from studies in maltreated children. Children of abusive parents are exposed to heightened levels of parental expressions of negative emotions and high rates of direct verbal and physical aggression. Studies^85–89 have shown that emotion-recognition mechanisms are significantly shaped by such experiences. School-aged children with a history of being physically abused by their parents exhibit generally normally organized perceptual representations of basic facial expressions such as fearful, sad, and happy facial expressions, but they exhibit heightened sensitivity and a broader perceptual category for signals of anger as compared to children who are reared in typical environments. Compared to non-maltreated children, abused children show a response bias for anger, which means that they are more likely to respond as if a person is angry (displays an angry expression) when the nature of the emotional situation (the emotional state of a protagonist in a story) is ambiguous⁸⁵. They also allocate a disproportionate amount of processing resources (as inferred by the amplitude of the attention-sensitive ERPs) to angry facial expressions⁸⁷. Finally, abused children show a perceptual bias in the processing of angry faces, which causes them to classify a broader range of facial expressions as perceptually similar to angry faces and also to recognize anger on the basis of partial sensory cues^{86, 88}. Together, these different indices of increased perceptual sensitivity to visual cues of anger may reflect an adaptive process in which the perceptual mechanisms that underlie emotion recognition become attuned to those social signals that serve as important predictive cues in abusive environments⁸⁸.

Given the generally normal recognition of facial expressions in abused children (except for the broadened perceptual category for signals of anger), the effect of such species-atypical experience seem to reflect a tuning shift rather than a gross alteration of representations of facial expressions. It seems, therefore, that the basic organization of the emotion-recognition networks is specified by an experience-expectant neural circuitry that emerges during a sensitive period of development – perhaps the first few years of life - and that rapid refinement of this circuitry occurs by exposure to universal features of expressions. Individual-specific experiences may, however, alter the category boundaries of facial expressions. In theory, because experience-dependent processes are not tied to a particular point in development, this speaks to the brain’s continual plasticity in both adaptive and maladaptive responses; for example, a tendency to view ambiguous faces in a positive light vs. a negative light. In addition, the perceptual biases that transpire through experience-dependent changes should, again in theory, be modifiable. Thus, a maltreated child who acquires a bias to see anger more readily than other emotions should be able to unlearn this same bias. This is quite different, however, from the early perceptual biases that come about through experience-expectant development; thus, for example, it is unlikely that one can “unlearn” the bias to respond quickly to fear, since doing so a) was acquired during a sensitive period of development, leading to a crystallization of neural circuits, and b) being tuned to fearful faces may confer survival.

Recent studies are consistent with the view that components of the emotion-processing network retain some plasticity throughout the life span and can quickly alter their response properties to stimuli that are associated with rewarding or aversive experiences^{13, 26, 90}. Cells in the monkey amygdala, for example, come to represent such associations very rapidly, often on the basis of a single exposure to a stimulus and subsequent reward or aversive stimulation⁹⁰. Although representations of salient stimuli may first be stored in the amygdala and the orbitofrontal cortex, it is likely that plasticity also occurs in connected visual regions that are relevant for the processing of visual and intersensory information from emotional expressions. In rats, for example, neurons in the primary auditory cortex can tune their receptive fields to the frequency of stimuli that are associated with appetitive or aversive reinforcements⁹¹. Such tuning shifts are acquired rapidly and are retained for up to eight weeks. Tuning shifts may be mediated by projections from the amygdala to the cholinergic nucleus basalis, resulting in increased release of acetylcholine from the nucleus basalis to the cerebral cortex⁹¹. Although such long-term plasticity has not been demonstrated in the context of human facial-expression processing, recent findings have shown similar-type changes including heightened perceptual sensitivity and strengthened cortical representation of pictures of facial expressions that are paired with affectively significant events^92–94.

INDIVIDUAL DIFFERENCES IN DEVELOPMENT

Initial biases in emotion-related brain circuits and their experience-driven refinement are likely to contribute not only to the general development of facial-expression processing but also to individual differences in this developmental process. Explicating these mechanisms is important given that heightened sensitivity to signals of some emotions (such as threat) may predispose an individual to learn fears in social settings⁹⁵ and is known to have a causal role in vulnerability to emotion-related disorders⁹⁶.

One possibility is that genetic factors, such as common variants in gene sequence (polymorphisms) that affect major neurotransmitter systems, contribute to the reactivity of emotion-related brain circuits. A promising line of research has shown, for example, that a polymorphism in a gene that regulates the serotonin transporter (5-HTT) and affects brain serotonin transmission, is associated with the reactivity of the amygdala and associated perceptual representation areas in response to fearful and angry facial expressions⁹⁷. Individuals with one or two copies of the “short” (S) allele of the 5-HTT polymorphism (i.e., the allele that is associated with reduced 5-HTT availability and vulnerability to depression)⁹⁸ exhibit relatively greater amygdala responses to threatening facial expressions compared to individuals who are homozygous for the 5-HTT “long” (L) allele⁹⁷. Given that such genetically driven differences in serotonin function are likely to be present from birth^{99, 100}, they may, in combination with environmental factors (such as exposure to negative emotions), set the stage for the development of increased perceptual sensitivity to negative emotions. It is important to note, however, that the heightened attention to potent signs of danger as seen in adults with anxiety disorders⁹⁶ is also likely to depend on other factors, such as the integrity of later developing cortico-amygdalar control mechanisms that act to regulate stimulus selection and the allocation of attentional resources to negative emotional cues¹⁰¹.

CONCLUSIONS AND FUTURE DIRECTIONS

A network of emotion-related brain systems (amygdala and orbitofrontal cortex) and higher-level visual representation areas in the occipital-temporal cortex mediates the capacity to efficiently detect and attend to facial expressions of emotions. The evidence reviewed here suggests that the amygdala and orbitofrontal cortex emerge in early infancy and are to a limited extent biased towards processing and storing information about certain biologically salient cues. Based on these findings, we propose that the early emergence of emotion-related brain systems, the initial biases in these systems, and the functional coupling of these systems with cortical perceptual areas that are supportive of more fine-grained perceptual processing and integration of different emotion-relevant cues (such as the STS, see BOX 2) provide a foundation for rapid acquisition of representations of species-typical facial expressions. Thus, the acquisition of representations of facial expressions may be based on a combination of initial biases in emotion-related neural systems and their experience-driven refinement, rather than on experience-independent maturation of a highly specialized system.

There are some indications that the emotion-relevant brain network may be particularly sensitive to the expected experience around the time of their functional emergence between 5 and 7 months of age. The amount of experiential input that is required for these systems to develop normally is not known but the evidence for similar development of emotion recognition in different cultures and even in severely deprived environments^{102, 103}, suggests that rudimentary perceptual representations of the universal features of facial expressions are acquired on the basis of limited environmental input. The rapid alterations of the response properties of neurons in the amygdala in response to stimulus-emotion associations²⁶ further suggests that emotion-processing networks can also quickly adapt to individual-specific experiences in the environment.

The proposal that rudimentary representations of some universal features of facial expressions are acquired early in life (possibly during a sensitive period) does not preclude the possibility that functional changes in emotion-processing networks occur later in childhood. Indeed, behavioral studies have shown age-related improvement throughout childhood in tasks that measure the ability to label facial expressions¹⁰⁴. Although such changes may be partly due to general cognitive improvement, they may also reflect functional changes in the brain network that underlies facial emotion processing. For example, fMRI studies in children and adolescents have shown age-related changes in childhood and adolescence in amygdala responses to facial expressions and in connections between amygdala and ventral prefrontal cortex (e.g., anterior cingulate cortex) ^{39, 105}. Amygdalo-prefrontal connections may be of critical importance for the ability to label facial expressions¹⁰⁶ and for the utilization of contextual information to modulate responses to facial expressions¹⁰⁷. Recent studies have also shown that cortical face-sensitive regions (fusiform gyrus) are relatively immature in 5–8 year old children and continue to specialize for face processing until adolescence ¹⁰⁸. Thus, although the basic connectivity pattern in the emotion-processing network and some of its response properties (e.g., differential response to neutral and fearful facial expressions)³⁹ may emerge early in life, other aspects of emotion processing such as those that include prefrontal-amygdala connections and fine-tuning of responses to specific facial expressions may continue to develop until adolescence (FIG. 4).

Figure 4. The proposed model of the development of emotion-recognition mechanisms.

It is proposed that the basic organization of the emotion-recognition networks is specified by an experience-expectant neural circuitry that emerges at 5–7 months of age and is rapidly refined by exposure to universal features of expressions during a sensitive period of development (perhaps the first few years of life). The network retains some plasticity throughout life span and can be fine-tuned by individual-specific experiences (i.e., experience-dependent development). Also, functional connectivity between emotion processing networks and other prefrontal regulatory systems continues to develop until adolescence. The development is affected by genetic factors (e.g., functional polymorphisms that affect reactivity of relevant neural systems), environmental factors (frequency of certain emotional expressions), and their interaction. The depicted time points might become more specific as more data become available.

Several interesting directions for future research emerge from this review. First, further investigation into the normative changes in perceptual-discrimination abilities over the course of the first year and the neural correlates of these changes will shed further light on the existence of sensitive periods in development during which the underlying neural mechanisms “expect” exposure to emotional expressions. Second, to understand better the early foundations of emotion recognition, an important goal for future studies is to elucidate the neural bases of emotion processing in infants and the exact stimulus features to which early developing emotion-processing networks are responsive. With the new developments in techniques that allow investigation of the brain basis of different cognitive functions in developing populations (e.g., high-density recordings of event-related potential and brain oscillations, near infrared spectroscopy), these questions are now more approachable than they were even a decade ago. Third, merging of molecular genetics, brain imaging methods, and psychological characterizations of critical environmental variables will shed new light on the developmental pathways through which the brain is shaped towards normal and vulnerable patterns of responding to social cues of emotions. Such studies will be important for elucidating the early precursors of vulnerability to emotional and social disorders later in life.