Skip to main content
Proceedings of the National Academy of Sciences of the United States of America logoLink to Proceedings of the National Academy of Sciences of the United States of America
. 2012 Jun 22;109(Suppl 1):10717–10724. doi: 10.1073/pnas.1204335109

Integration of faces and vocalizations in ventral prefrontal cortex: Implications for the evolution of audiovisual speech

Lizabeth M Romanski 1,1
PMCID: PMC3386875  PMID: 22723356

Abstract

The integration of facial gestures and vocal signals is an essential process in human communication and relies on an interconnected circuit of brain regions, including language regions in the inferior frontal gyrus (IFG). Studies have determined that ventral prefrontal cortical regions in macaques [e.g., the ventrolateral prefrontal cortex (VLPFC)] share similar cytoarchitectonic features as cortical areas in the human IFG, suggesting structural homology. Anterograde and retrograde tracing studies show that macaque VLPFC receives afferents from the superior and inferior temporal gyrus, which provide complex auditory and visual information, respectively. Moreover, physiological studies have shown that single neurons in VLPFC integrate species-specific face and vocal stimuli. Although bimodal responses may be found across a wide region of prefrontal cortex, vocalization responsive cells, which also respond to faces, are mainly found in anterior VLPFC. This suggests that VLPFC may be specialized to process and integrate social communication information, just as the IFG is specialized to process and integrate speech and gestures in the human brain.

Keywords: frontal lobe, sensory integration, primate, neurophysiology


The area dedicated to language processing in the frontal lobe is located within the inferior frontal gyrus (IFG), which can be further subdivided into the pars opercularis (most posterior portion of the IFG), the pars triangularis, and the pars orbitalis (cortex inferior and anterior to the horizontal ramus of the lateral fissure). These subdivisions include Brodmann areas 44, 45, and 47. Our understanding of the functions within this specialized area of cortex is hampered by the fact that no other mammal has a frontal lobe of similar organization or complexity, leaving few animal models to investigate. Within nonhuman primates, only catarrhines have a well-developed frontal lobe with cytoarchitectonic evidence of Brodmann areas 44, 45, and 47 (1). In contrast, New World monkeys including marmosets and squirrel monkeys have a lissencephalic frontal lobe with previously identified motor and premotor cortices but less clearly defined prefrontal regions (2). Recent work has identified an area in the ventrolateral prefrontal cortex (VLPFC) of rhesus macaques (Macaca mulatta) that is involved in the processing and integration of vocalizations and faces. We have hypothesized that the ventral prefrontal cortex (VLPFC) became specialized for the processing and integration of auditory and visual communication signals, in at least early anthropoid primates, and ultimately this region was modified and lateralized to the left cerebral hemisphere to subserve language in modern humans.

Ventral PFC: Anatomical Considerations

Organization of VLPFC.

The frontal lobe of the macaque monkey has been studied extensively with anatomical, electrophysiological, and functional methodologies compared with other primate species. The area of the VLPFC, also referred to as the inferior convexity of the PFC, in the macaque monkey, includes the cortical region ventral to the principal sulcus and anterior to the inferior limb of the arcuate sulcus (Fig. 1). The cytoarchitectonic areas of VLPFC in the macaque are arranged in a similar fashion to that of the human frontal lobe and include regions on the lateral frontal surface: area 45, which lies just anterior to the inferior arcuate sulcus; area 12 (or 12/47), which lies anterior to area 45 and ventral to area 46; and the most ventrolateral extent of the inferior convexity, which wraps around the inferior gyral surface and extends to the lateral orbital sulcus: area 12 orbital (3). Additional architectonic studies have described areas within the arcuate sulcus and premotor cortex as well as the subdivisions of the orbital cortex (1, 46), but we will confine our discussion to VLPFC, areas 45 and 12/47. We will refer to area 12 in general as area 12/47 to convey the homology of area 12 in the macaque with human area 47 as introduced by Petrides and Pandya (1). However, to differentiate between the ventrolateral region below the principal sulcus and the lateral orbital cortex we will use 12 ventrolateral (12vl) and 12 orbital (12o), respectively, as defined by Preuss and Goldman-Rakic (3).

Fig. 1.

Fig. 1.

Organization of ventral PFC. Maps of the cytoarchitectonic organization of ventral PFC are shown for the human (A and B) and macaque brain (CE). (A) Brodmann map (1909) of the human brain with areas 44, 45, and 47 marked on the IFG. Reproduced with permission from Brodmann (1909). (B) Map of the human brain color-coded to match the corresponding homologous regions in the macaque brain shown in D. Reproduced with permission from ref. 1. (C) Map of M. mulatta. Reproduced with permission from ref. 3. (D) Map of macaque brain. Reproduced with permission from ref. 1. (E) Photomicrograph of VLPFC in macaque monkey with cytoarchitectonic areas labeled. The IPD is marked in D and in E. Image courtesy of G. Luppino (ref. 5).

The VLPFC also commonly includes a small sulcus—termed the inferior frontal sulcus by Winters et al. (7) and the infraprincipal dimple or the inferior prefrontal dimple (IPD) by others (1, 8, 9)—which varies in its position and depth in M. mulatta. Some schematics of VLPFC depict the IPD as running in a rostral-to-caudal direction and separating area 45A from area 46 (1). However, in our neurophysiological recordings (10, 11) and in other studies (8), it is depicted as running dorsal to ventral and separating area 45 from area 12/47. It is not always described or visible in studies of other subspecies of macaque monkeys. Thus, there is variability in the position of area 12/47 and area 45 not only between the subspecies of macaques but also within M. mulatta individuals. As explained later, the IPD is the primary location in which auditory responsive neurons and audiovisual responsive cells have been reliably located in several studies, and may be a critical landmark for delineating the functional auditory responsive prefrontal region in macaques. Whether it defines the border of areas 12/47 and 45 is unclear.

In the human brain, areas 44 and 45 have been associated with language processing confirmed by electrical stimulation, PET and functional MRI (fMRI). However, the areas that control vocalization production in Old World monkeys are not as well understood and could include VLPFC, whereas other studies have implicated ventral premotor and the cingulate vocalization area (8, 12, 13).

Cytoarchitectonic Organization of VLPFC.

The cytoarchitectonic descriptions here are taken from Preuss and Goldman-Rakic (3), who described the frontal lobe of the rhesus macaque, M. mulatta, which is the same species that has been examined in most neurophysiology studies of VLPFC. These descriptions are in general agreement with Petrides and Pandya (1). Area 45 is located ventral to the caudal principal sulcus within the ventral limb of the arcuate and extends onto the cortical surface (Fig. 1 CE). It is composed of large pyramidal cells in layer V and deep layer III. Layer IV is thick with densely packed small cells, with some of the larger pyramidal cells from deep layer III and superficial layer V intruding on layer IV. It is densely myelinated. Area 45 is bordered dorsally by area 8a (3), which can be distinguished from area 45 by the presence of extremely large pyramidal cells in layer Va of area 8. Area 12 (areas 12vl and 12o), which covers the surface of the ventrolateral convexity and extends onto the lateral orbital surface as far as the lateral orbital sulcus, can be distinguished from area 46 by its more heavily myelinated appearance. The disappearance of the large layer III pyramidal cells marks the transition from area 45 to area 12vl. Area 12vl on the ventrolateral surface has been distinguished from area 12o by the more diffuse myelinated appearance of 12o and the more granular layer IV of 12vl. A series of comparative cytoarchitectonic studies have examined the similarities of area 12 in macaque and area 47 in the human brain (1). As a result, area 12 has been referred to as area 12/47 even though the assignation of 12 was renewed in a recent analysis of VLPFC connections (5). In addition, studies by Petrides and Pandya (1) and Gerbella et al. (5) have suggested that area 45 be divided into subdivisions 45B, closest to and within the anterior limb of the anterior bank of the arcuate sulcus, and area 45A, located rostral to 45B and extending across the surface of the inferior convexity to the IPD, with area 12vl (i.e., 47) bordering 45A rostrally and ventrally (Fig. 1). The precise location of area 12vl relative to area 45A may vary somewhat in individuals and may be better determined from a combination of connectivity studies and physiological recordings.

Cortical Connectivity of VLPFC.

Connectivity of VLPFC with cortical visual processing regions.

Much of what we know about the cellular functions of the primate PFC is based on the processing of visual information. Thus, it is not surprising that many studies have examined projections from visual association cortex to the primate PFC. Results indicate that VLPFC receives afferents from extrastriate visual cortical areas in the inferotemporal cortex, including area TE. Early anatomical studies by Barbas, Pandya, and others examined the innervation of the entire prefrontal mantle by visual association areas (1416). Barbas was among the first to note that basoventral prefrontal cortices were more strongly connected with extrastriate, ventral visual areas, which have been implicated in object recognition and feature discrimination. In contrast, medial and dorsal prefrontal cortices are more densely connected with medial and dorsolateral occipital and parietal areas, which are associated with visuospatial functions (14). This dissociation was confirmed by Bullier et al. (17), who found a segregation of inputs to caudal PFC when paired injections of tracers were placed into temporal and parietal visual processing regions. In their study, visual temporal cortex projected mainly to ventrolateral PFC, area 45, whereas parietal cortex sent projections to ventrolateral PFC and dorsolateral PFC (areas 8a and 46) (17). Tracing and lesion studies by Ungerleider et al. (18) showed that area TE projected specifically to three ventral prefrontal targets, including the ventral limb of the arcuate sulcus (area 45), the inferior convexity just ventral to the principal sulcus (area 12vl), and within the lateral orbital cortex (areas 11 and 12o). These projections are via the uncinate fasciculus (18). Furthermore, ventrolateral PFC areas 12vl and 45, which contain object- and face-selective neurons (1921), were shown to be connected with inferotemporal areas TE and TEO (22), with the strongest innervation of ventrolateral PFC and orbitofrontal areas 11 and 12o originating in TE.

Auditory projections to PFC.

In contrast to the visual pathways, the prefrontal targets of central auditory pathways have not been studied as extensively despite the accepted role of the frontal lobe in language. In early anatomical studies, lesion/degeneration techniques were used to reveal projections from the caudal superior temporal gyrus (STG) to the principal sulcus region, the arcuate cortex, and the inferior convexity of the frontal lobe, and from the middle and rostral STG to the rostral principal sulcus and orbital regions (16, 23, 24). Additional studies revealed connections between the lateral PFC and cortical areas within the STG (14, 15, 25, 26). There was a suggestion of rostrocaudal topography in these studies whereby anterior and middle aspects of the principal sulcus, including areas 9, 10, and rostral 46, were connected with the middle STG, whereas area 8 received projections from mostly caudal STG (14, 26, 27). It became clear that VLPFC received afferents from the STG, inferotemporal cortex, and multisensory regions within the superior temporal sulcus (STS).

Importantly, detailed anatomical studies by Morel et al. (28), Jones et al. (29), and Hackett et al. (30), together with parallel neurophysiological studies by Rauschecker et al. (31), provided evidence that primate auditory cortices were organized as a core-belt system with a third zone, the parabelt just lateral to the belt (2830). A series of landmark neurophysiology studies provided the first electrophysiological evidence for three separate tonotopic regions (AL, ML, and CL) in the belt cortex that could be distinguished from the core A1 (31). Additional studies have described functional dissociations of anterior and posterior belt and parabelt regions (32). Rauschecker et al. (31), Morel et al. (28), and Hackett et al. (30) used a common terminology to delineate auditory cortex in anatomical and physiological studies, which enabled cross-talk and comparisons that fostered progress in the study of auditory cortical processing and organization.

Combining physiological recording with anatomical tract tracing, Romanski et al. (33) analyzed the connections of physiologically defined areas of the belt and parabelt auditory cortex. They determined that rostral and ventral PFC receives projections from the anterior auditory association cortex (areas AL and anterior parabelt) and caudal prefrontal regions are innervated by posterior auditory cortex (areas CL and caudal parabelt; Fig. 2). Together with auditory physiological recordings from the lateral belt (32) and from the PFC (10, 11), these studies suggest that separate auditory streams originate in the caudal and rostral auditory cortex and target dorsolateral spatial and anterior-ventrolateral object domains in the frontal lobe, respectively (34). This is similar to the dorsal and ventral streams described for the visual system (35). Ultimately, this also implies that auditory and visual afferents target similar functional domains of dorsal and ventral PFC (11). The convergence of auditory and visual ventral stream inputs to the same VLPFC domain suggests that they may be integrated and combined to serve a similar function, for example that of object recognition, which is aided by the integration of multiple sensory inputs.

Fig. 2.

Fig. 2.

Dual streams of auditory afferents target the PFC. (A) The auditory cortex core (A1) surrounded by the belt (box delineates the lateral belt cortex shown in B). Injections placed into the auditory belt cortex at similar frequency mapped locations in AL (red), ML (green), and CL (blue) are shown as colored polygons. (C) The resulting labeled retrograde cells and anterograde fibers are shown in three coronal sections through the PFC from rostral to caudal, with the same color coding as in B to indicate the source of the injections. (D) Summary of projections. Rostral and VLPFC receives stronger innervation from the anterior belt and adjacent parabelt regions whereas dorsolateral PFC receives the greatest innervation from caudal auditory belt and parabelt regions. Adapted from ref. 33.

Examination of the connections of VLPFC without accompanying physiology has suggested that area 45A receives greater inputs from the STG than from inferotemporal cortex (1, 5). This is in contrast to previous analysis of anterograde projections of the STG and of inferotemporal cortex. These anterograde studies suggest that STG and STS innervate area 12/47 whereas inferotemporal and STS cortex project to area 45 and area 12/47. Much of the debate appears centered on where the boundary between area 45 and area 12/47 occurs, and may be clarified with additional neurophysiological recordings and combined anatomical connectivity studies.

Functional Studies of VLPFC

Visual Processing in VLPFC.

Decades of research have demonstrated the frontal lobe’s involvement in cognitive functions including working memory, decision-making, and social communication processes such as language and face–voice processing. Single unit recording studies in animal models have characterized dorsolateral prefrontal cortex (DLPFC) neuronal involvement in visuospatial processing, saccadic eye movements, and working memory (3640). Further investigations have emphasized a process-oriented role for DLPFC and have described single-unit activity of prefrontal neurons during decision-making, categorization, numerosity, and the coding of abstract rules (4144).

In contrast, investigation of the cellular activity in the VLPFC has focused on object processing and social communication. Early studies of VLPFC showed that neurons in this region were responsive to simple and complex visual stimuli presented at the fovea (45, 46). Face-responsive neurons were documented by Thorpe et al. and Rolls et al. (47, 48) and later described in detail by Goldman-Rakic and coworkers (1921). In these studies, Wilson et al. (21) showed that DLPFC and VLPFC neurons responded differentially to spatial and object features of visual stimuli. These studies were the first to demonstrate a functional dissociation between DLPFC and VLPFC by using single-unit electrophysiology. Wilson et al. (21) showed that DLPFC neurons were selectively engaged by visuospatial memory tasks and VLPFC neurons were selective for color, shape, or type of visual objects. An earlier study by Mishkin and Manning (49) showed that lesions of VLPFC in nonhuman primates interfere with the processing of nonspatial information, including color and form. Electrophysiological recordings demonstrated that VLPFC face cells had a twofold increase in firing rate to face stimuli compared with nonface stimuli during passive presentations or during working memory tasks (19, 20). Face cells were found only in the VLPFC and not in DLPFC, and were localized to three small parts of VLPFC, including a patch on the lateral convexity close to the lower limb of the arcuate sulcus (area 45), within and around the IPD (area 12vl), and a small number of cells in the lateral orbital cortex (19). VLPFC face cells were sensitive to changes in facial features, expressions, or the angle of gaze, much like the inferotemporal cortical regions, which project to these VLPFC cells. These studies have suggested that VLPFC cells may encode identity, expression, and face view (19, 20, 48, 50). Data from the single-unit recordings have been confirmed with fMRI studies in macaque monkeys (51), which have demonstrated activation of face-responsive “patches” in the same arcuate, ventrolateral, and orbitofrontal locations shown by O’Scalaidhe et al. (19, 20). Demonstration by both methods of visual responsiveness and face selectivity substantiates the notion that VLPFC in the macaque monkey is involved in object and face processing (Fig. 3).

Fig. 3.

Fig. 3.

Face-responsive neurons in the VLPFC. (A and B) Face-responsive neurons recorded by O’Scalaidhe et al. (19) are depicted. Adapted from ref. 19. (A) Region recorded in the PFC is indicated with a circle on the lateral brain schematic of the rhesus macaque brain. (B) Flat map of the recorded region in the PFC is shown with face cells outlined in black and gray. (C) Face responsive cells that were selective for forward and 30° rotated face view (50). Stimulus images courtesy of Michael J. Tarr, Center for the Neural Basis of Cognition and Department of Psychology, Carnegie Mellon University, http://www.tarrlab.org/. (D) Location of face selective patches in the VLPFC and in the orbitofrontal cortex is shown in yellow (51). A red arrow indicates a similar portion of the VLPFC in the single unit data portrayed in B and in the fMRI data in D. Reprinted by permission from Macmillan Publishers Ltd: Nature Neuroscience (ref. 51), copyright (2008).

Auditory Responses and Function in Ventral PFC.

The ventral frontal lobe has long been linked with complex auditory function through its association with language functions in the IFG. The results of some studies have suggested parcellation of function in the human IFG. The anterior region, the pars triangularis (area 45), along with the pars orbitalis (area 47), has been suggested to be more involved in semantic processing, comprehension, and auditory working memory (5259). In contrast, the pars opercularis (area 44) and ventral premotor cortex are more active during phonological processing and speech production. The precise neuronal mechanisms that occur in the frontal lobe during the processing of complex auditory information are unknown but might be indirectly assessed with neurophysiological recordings in animals with similar ventral frontal lobe regions, such as macaque monkeys.

Neuronal responses to acoustic stimuli have been sporadically noted in the frontal lobes of Old and New World monkeys (6062). However, when recordings targeted cortical areas that had been shown to receive projections from acoustically characterized regions of the auditory belt and parabelt cortex (33), a discrete auditory responsive region was localized in VLPFC (10). This VLPFC cortical region is thought to be the termination of a ventral auditory processing stream, specialized for the processing of nonspatial (i.e., object) auditory information (33, 34, 63, 64). The auditory responsive region of VLPFC is located rostral to the ventral limb of the arcuate sulcus below the principal sulcus, in the area of the IPD. This region receives projections from ventral stream auditory cortical regions and polymodal cortex of the STS, as discussed earlier (1, 4, 33, 65, 66). VLPFC auditory neurons are responsive to complex auditory stimuli, including vocalization and complex nonvocal stimuli (10). This small ventrolateral prefrontal auditory region has also been shown to be active in neuroimaging studies in rhesus monkeys during presentation of complex acoustic stimuli (67).

The VLPFC auditory area was analyzed with a large library of rhesus macaque vocalizations to test selectivity to specific call categories, as previous analysis had implied some selectivity for calls with common functions (68). Analysis of these vocalization responses with exemplars from 10 different types of calls demonstrated that neurons tended to respond to two or three vocalization types that had similar acoustic morphology rather than similar behavioral referents (Fig. 4; 11). Additional electrophysiological recording studies by Gifford et al. (68) and Russ et al. (69) have suggested that VLPFC neuronal activity is modulated during categorization of acoustic stimuli and in auditory decision-making (70). These combined data are consistent with a role for VLPFC in a ventral auditory processing stream for auditory objects, including vocalizations. The localization of this auditory processing area to the ventral prefrontal region of Old World monkeys suggests a functional similarity between it and human language-processing regions in the ventral or inferior frontal lobe of the human brain (10, 71, 72).

Fig. 4.

Fig. 4.

Auditory responsive neurons in VLPFC. A single-cell example of responses to four different vocalization stimuli is shown in the top part of the figure. The response is shown as raster and shaded spike density function in response to a “grunt” vocalization (low-value food call), an aggressive “pant threat” vocalization, a “coo” vocalization (low-value food and affiliative) and a harmonic arch (high-value food). The waveforms of the calls are shown below the rasters. A schematic of the PFC is shown indicating the locations of auditory responsive neurons. Adapted from ref. 11.

Multisensory Responses in VLPFC.

The initial physiological studies of VLPFC suggested that auditory and visual object processing regions were located adjacent to one another in VLFPC. Any overlap in these auditory and visual responsive zones could thus be sites for multisensory integration of complex auditory and visual information. As neurons in this region are face- and vocalization-responsive, multisensory neurons in the macaque VLPFC might integrate face and vocal information. Given that the percentage of neurons responsive to visual stimuli was much greater than the number of auditory responsive cells (55% vs. 18%), we reasoned that multisensory cells are more likely to be located in regions where auditory cells had been recorded and predicted that multisensory neurons might be found only in this region. In our neurophysiological investigation, we presented movies of familiar monkeys vocalizing to macaque monkeys while single neurons were recorded from the VLPFC (73). These movies were separated into audio and video streams, and neural responses to the unimodal stimuli were compared with the responses to the combined audiovisual stimuli. Interestingly, approximately half the neurons recorded in the VLPFC were multisensory in that they responded to unimodal auditory and visual stimuli, i.e., bimodal responses; or were multisensory because of an enhanced or decreased response to the combined audiovisual stimulus (face and vocalization) compared with the response to the unimodal stimuli (73). This is likely to be an underestimate of the percentage of multisensory responses because we used a limited set of audiovisual stimuli and neurons were found to be selective for particular face–vocalization pairs.

VLPFC neurons exhibited multisensory enhancement or suppression (Fig. 5) just as neurons do in the superior colliculus, the STS, and auditory cortex during multisensory integration (7477). It was also interesting that face/voice stimuli evoked multisensory responses more frequently than nonface/nonvoice audiovisual stimuli. This adds support to the notion that VLPFC is part of a circuit that is specialized for the integration of social communication information rather than sensory stimuli in a general sense. In localizing these multisensory responses to the PFC, there appeared to be two somewhat separate VLPFC regions for multisensory processing. Interestingly, these two separate clusters of multisensory neurons overlap with two prefrontal face patches described by O’Scalaidhe et al. (19) and Tsao et al. (51) in the arcuate and ventrolateral PFC areas. In our study, there was a large pool of unimodal visual neurons with a small number of multisensory cells located in posterior VLPFC (area 45). Unimodal neurons in this area are mostly visual and respond to faces and nonface stimuli such as objects, shapes, and patterns. The multisensory neurons in this arcuate region (Fig. 5) have strong visual responses modulated by the simultaneous presentation of auditory stimuli. There are strong projections to this area from the inferotemporal cortex and the polymodal STS, which have been associated with the processing of facial identity and facial expression. Previous studies in nonhuman primates of visual working memory, decision-making, and visual search (21, 41, 42) have noted responsive neurons within this arcuate region as well as in the more commonly recorded principal sulcus region.

Fig. 5.

Fig. 5.

Multisensory neurons in the VLPFC. The responses of two single units are shown in A and B as raster/spike density plots to an auditory stimulus alone (a vocalization, AUD) and face (Vis) and both presented simultaneously (AV). Right: Bar graph of mean response to these stimuli. Cell in A exhibited multisensory enhancement and cell in B exhibited multisensory suppression. Locations where multisensory stimuli (red squares), visual responses (yellow diamonds), and auditory responses (blue circles) are depicted on lateral view of frontal lobe in D, with the inset shown larger in C. Adapted from ref. 63.

A smaller, potentially more specialized pool of multisensory neurons is located in VLPFC, anterior and lateral to the first pool (Fig. 5). These neurons are found near the IPD, and within its banks in area 12vl of Preuss and Goldman-Rakic (3). This is the region where unimodal auditory responsive neurons were predominantly localized in previous studies (10, 11). These anterolateral VLPFC neurons respond to vocalizations and to faces, but only weakly to other visual stimuli (10, 73). This area receives afferents from mainly polymodal STS cortical regions and also from auditory association cortex, including a small amount of afferents from the belt, more from the parabelt, and the largest contribution from the rostral temporal lobe (33, 65, 66). Multisensory responses here favor vocalizations and their corresponding faces, suggesting a more specialized role in the integration of social communication information. Face responsive cells recorded in this area, which were selective for forward gaze, such as that which occurs in face-to-face communication, were also more likely to be auditory responsive (50). Most of the multisensory neurons exhibited suppression rather than enhancement. This nonlinear interaction has also been noted in auditory cortex (76, 78). This anterolateral pool of multisensory neurons may be specialized for the integration of social communication sounds with facial gestures and other communication oriented information. In contrast, the more posterior multisensory neurons may serve a more general integrative purpose.

Audiovisual Integration in Human Brain and Implications for Evolution of Audiovisual Speech

How does this compare with the ventral frontal lobe in the human brain? Whereas many associate the human IFG with only spoken language and verbal processing, communication is, in fact, a multisensory process. Several well-known illusions owe their effects to specific aspects of multisensory integration, including the McGurk and ventriloquist effects (79, 80). Although crossmodal integration takes place over a network of areas in the brain (81, 82), the same areas that underlie speech and language processing in the temporal and frontal cortex play an essential role in the integration of audiovisual communication information. The STS and the ventral frontal lobe are both sites of activation during the processing and integration of speech and gestures (8386). In an fMRI study of speech and gesture, Xu et al. found overlap of activation in two regions of the IFG when subjects viewed gestures or listened to a voicing of the phrase that fit the gesture (87). The activated regions included a large cluster in the pars triangularis and pars opercularis (areas 44 and 45) and a smaller focal cluster in pars orbitalis, area 47 (Fig. 6). Xu et al. (87) argued that the IFG most likely plays a larger role in communication than classical auditory-speech processing and theorized that linking meaning with acoustic or visual symbols may be the essential function of these inferior frontal regions.

Fig. 6.

Fig. 6.

Convergence of speech and gesture responsive regions in the human frontal lobe. Shown are three activation clusters that were active for speech and gesture conditions. There is a large activation cluster in the posterior temporal lobe and in the IFG areas 44 and 45 and a small focal activation rostrally in area 47. Reproduced from ref. 87.

Thus, the process of linking, or integrating phonological constructs with auditory objects, results in the perception of spoken words, whereas integration of a visual image of letters with their learned meanings conveys the concept of a word. Integration might then be one of many basic processes the human frontal lobe performs during speech, language, and communication. The linking, or integrating of face and vocal information, in the macaque monkey frontal lobe could be seen as a precursor to the more complex functions that the IFG performs in the human brain whereby abstract concepts are united with images and sounds. In the human brain, words, sounds, gestures, and visual images are each integrated with meaning and with each other (87). In more primitive primates, such as macaques, in which abstraction is not likely to occur, faces are integrated with vocalizations. It has been previously suggested that the ventral PFC is essential in associating a visual cue with an action (88). Our data suggest that VLPFC may associate auditory cues with gestural actions, which is necessary during communication.

Depending on the aspects of the stimuli that are integrated, several communication-relevant functions may be accomplished through integration. In humans, adding mouth movements or facial expressions to spoken words can clarify or even alter the meaning of an utterance (79). Incongruency between face identity and a voice or between a facial expression and a vocal sound is detected by humans and activates prefrontal and temporal cortical regions. Some human neuroimaging studies have demonstrated a decrease in ventral prefrontal activity for incongruent faces and voices (85, 86, 89). Others report increased activations during incongruent stimuli (9092). Recordings of macaque VLPFC neurons show that incongruent stimuli also evoke an increase or a decrease in neuronal activity depending on the original response to bimodal stimuli (50). The sign of the neuronal response may also be affected by facial features and emotional valence of the audiovisual stimuli.

Identity, or recognition, is another process which greatly benefits from the integration of face and vocal information (reviewed in ref. 93) and studies have shown that animals match faces and corresponding voices as we do (94, 95). The circuit for the processing of face identity includes the cortex within the STS and inferotemporal cortex, and single cells in these areas respond to facial identity and facial expression (96, 97). How multisensory neurons in the STS integrate face and vocalization information to enhance recognition is not known at the single-cell level even though pairing of incongruent faces and vocalizations alters activity in this region. The STS has a robust connection with VLPFC and is likely to send unimodal and multisensory identity information to VLPFC neurons. Selectivity of face-responsive cells in VLPFC has been shown for particular individuals, expressions, or categories of face stimuli (19, 20, 48, 50).

The accumulation of evidence to date shows that cells in the ventral PFC of the macaque monkey respond to and integrate audiovisual information. VLPFC cells respond optimally to face and vocalization stimuli and exhibit multisensory enhancement or suppression when face-vocalization stimuli are combined. Thus, the ventral frontal lobe of nonhuman primates may have some basic functional homologies to the human frontal lobe, although more evidence from additional primate species is needed. The basic process of associating a face, or facial gesture, with a vocal stimulus, which occurs in the macaque PFC, may be a precursor to the more complex functions of the human frontal lobe, where semantic meaning is linked with acoustic or visual symbols.

Footnotes

The author declares no conflict of interest.

This paper results from the Arthur M. Sackler Colloquium of the National Academy of Sciences, “In the Light of Evolution VI: Brain and Behavior,” held January 19–21, 2012, at the Arnold and Mabel Beckman Center of the National Academies of Sciences and Engineering in Irvine, CA. The complete program and audio files of most presentations are available on the NAS Web site at www.nasonline.org/evolution_vi.

This article is a PNAS Direct Submission.

References

  • 1.Petrides M, Pandya DN. Comparative cytoarchitectonic analysis of the human and the macaque ventrolateral prefrontal cortex and corticocortical connection patterns in the monkey. Eur J Neurosci. 2002;16:291–310. doi: 10.1046/j.1460-9568.2001.02090.x. [DOI] [PubMed] [Google Scholar]
  • 2.Preuss T. In: Primate Origins: Adaptations and Evolution. Ravosa M, Dagosto M, editors. New York: Springer; 2007. [Google Scholar]
  • 3.Preuss TM, Goldman-Rakic PS. Myelo- and cytoarchitecture of the granular frontal cortex and surrounding regions in the strepsirhine primate Galago and the anthropoid primate Macaca. J Comp Neurol. 1991;310:429–474. doi: 10.1002/cne.903100402. [DOI] [PubMed] [Google Scholar]
  • 4.Carmichael ST, Price JL. Sensory and premotor connections of the orbital and medial prefrontal cortex of macaque monkeys. J Comp Neurol. 1995;363:642–664. doi: 10.1002/cne.903630409. [DOI] [PubMed] [Google Scholar]
  • 5.Gerbella M, Belmalih A, Borra E, Rozzi S, Luppino G. Cortical connections of the macaque caudal ventrolateral prefrontal areas 45A and 45B. Cereb Cortex. 2010;20:141–168. doi: 10.1093/cercor/bhp087. [DOI] [PubMed] [Google Scholar]
  • 6.Saleem KS, Kondo H, Price JL. Complementary circuits connecting the orbital and medial prefrontal networks with the temporal, insular, and opercular cortex in the macaque monkey. J Comp Neurol. 2008;506:659–693. doi: 10.1002/cne.21577. [DOI] [PubMed] [Google Scholar]
  • 7.Winters W, Kado R, Adey W. A Stereotaxic Brain Atlas for Macaca nemestrina. Los Angeles: Univ California Press; 1969. [Google Scholar]
  • 8.Petrides M, Cadoret G, Mackey S. Orofacial somatomotor responses in the macaque monkey homologue of Broca’s area. Nature. 2005;435:1235–1238. doi: 10.1038/nature03628. [DOI] [PubMed] [Google Scholar]
  • 9.Paxinos G, Huang X, Toga AW. The Rhesus Monkey Brain. San Diego: Academic; 2000. [Google Scholar]
  • 10.Romanski LM, Goldman-Rakic PS. An auditory domain in primate prefrontal cortex. Nat Neurosci. 2002;5:15–16. doi: 10.1038/nn781. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Romanski LM, Averbeck BB, Diltz M. Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. J Neurophysiol. 2005;93:734–747. doi: 10.1152/jn.00675.2004. [DOI] [PubMed] [Google Scholar]
  • 12.Jürgens U. The neural control of vocalization in mammals: A review. J Voice. 2009;23:1–10. doi: 10.1016/j.jvoice.2007.07.005. [DOI] [PubMed] [Google Scholar]
  • 13.Coudé G, et al. Neurons controlling voluntary vocalization in the macaque ventral premotor cortex. PLoS ONE. 2011;6:e26822. doi: 10.1371/journal.pone.0026822. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Barbas H. Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. J Comp Neurol. 1988;276:313–342. doi: 10.1002/cne.902760302. [DOI] [PubMed] [Google Scholar]
  • 15.Barbas H, Pandya DN. Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey. J Comp Neurol. 1989;286:353–375. doi: 10.1002/cne.902860306. [DOI] [PubMed] [Google Scholar]
  • 16.Chavis DA, Pandya DN. Further observations on corticofrontal connections in the rhesus monkey. Brain Res. 1976;117:369–386. doi: 10.1016/0006-8993(76)90089-5. [DOI] [PubMed] [Google Scholar]
  • 17.Bullier J, Schall JD, Morel A. Functional streams in occipito-frontal connections in the monkey. Behav Brain Res. 1996;76:89–97. doi: 10.1016/0166-4328(95)00182-4. [DOI] [PubMed] [Google Scholar]
  • 18.Ungerleider LG, Gaffan D, Pelak VS. Projections from inferior temporal cortex to prefrontal cortex via the uncinate fascicle in rhesus monkeys. Exp Brain Res. 1989;76:473–484. doi: 10.1007/BF00248903. [DOI] [PubMed] [Google Scholar]
  • 19.O Scalaidhe SP, Wilson FA, Goldman-Rakic PS. Areal segregation of face-processing neurons in prefrontal cortex. Science. 1997;278:1135–1138. doi: 10.1126/science.278.5340.1135. [DOI] [PubMed] [Google Scholar]
  • 20.Scalaidhe SP, Wilson FAW, Goldman-Rakic PS. Face-selective neurons during passive viewing and working memory performance of rhesus monkeys: Evidence for intrinsic specialization of neuronal coding. Cereb Cortex. 1999;9:459–475. doi: 10.1093/cercor/9.5.459. [DOI] [PubMed] [Google Scholar]
  • 21.Wilson FA, Scalaidhe SP, Goldman-Rakic PS. Dissociation of object and spatial processing domains in primate prefrontal cortex. Science. 1993;260:1955–1958. doi: 10.1126/science.8316836. [DOI] [PubMed] [Google Scholar]
  • 22.Webster MJ, Bachevalier J, Ungerleider LG. Connections of inferior temporal areas TEO and TE with parietal and frontal cortex in macaque monkeys. Cereb Cortex. 1994;4:470–483. doi: 10.1093/cercor/4.5.470. [DOI] [PubMed] [Google Scholar]
  • 23.Pandya DN, Hallett M, Kmukherjee SK. Intra- and interhemispheric connections of the neocortical auditory system in the rhesus monkey. Brain Res. 1969;14:49–65. doi: 10.1016/0006-8993(69)90030-4. [DOI] [PubMed] [Google Scholar]
  • 24.Jones EG, Powell TP. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain. 1970;93:793–820. doi: 10.1093/brain/93.4.793. [DOI] [PubMed] [Google Scholar]
  • 25.Galaburda AM, Pandya DN. The intrinsic architectonic and connectional organization of the superior temporal region of the rhesus monkey. J Comp Neurol. 1983;221:169–184. doi: 10.1002/cne.902210206. [DOI] [PubMed] [Google Scholar]
  • 26.Barbas H, Mesulam MM. Cortical afferent input to the principalis region of the rhesus monkey. Neuroscience. 1985;15:619–637. doi: 10.1016/0306-4522(85)90064-8. [DOI] [PubMed] [Google Scholar]
  • 27.Petrides M, Pandya DN. Association fiber pathways to the frontal cortex from the superior temporal region in the rhesus monkey. J Comp Neurol. 1988;273:52–66. doi: 10.1002/cne.902730106. [DOI] [PubMed] [Google Scholar]
  • 28.Morel A, Garraghty PE, Kaas JH. Tonotopic organization, architectonic fields, and connections of auditory cortex in macaque monkeys. J Comp Neurol. 1993;335:437–459. doi: 10.1002/cne.903350312. [DOI] [PubMed] [Google Scholar]
  • 29.Jones EG, Dell’Anna ME, Molinari M, Rausell E, Hashikawa T. Subdivisions of macaque monkey auditory cortex revealed by calcium-binding protein immunoreactivity. J Comp Neurol. 1995;362:153–170. doi: 10.1002/cne.903620202. [DOI] [PubMed] [Google Scholar]
  • 30.Hackett TA, Stepniewska I, Kaas JH. Thalamocortical connections of the parabelt auditory cortex in macaque monkeys. J Comp Neurol. 1998;400:271–286. doi: 10.1002/(sici)1096-9861(19981019)400:2<271::aid-cne8>3.0.co;2-6. [DOI] [PubMed] [Google Scholar]
  • 31.Rauschecker JP, Tian B, Hauser M. Processing of complex sounds in the macaque nonprimary auditory cortex. Science. 1995;268:111–114. doi: 10.1126/science.7701330. [DOI] [PubMed] [Google Scholar]
  • 32.Tian B, Reser D, Durham A, Kustov A, Rauschecker JP. Functional specialization in rhesus monkey auditory cortex. Science. 2001;292:290–293. doi: 10.1126/science.1058911. [DOI] [PubMed] [Google Scholar]
  • 33.Romanski LM, et al. Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci. 1999;2:1131–1136. doi: 10.1038/16056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Romanski LM. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb Cortex. 2007;17(suppl 1):i61–i69. doi: 10.1093/cercor/bhm099. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Ungerleider LG, Mishkin M. Two cortical visual systems. In: Ingle DJ, Goodale MA, Mansfield RJW, editors. In Analysis of Visual Behavior. Cambridge, MA: MIT Press; 1982. pp. 549–586. [Google Scholar]
  • 36.Bruce CJ, Goldberg ME. Primate frontal eye fields. I. Single neurons discharging before saccades. J Neurophysiol. 1985;53:603–635. doi: 10.1152/jn.1985.53.3.603. [DOI] [PubMed] [Google Scholar]
  • 37.Funahashi S, Bruce CJ, Goldman-Rakic PS. Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex. J Neurophysiol. 1989;61:1–19. doi: 10.1152/jn.1989.61.2.331. [DOI] [PubMed] [Google Scholar]
  • 38.Funahashi S, Chafee MV, Goldman-Rakic PS. Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature. 1993;365:753–756. doi: 10.1038/365753a0. [DOI] [PubMed] [Google Scholar]
  • 39.Quintana J, Fuster JM. Mnemonic and predictive functions of cortical neurons in a memory task. Neuroreport. 1992;3:721–724. doi: 10.1097/00001756-199208000-00018. [DOI] [PubMed] [Google Scholar]
  • 40.Chafee MV, Goldman-Rakic PS. Matching patterns of activity in primate prefrontal area 8a and parietal area 7ip neurons during a spatial working memory task. J Neurophysiol. 1998;79:2919–2940. doi: 10.1152/jn.1998.79.6.2919. [DOI] [PubMed] [Google Scholar]
  • 41.Kim JN, Shadlen MN. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nat Neurosci. 1999;2:176–185. doi: 10.1038/5739. [DOI] [PubMed] [Google Scholar]
  • 42.Freedman DJ, Miller EK. Neural mechanisms of visual categorization: Insights from neurophysiology. Neurosci Biobehav Rev. 2008;32:311–329. doi: 10.1016/j.neubiorev.2007.07.011. [DOI] [PubMed] [Google Scholar]
  • 43.Nieder A, Freedman DJ, Miller EK. Representation of the quantity of visual items in the primate prefrontal cortex. Science. 2002;297:1708–1711. doi: 10.1126/science.1072493. [DOI] [PubMed] [Google Scholar]
  • 44.Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Annu Rev Neurosci. 2001;24:167–202. doi: 10.1146/annurev.neuro.24.1.167. [DOI] [PubMed] [Google Scholar]
  • 45.Rosenkilde CE, Bauer RH, Fuster JM. Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Res. 1981;209:375–394. doi: 10.1016/0006-8993(81)90160-8. [DOI] [PubMed] [Google Scholar]
  • 46.Suzuki H, Azuma M. Topographic studies on visual neurons in the dorsolateral prefrontal cortex of the monkey. Exp Brain Res. 1983;53:47–58. doi: 10.1007/BF00239397. [DOI] [PubMed] [Google Scholar]
  • 47.Thorpe SJ, Rolls ET, Maddison S. The orbitofrontal cortex: Neuronal activity in the behaving monkey. Exp Brain Res. 1983;49:93–115. doi: 10.1007/BF00235545. [DOI] [PubMed] [Google Scholar]
  • 48.Rolls ET, Critchley HD, Browning AS, Inoue K. Face-selective and auditory neurons in the primate orbitofrontal cortex. Exp Brain Res. 2006;170:74–87. doi: 10.1007/s00221-005-0191-y. [DOI] [PubMed] [Google Scholar]
  • 49.Mishkin M, Manning FJ. Non-spatial memory after selective prefrontal lesions in monkeys. Brain Res. 1978;143:313–323. doi: 10.1016/0006-8993(78)90571-1. [DOI] [PubMed] [Google Scholar]
  • 50.Romanski LM, Diehl MM. Neurons responsive to face-view in the primate ventrolateral prefrontal cortex. Neuroscience. 2011;189:223–235. doi: 10.1016/j.neuroscience.2011.05.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Tsao DY, Schweers N, Moeller S, Freiwald WA. Patches of face-selective cortex in the macaque frontal lobe. Nat Neurosci. 2008;11:877–879. doi: 10.1038/nn.2158. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Démonet JF, et al. The anatomy of phonological and semantic processing in normal subjects. Brain. 1992;115:1753–1768. doi: 10.1093/brain/115.6.1753. [DOI] [PubMed] [Google Scholar]
  • 53.Paulesu E, Frith CD, Frackowiak RSJ. The neural correlates of the verbal component of working memory. Nature. 1993;362:342–345. doi: 10.1038/362342a0. [DOI] [PubMed] [Google Scholar]
  • 54.Buckner RL, Raichle ME, Petersen SE. Dissociation of human prefrontal cortical areas across different speech production tasks and gender groups. J Neurophysiol. 1995;74:2163–2173. doi: 10.1152/jn.1995.74.5.2163. [DOI] [PubMed] [Google Scholar]
  • 55.Demb JB, et al. Semantic encoding and retrieval in the left inferior prefrontal cortex: A functional MRI study of task difficulty and process specificity. J Neurosci. 1995;15:5870–5878. doi: 10.1523/JNEUROSCI.15-09-05870.1995. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Stromswold K, Caplan D, Alpert N, Rauch S. Localization of syntactic comprehension by positron emission tomography. Brain Lang. 1996;52:452–473. doi: 10.1006/brln.1996.0024. [DOI] [PubMed] [Google Scholar]
  • 57.Price CJ. The functional anatomy of word comprehension and production. Trends Cogn Sci. 1998;2:281–288. doi: 10.1016/s1364-6613(98)01201-7. [DOI] [PubMed] [Google Scholar]
  • 58.Gelfand JR, Bookheimer SY. Dissociating neural mechanisms of temporal sequencing and processing phonemes. Neuron. 2003;38:831–842. doi: 10.1016/s0896-6273(03)00285-x. [DOI] [PubMed] [Google Scholar]
  • 59.Poldrack RA, et al. Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage. 1999;10:15–35. doi: 10.1006/nimg.1999.0441. [DOI] [PubMed] [Google Scholar]
  • 60.Newman JD, Lindsley DF. Single unit analysis of auditory processing in squirrel monkey frontal cortex. Exp Brain Res. 1976;25:169–181. doi: 10.1007/BF00234901. [DOI] [PubMed] [Google Scholar]
  • 61.Tanila H, Carlson S, Linnankoski I, Kahila H. Regional distribution of functions in dorsolateral prefrontal cortex of the monkey. Behav Brain Res. 1993;53:63–71. doi: 10.1016/s0166-4328(05)80266-9. [DOI] [PubMed] [Google Scholar]
  • 62.Wollberg Z, Sela J. Frontal cortex of the awake squirrel monkey: responses of single cells to visual and auditory stimuli. Brain Res. 1980;198:216–220. doi: 10.1016/0006-8993(80)90358-3. [DOI] [PubMed] [Google Scholar]
  • 63.Romanski LM, Averbeck BB. The primate cortical auditory system and neural representation of conspecific vocalizations. Annu Rev Neurosci. 2009;32:315–346. doi: 10.1146/annurev.neuro.051508.135431. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Cohen YE, et al. A functional role for the ventrolateral prefrontal cortex in non-spatial auditory cognition. Proc Natl Acad Sci USA. 2009;106:20045–20050. doi: 10.1073/pnas.0907248106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Romanski LM, Bates JF, Goldman-Rakic PS. Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. J Comp Neurol. 1999;403:141–157. doi: 10.1002/(sici)1096-9861(19990111)403:2<141::aid-cne1>3.0.co;2-v. [DOI] [PubMed] [Google Scholar]
  • 66.Hackett TA, Stepniewska I, Kaas JH. Prefrontal connections of the parabelt auditory cortex in macaque monkeys. Brain Res. 1999;817:45–58. doi: 10.1016/s0006-8993(98)01182-2. [DOI] [PubMed] [Google Scholar]
  • 67.Poremba A, Mishkin M. Exploring the extent and function of higher-order auditory cortex in rhesus monkeys. Hear Res. 2007;229:14–23. doi: 10.1016/j.heares.2007.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Gifford GW, 3rd, MacLean KA, Hauser MD, Cohen YE. The neurophysiology of functionally meaningful categories: Macaque ventrolateral prefrontal cortex plays a critical role in spontaneous categorization of species-specific vocalizations. J Cogn Neurosci. 2005;17:1471–1482. doi: 10.1162/0898929054985464. [DOI] [PubMed] [Google Scholar]
  • 69.Russ BE, Lee YS, Cohen YE. Neural and behavioral correlates of auditory categorization. Hear Res. 2007;229:204–212. doi: 10.1016/j.heares.2006.10.010. [DOI] [PubMed] [Google Scholar]
  • 70.Lee JH, Russ BE, Orr LE, Cohen YE. Prefrontal activity predicts monkeys’ decisions during an auditory category task. Front Integr Neurosci. 2009;3:16. doi: 10.3389/neuro.07.016.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Deacon TW. Cortical connections of the inferior arcuate sulcus cortex in the macaque brain. Brain Res. 1992;573:8–26. doi: 10.1016/0006-8993(92)90109-m. [DOI] [PubMed] [Google Scholar]
  • 72.Aboitiz F. Gestures, vocalizations, and memory in language origins. Front Evol Neurosci. 2012;4:2. doi: 10.3389/fnevo.2012.00002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Sugihara T, Diltz MD, Averbeck BB, Romanski LM. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J Neurosci. 2006;26:11138–11147. doi: 10.1523/JNEUROSCI.3550-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Stein BE, Meredith MA. The Merging of the Senses. Cambridge, MA: MIT Press; 1993. [Google Scholar]
  • 75.Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cogn Neurosci. 2005;17:377–391. doi: 10.1162/0898929053279586. [DOI] [PubMed] [Google Scholar]
  • 76.Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci. 2005;25:5004–5012. doi: 10.1523/JNEUROSCI.0799-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 77.Lakatos P, et al. The leading sense: Supramodal control of neurophysiological context by attention. Neuron. 2009;64:419–430. doi: 10.1016/j.neuron.2009.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Kayser C, Logothetis NK, Panzeri S. Visual enhancement of the information representation in auditory cortex. Curr Biol. 2010;20:19–24. doi: 10.1016/j.cub.2009.10.068. [DOI] [PubMed] [Google Scholar]
  • 79.McGurk H, MacDonald J. Hearing lips and seeing voices. Nature. 1976;264:746–748. doi: 10.1038/264746a0. [DOI] [PubMed] [Google Scholar]
  • 80.Bertelson P, Aschersleben G. Temporal ventriloquism: Crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. Int J Psychophysiol. 2003;50:147–155. doi: 10.1016/s0167-8760(03)00130-2. [DOI] [PubMed] [Google Scholar]
  • 81.Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57:11–23. doi: 10.1016/j.neuron.2007.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 82.Stein BE, Stanford TR. Multisensory integration: Current issues from the perspective of the single neuron. Nat Rev Neurosci. 2008;9:255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  • 83.Noppeney U, Ostwald D, Werner S. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci. 2010;30:7434–7446. doi: 10.1523/JNEUROSCI.0455-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Beauchamp MS, Nath AR, Pasalar S. fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci. 2010;30:2414–2417. doi: 10.1523/JNEUROSCI.4865-09.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 85.Jones JA, Callan DE. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport. 2003;14:1129–1133. doi: 10.1097/00001756-200306110-00006. [DOI] [PubMed] [Google Scholar]
  • 86.Homae F, Hashimoto R, Nakajima K, Miyashita Y, Sakai KL. From perception to sentence comprehension: The convergence of auditory and visual information of language in the left inferior frontal cortex. Neuroimage. 2002;16:883–900. doi: 10.1006/nimg.2002.1138. [DOI] [PubMed] [Google Scholar]
  • 87.Xu J, Gannon PJ, Emmorey K, Smith JF, Braun AR. Symbolic gestures and spoken language are processed by a common neural system. Proc Natl Acad Sci USA. 2009;106:20664–20669. doi: 10.1073/pnas.0909197106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 88.Passingham RE, Toni I, Rushworth MF. Specialisation within the prefrontal cortex: The ventral prefrontal cortex and associative learning. Exp Brain Res. 2000;133:103–113. doi: 10.1007/s002210000405. [DOI] [PubMed] [Google Scholar]
  • 89.Calvert GA, Hansen PC, Iversen SD, Brammer MJ. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage. 2001;14:427–438. doi: 10.1006/nimg.2001.0812. [DOI] [PubMed] [Google Scholar]
  • 90.Miller LM, D’Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci. 2005;25:5884–5893. doi: 10.1523/JNEUROSCI.0896-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 91.Ojanen V, et al. Processing of audiovisual speech in Broca’s area. Neuroimage. 2005;25:333–338. doi: 10.1016/j.neuroimage.2004.12.001. [DOI] [PubMed] [Google Scholar]
  • 92.Hein G, et al. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci. 2007;27:7881–7887. doi: 10.1523/JNEUROSCI.1740-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 93.Campanella S, Belin P. Integrating face and voice in person perception. Trends Cogn Sci. 2007;11:535–543. doi: 10.1016/j.tics.2007.10.001. [DOI] [PubMed] [Google Scholar]
  • 94.Jordan KE, Brannon EM, Logothetis NK, Ghazanfar AA. Monkeys match the number of voices they hear to the number of faces they see. Curr Biol. 2005;15:1034–1038. doi: 10.1016/j.cub.2005.04.056. [DOI] [PubMed] [Google Scholar]
  • 95.Sliwa J, Duhamel JR, Pascalis O, Wirth S. Spontaneous voice-face identity matching by rhesus monkeys for familiar conspecifics and humans. Proc Natl Acad Sci USA. 2011;108:1735–1740. doi: 10.1073/pnas.1008169108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 96.Eifuku S, De Souza WC, Tamura R, Nishijo H, Ono T. Neuronal correlates of face identification in the monkey anterior temporal cortical areas. J Neurophysiol. 2004;91:358–371. doi: 10.1152/jn.00198.2003. [DOI] [PubMed] [Google Scholar]
  • 97.Sugase Y, Yamane S, Ueno S, Kawano K. Global and fine information coded by single neurons in the temporal visual cortex. Nature. 1999;400:869–873. doi: 10.1038/23703. [DOI] [PubMed] [Google Scholar]

Articles from Proceedings of the National Academy of Sciences of the United States of America are provided here courtesy of National Academy of Sciences

RESOURCES