Multisensory connections of monkey auditory cerebral cortex

Abstract

Functional studies have demonstrated multisensory responses in auditory cortex, even in the primary and early auditory association areas. The features of somatosensory and visual responses in auditory cortex suggest that they are involved in multiple processes including spatial, temporal and object-related perception. Tract tracing studies in monkeys have demonstrated several potential sources of somatosensory and visual inputs to auditory cortex. These include potential somatosensory inputs from the retroinsular (RI) and granular insula (Ig) cortical areas, and from the thalamic posterior (PO) nucleus. Potential sources of visual responses include peripheral field representations of areas V2 and prostriata, as well as the superior temporal polysensory area (STP) in the superior temporal sulcus, and the magnocellular medial geniculate thalamic nucleus (MGm). Besides these sources, there are several other thalamic, limbic and cortical association structures that have multisensory responses and may contribute cross-modal inputs to auditory cortex. These connections demonstrated by tract tracing provide a list of potential inputs, but in most cases their significance has not been confirmed by functional experiments. It is possible that the somatosensory and visual modulation of auditory cortex are each mediated by multiple extrinsic sources.

1. Introduction

There is a growing realization that multisensory integration occurs in many areas of the cerebral cortex, even in primary sensory areas. This calls for a modification of the classical model of cortical multisensory convergence, in which primary and early association sensory areas were thought to have unimodal functions, and multimodal processing occurred mainly in higher order association areas. This model was based in part on anatomical tract tracing, which showed that early sensory areas had almost exclusively local connections. However, recent studies have demonstrated cross-modal connections between early sensory areas. Additionally, functional studies have identified multisensory responses in these early areas. The implication of these findings is that multisensory convergence is a multi-step process that occurs in early sensory areas as well as higher order association areas. At present, it is unclear how many parallel streams of multisensory processing are present in early stages of sensory processing, what is their behavioral importance, and how they influence later processing.

The region of cortex in and around the caudal end of the lateral sulcus is located at a confluence of auditory, somatosensory, visual and vestibular streams of cortical processing, and contains a variety of areas with different weightings of multisensory responses. While there appears to be a similar organization in humans, intracortical recordings in monkeys have provided some of the clearest profiles of the multisensory responses in these areas. For example, while monkey primary and secondary auditory areas are dominated by auditory responses, there are also significant somatosensory and visual responses (Fu et al., 2003; Ghazanfar et al., 2005; Kayser et al., 2005, 2007; Lakatos et al., 2007). Medial to auditory cortex, the retroinsular area (RI) is mainly a somatosensory area, but it also has auditory responses (Robinson and Burton, 1980b; Fu et al., 2003). Near the rostral border of RI is an area with prominent vestibular responses (Guldin and Grusser, 1998), and this is in turn bordered rostrally by the caudal insula that has predominantly somatosensory, but also some visual and auditory responses (Robinson and Burton, 1980b; Schneider et al., 1993; Bieser, 1998). Immediately caudal to auditory cortex is the temporo-parietal area (Tpt), which has a predominance of auditory responses, but additionally somatosensory, visual and vestibular responses (Leinonen et al., 1980; Guldin et al., 1992). These areas, located within the lateral sulcus, are in turn bordered by higher order multisensory areas: dorsally, the inferior parietal lobe has a predominance of both somatosensory and visual responses (Leinonen et al., 1979), and ventrally the superior temporal polysensory area (STP) in the superior temporal sulcus has trimodal responses to visual, auditory and somatosensory stimuli, with an emphasis on visual responses (Bruce et al., 1981). This partial listing of responses in this region demonstrates an apparent orchestration of sensory integration, with each area having a distinct multisensory emphasis.

The multiple channels of multisensory processing are likely to contribute to a variety of cognitive processes. For example, it has been suggested that the caudal temporal plane, with its combination of vestibular and tactile responses especially for the head and upper body, may be a center for spatial orientation of the head in space (Leinonen et al., 1980; Guldin and Grusser, 1998; Schroeder et al., 2001; Ghazanfar and Schroeder, 2006). This interpretation suggests that the caudal temporal plane may function in parallel with parietal areas, which have body and world centered spatial coordinate representations (Andersen, 1997; Rizzolatti and Matelli, 2003). Other somatosensory and visual inputs to this region apparently lack spatial information, and may have a primary role in temporal encoding, selectively enhancing temporally concurrent sensory stimuli (Lakatos et al., 2007; Ghazanfar et al., 2008; Kayser et al., 2008). One implication of this temporal encoding is to enhance vocal responses that occur concurrently with the visual perception of the speaker’s face (Ghazanfar et al., 2008; Kayser et al., 2008). These interpretations from monkey experiments have interesting implications for parallel functions of this region in humans, including multisensory spatial localization, phonetic identification and speech perception (Giard and Peronnet, 1999; Calvert, 2001; Foxe et al., 2002; Olson et al., 2002; Calvert and Campbell, 2003; Wright et al., 2003; Murray et al., 2005; Teder-Salejarvi et al., 2005; Pekkola et al., 2006; Martuzzi et al., 2007; Antal et al., 2008; Bernstein et al., 2008).

Auditory cortex, with its reasonably well defined areas and connections, provides an accessible system to study the role of sensory convergence at both early and late stages of sensory processing, in the context of real-world behaviors. An important question for understanding the organization of this multisensory system is: what are the specific anatomical pathways that convey non-auditory sensory information into auditory cortex? Given the complexity of multisensory connections in the surrounding cortical areas, it is likely that these sources are heterogeneous, and may include thalamic as well as cortical sources. Tract tracing studies have produced evidence for both of these sources. In the following, we consider separately the evidence for sources of somatosensory, and then visual, projections to primary and association auditory areas.

2. Somatosensory inputs to auditory cortex

Auditory cortex is composed of primary or “core” areas, surrounded by first order auditory association or “belt” areas, which in turn are bordered by second order association (“parabelt”) areas and other association areas (Figure). Schroeder and colleagues made detailed mappings of somatosensory responses in the caudal medial belt area (CM), and also within the adjacent core area A1 (Fu et al., 2003; Lakatos et al., 2005; Lakatos et al., 2007). A surprising finding was that the somatosensory responses in CM and A1 had distinct laminar profiles, suggesting that they might not arise from the same source. In A1 they encountered somatosensory responses mainly in the superficial layers that did not drive action potentials, but instead had a modulatory effect on concurrent auditory responses. The short response latency of this somatosensory input (<10 msec) was very similar to the onset latency of auditory inputs to layer IV, and the authors demonstrated that the somatosensory input enhanced temporally synchronous auditory responses (Lakatos et al., 2007). In CM, the somatosensory responses had a feedforward profile, and were capable of driving action potentials (Fu et al., 2003). The earliest somatosensory response latency appeared to be later than that of A1, and nearly coincidental or slightly slower than the 11 millisecond auditory response latency in CM (Schroeder et al., 2001; Schroeder and Foxe, 2002). This profile of somatosensory responses was clearly distinct from that of A1: in fact, mapping studies showed that this response profile ended abruptly if electrodes were moved forward from CM to the caudal border of A1 (Fu et al., 2003). Within CM, 83% of recording sites had clear auditory response, and 72% of these had somatosensory responses, mostly commonly to cutaneous but also to proprioceptive input, and mainly from the head and neck but occasionally from the hand. This profile led the authors to suggest that CM may be part of a network of spatial orientation, possibly in head-centered coordinates (Schroeder et al., 2001). Thus, comparison of the somatosensory responses in A1 and CM showed that these adjacent areas have different onset latencies, laminar profile, and possible functional consequences. The somatosensory input to A1 is unlikely to be the source of that in CM, because only the latter is capable of driving action potentials. Conversely, area CM might relay somatosensory information to A1 via feedback projections to the superficial layers. However, arguing against this scenario, at least preliminary measurements of somatosensory response latencies in CM showed that they occur slightly later than those of A1 (Schroeder et al., 2001; Schroeder and Foxe, 2002). An alternative explanation is that the two areas receive somatosensory inputs from different sources outside of auditory cortex.

Figure 1.

Schematic summaries of the possible sources of non-auditory sensory inputs to auditory cortex are presented. In these schematics, area Tpt is included with the auditory areas, because it has multisensory connections very similar to those of the adjacent caudal auditory areas. Typically, Tpt is not considered an auditory area, but rather an “auditory-related area” (Hackett and Kaas, 2004). The inset at the upper right shows a schematic summary of auditory areas on the ventral bank of the lateral sulcus (Hackett and Kaas, 2004). In monkeys, there are at least 2 core areas (A1 and R, in blue color), surrounded by several belt areas (yellow color), and flanked laterally be parabelt areas and caudally by area Tpt. Additional core and belt areas may lie rostral to R, but these are still poorly defined. In the following schematics, the belt areas surrounding A1 are treated as a single area. Even though the belt connections were demonstrated in areas CM and CL, these connections are assumed to be present as well in MM and ML. A. A summary of somatosensory cortex to auditory cortex connections (red arrows) is shown on a schematic of the opened lateral sulcus. Arrows with dashed lines show connections that are less certain. In both macaques and marmosets, the caudal belt and parabelt, as well as area Tpt, receive inputs at least from Ri and Ig. There is some evidence that these connections are also sparsely present in A1, and there is some evidence that S2 also projects to auditory areas. Similar somatosensory inputs to rostral areas were generally not observed, with the exception of one injection in somatosensory area PV that showed a rostral auditory connection (Disbrow et al., 2003). B. Purple arrows summarize the connections of auditory cortex with areas of early visual processing. Visual inputs to caudal auditory belt, parabelt and Tpt originated in areas V2 and prostriata. Projections from these same auditory areas, and possibly also A1, were found in V2, prostriata, and area V1. Nearly all connections were with the peripheral field representations in visual cortex. C. Black arrows show the connections of several nuclei of the caudal thalamus that project to auditory cortex and may be sources of cross-modal input. All of these multisensory nuclei project to the caudal auditory areas, but the rostral areas receive significant inputs only from MGm, and more sparsely from the medial pulvinar (MP) and the SG. D. Black arrows show inputs from cortical and limbic association areas that might provide multisensory input to auditory cortex. These include areas 23 and 31 located along the caudal cingulate gyrus, area 7a on the inferior parietal cortex, area STP in the superior temporal sulcus, several areas on the lateral surface of the prefrontal cortex, and the claustrum.

At present it is uncertain to what extent these somatosensory responses in A1 and CM are also present in other auditory areas. In vivo imaging in monkeys (Kayser et al., 2005) and humans (Foxe et al., 2002) suggests that at least most of the caudal auditory areas have somatosensory responses, and the following sections describe anatomical inputs from somatosensory sources that seem to extend to most of the caudal superior temporal plane.

Tract tracing studies in the monkey have identified several potential sources of somatosensory input to auditory cortex. Among these, the most likely candidates appear to be: 1) the adjacent somatosensory areas along the dorsal bank of the lateral sulcus, especially including the retroinsular area (Ri), the granular insula (Ig) and possibly the second somatosensory areas (S2), and 2) the “multisensory” nuclei located in the posterior thalamus, including the posterior (PO), magnocellular medial geniculate (MGm), supragranular (SG) and limitans nuclei. Besides these structures, auditory cortex also has connections with the pulvinar nucleus of the thalamus, and with several limbic and higher order association cortical areas. While these are also potential sources of somatosensory inputs, the evidence is somewhat less compelling. The following paragraphs summarize the evidence for each of these potential sources of somatosensory input to CM, A1, and other auditory areas.

2.1. RI, Ig and S2 as sources of somatosensory input to auditory cortex

The identification of the somatosensory areas in the parietal operculum is an ongoing research, and it is necessary to briefly comment on the organization of this region in monkey. Much of this region is occupied by an area that was in the earlier literature referred to as the second somatosensory area (S2). However, detailed mappings showed that there were multiple areas with distinct somatotopic maps (Krubitzer et al., 1995; Qi et al., 2002; Disbrow et al., 2003). These included areas extending across the medial to lateral width of the parietal operculum that are currently referred to as a caudal area (S2), a more rostral parietal ventral somatosensory area (PV), and even more rostrally, a less well defined rostral ventral parietal area (PR). At least areas S2/PV have features of somatosensory association cortex, in that they have substantial inputs from primary somatosensory areas 3b and 1 in the postcentral gyus, they have larger receptive fields than primary areas, and they are involved in more complex sensory processing such as identification of shapes and textures (Robinson and Burton, 1980b; 1980c; Murray and Mishkin, 1984). Their major thalamic input is from the inferior division of the ventral posterior nucleus (VPI), in contrast to primary somatosensory areas 3b and 1 that have inputs mainly from the superior division (VPS) (Friedman and Murray, 1986; Qi et al., 2002; Disbrow et al., 2003). Disbrow et al. (2003) review the differences between areas S2 and PV.

Ventral to S2/PV, lying at the fundus of the lateral sulcus, are areas RI caudally and Ig rostrally. Area Ig comprises about the caudal third of the insula, and can be distinguished from more rostral insula by its connections and cytoarchitectonic appearance. It has mainly somatosensory responses, usually with large bilateral receptive fields that are at least roughly somatotopically organized (Robinson and Burton, 1980a; Schneider et al., 1993). Area RI was originally defined as the anatomically distinctive area caudal to the insula, at the fundus of the lateral sulcus, that had a distinctive pattern of thalamic inputs (Burton and Jones, 1976; Jones and Burton, 1976). In contrast to Ig, the somatosensory receptive fields in RI have more sharply defined borders, and are about half contralateral and half bilateral (Robinson and Burton, 1980a). Electrode mapping in macaque monkeys showed that RI has a separate somatotopic map (Robinson and Burton, 1980a), and this finding was replicated in marmosets, owl monkeys and macaques, although there were some differences in the precise borders of RI (Cusick et al., 1989; Krubitzer et al., 1995; Qi et al., 2002). In particular, the latter studies described a “ventral somatosensory area” that appeared to include RI but also extended somewhat rostrally into the insula. Additionally, there was some evidence for more than one somatotopic map in this ventral somatosensory area (Burton et al., 1995; Krubitzer et al., 1995). Burton et al. (1995) proposed a possible synthesis of these findings, suggesting that RI may extend into the dorsal-caudal end of the insula, or alternatively that there may be a second “RI-like” area at this location. Therefore, the reader should be mindful that the rostral border of area RI is poorly defined, and the anatomical connections from Ig to auditory cortex described below might be in part from this rostral extension of RI at the caudal-dorsal insula. Additionally, it remains unclear whether the vestibular responsive area found near the rostral end of RI represents a separate area, or an integral part of RI (Guldin et al., 1992; Guldin and Grusser, 1998).

Besides these ambiguities, it is probable that some of the physiological mappings of RI also included auditory area CM, as both areas have somatosensory and auditory responses, and the anatomical border between them has been placed differently in different studies. Nevertheless, tract tracing studies in rhesus and marmoset monkeys have demonstrated that the border between RI and CM can be located precisely based on their differential connections (de la Mothe et al., 2006a; 2006b; Hackett et al., 2007; Smiley et al., 2007). For example, tracer injections restricted to CM show a predominance of temporal lobe connections, with only limited parietal lobe connections mainly with RI and Ig, whereas injections that encroach onto the border of RI have robust parietal lobe connections.

In both macaque and marmoset monkeys, small injections of retrograde tracers into macaque area CM demonstrated direct inputs from the adjacent areas RI and Ig (Figure, panel A) (de la Mothe et al., 2006a; Smiley et al., 2007). Only a few and inconsistent connections were found with areas S2. However, a study in marmosets did show evidence for a projection from S2 to A1 (Cappe and Barone, 2005), and another study in macaques showed a connection between PV and rostral auditory cortex, at or near the rostral core area R (Disbrow et al., 2003). Projections from RI/Ig are not restricted to belt area CM, but have also been demonstrated to other areas of the caudal lateral sulcus, including at least belt areas CL, area Tpt, and the surface of the caudal superior temporal gyrus (Galaburda and Pandya, 1983; Cipollini and Pandya, 1989; de la Mothe et al., 2006a; Smiley et al., 2007). Additionally we observed sparse projections from RI/Ig to primary cortex (A1) (unpublished), consistent with some previous findings (Mesulam and Mufson, 1982; Cipolloni and Pandya, 1999). Therefore, it is probable that the connections of RI/Ig extend to all of the auditory and auditory-related areas of the caudal superior temporal plane. In contrast, it is notable that similar projections from RI/Ig were usually not found with tracer injections in more rostral areas, including rostral belt area RM (de la Mothe et al., 2006a) and the rostral superior temporal gyrus (Galaburda and Pandya, 1983; Cipollini and Pandya, 1989). Nevertheless, as mentioned above, PV was found to be connected with rostral auditory areas (Disbrow et al., 2003).

Areas RI and Ig are dominated by somatosensory responses (Sudakov et al., 1971; Robinson and Burton, 1980a; 1980b; 1980c; Schneider et al., 1993). In contrast to nearby areas S2/PV, their somatosensory responses are somewhat more complex, and they also have some limited responsiveness to visual and auditory stimuli (Robinson and Burton, 1980b). Also in contrast to S2, their thalamic connections arise mainly from “multisensory” thalamic nuclei (PO, MGm, SG, and limitans) with a comparatively minor input from VPI (Friedman and Murray, 1986; de la Mothe et al., 2006b; Hackett et al., 2007) whereas the dominant thalamic connection of S2/PV is from VPI (Friedman and Murray, 1986; Qi et al., 2002; Disbrow et al., 2003). The cortical connections of RI and Ig can also be contrasted with those of S2/PV. The latter have a substantial input from primary somatosensory areas 3b and 1 on the postcentral gyrus, and the laminar distribution of labeling is consistent with feedforward projections to S2/PV (Friedman’86; Burton, 95). Additionally, S2/PV have connections with parietal areas 7b, and at least PV has connections with the intraparietal sulcus (Krubitzer and Kaas, 1990; Qi et al., 2002; Disbrow et al., 2003). In contrast, RI and Ig have only sparse inputs from primary areas 3b/1, which also has a feedforward-type laminar profile (Burton, 1995; Mufson and Mesulam, 1982). The major input to Ig is from S2/PV (Friedman and Murray, 1986), and the feedforward laminar pattern of this connection is consistent with evidence that Ig is a critical node for the relay of somatosensory information from parietal cortex to the amygdala and hippocampus (Mishkin, 1979). While RI also has a major connection with S2, the laminar pattern suggests that this is a feedforward connection from RI to S2 (Friedman, 1983; Friedman and Murray, 1986). Besides these connections, other parietal cortex connections of RI and Ig include area 5 (Friedman et al., 1986), and RI has connections with parietal areas VIP, 7b, 7a and possibly AIP (Lewis and Van Essen, 2000; Rozzi et al., 2006; Smiley et al., 2007).

In summary, area CM and probably most other areas in the caudal superior temporal plane have connections with areas RI and Ig. Additionally, area S2 and PV might contribute sparse inputs to auditory areas. While areas S2 and PV receive robust feedforward inputs from primary somatosensory areas 3b/1 and the thalamic somatosensory relay nucleus VPI, areas RI and Ig have minor inputs from these structures. They have more substantial inputs from the multisensory nuclei of the thalamus (discussed below), and less well characterized connections with somatosensory-related areas 5, VIP, 7b and 7a, and possibly AIP. While Ig has clear inputs from area S2, RI provides feedforward inputs to S2.

2.2. Thalamus as a source of somatosensory input to auditory cortex

Two recent retrograde tracing studies of CM and other auditory areas analyzed the distribution of thalamic inputs in marmoset and rhesus macaque monkeys (de la Mothe et al., 2006b; Hackett et al., 2007), and found major inputs from auditory nuclei of the medial geniculate complex, and from nearby “multisensory” nuclei of the posterior thalamus. The auditory input to CM was found mainly in the anterior dorsal division (MGad), with smaller connections from the ventral (MGv) and posterior dorsal divisions (MGpd), and these connections are consistent with the robust auditory responses in CM. The multisensory thalamic inputs to CM were primarily from PO and MGm, with less robust inputs from the SG, limitans, and medial pulvinar nuclei. Aside from the pulvinar, these latter nuclei are called multisensory because electrode recordings, mainly in cats and rodents, have demonstrated that they have different complements of responses to auditory, visual, vestibular and somatosensory stimuli. While the medial pulvinar is typically thought of as a nucleus mediating connections between higher order association cortices, it also has inputs from subcortical sensory nuclei and substantial connections with auditory areas. In monkeys, the multisensory thalamic nuclei are located along the medial, dorsal and rostral edges of the medial geniculate nucleus, near and within the fibers of the brachium of the superior colliculus. Much of our understanding of their function is derived from studies in cats and rats, and while the findings may not apply precisely to primates, tract tracing in monkeys have generally shown connections of these nuclei similar to those found in other species. The following paragraphs briefly summarize the evidence that these nuclei may be a source of somatosensory input to CM and other auditory areas.

The PO nucleus is perhaps the most likely thalamic source of somatosensory input to CM. In addition to the retrograde tracer studies mentioned above, another study in rhesus and marmoset monkeys injected 3H-amino acids into PO to demonstrate its anterograde connections, and concluded that its main output was to lower layer III and sparsely to layer IV of area RI and area CM (their area Pa) (Burton and Jones, 1976). While subsequent retrograde tracer experiments confirmed these findings, they additionally showed that PO projects to other caudal belt areas, to areas Tpt, and a sparsely to A1 (de la Mothe et al., 2006b; Hackett et al., 2007). Notably, however, rostral belt areas appeared to lack this connection with area PO (de la Mothe et al., 2006b). Thus the projections from PO appear to be most robust to areas RI and CM, but may additionally influence the entire caudal superior temporal plane, and at least in areas RI and CM it mainly targets the middle (i.e., feedforward) cortical layers. Subcortically, PO in monkeys has substantial inputs from somatosensory pathways including the dorsal column nuclei (Berkley, 1980) and spinothalamic pathways (Apkarian and Hodge, 1989; Davidson et al., 2008). At least in cats, PO also receives auditory inputs from the inferior colliculus (Andersen et al., 1980). Findings in the cat suggest that there is some separation of the inputs in PO, as somatosensory inputs target the medial PO and auditory projections target the lateral PO. Similarly, in monkeys the medial PO projects more the RI, and the lateral PO projects more to CM (Burton and Jones, 1976). Sensory responses in PO have not been directly studied in monkeys, but in cats PO has somatosensory, auditory and visual response (Poggio and Mountcastle, 1960; Curry, 1972a; Phillips and Irvine, 1979). Only a small minority of cells had bimodal somatosensory- auditory or somatosensory-visual responses (Blum and Gilman, 1979; Avanzini et al., 1980). Together, these findings show that PO is a potential source of the feedforward somatosensory inputs to CM, and possibly to other areas of the caudal superior temporal plane.

Retrograde tracing in monkeys also showed that MGm is a significant source of input to CM, and like PO projects to most or all other areas of the caudal superior temporal plane including primary auditory cortex, but unlike PO also has clear connections with rostral areas R and RM (Morel and Kaas, 1992; Morel et al., 1993; Molinari et al., 1995; de la Mothe et al., 2006b; Hackett et al., 2007). The subcortical connections of MGm show some evidence for spinal cord inputs from dorsal column and spinothalamic pathways, but these were only sparse compared to those of PO (Berkley, 1980; Apkarian and Hodge, 1989; Davidson et al., 2008). Electrode recordings in non-primates showed robust auditory inputs in MGm (e.g. (Rouiller et al., 1989)) and additionally abundant somatic, vestibular and visual responses (Curry, 1972b; Bordi and LeDoux, 1994). Investigations in both the cat and rat emphasized that individual cells responsive to multiple modalities were especially common in this nucleus (Wepsic, 1966; Bordi and LeDoux, 1994). Injections of anterograde 3H-amino acid tracers in monkeys indicated that this nucleus projects mainly to the superficial layers across auditory areas (Burton and Jones, 1976). However, more sensitive anterograde tracer showed that it also has some projections to the middle cortical layers (Hashikawa et al., 1995). Additionally, parvalbumin containing neurons that project to the middle layers of auditory cortex are relatively abundant in the MGm (Molinari et al., 1995). Thus, MGm is a potential source of inputs from somatic and other sensory modality to auditory cortex. Its output might to be preferentially weighted toward superficial layers, but it has some projections to middle layers.

Additional but sparser inputs to auditory areas arise from the SG and limitans nuclei, which are located together and often referred to as “SG-limitans” (Molinari et al., 1995; de la Mothe et al., 2006b; Hackett et al., 2007). Subcortical connections of these nuclei are most abundant with the inferior and superior colliculi of the brainstem, consistent with a weighting toward auditory and visual inputs (Benevento and Fallon, 1975; Ledoux et al., 1987; Linke, 1999). As with the MGm, there may be some sparse somatosensory terminations from the spinal cord (Berkley, 1980; Apkarian and Hodge, 1989; Davidson et al., 2008). While SG and limitans have sparse projections across the caudal superior temporal gyrus, anterograde tracer injections showed that their densest projections are to the insula, where they terminate in the middle cortical layers (Burton and Jones, 1976).

The medial pulvinar nucleus also has projections to CM, and sparsely to A1 (de la Mothe et al., 2006b; Hackett et al., 2007). Anterograde tracings with 3H-amino acids show that the medial pulvinar projects to essentially all parts of the superior temporal gyrus, and additional to prefrontal association areas (Burton and Jones, 1976; Trojanowski and Jacobson, 1976). In the superior temporal gyrus, its axon terminals are concentrated mainly in layer III, and are denser in the upper bank of the superior temporal sulcus and less dense on the superior temporal plane. The medial pulvinar is known to have multisensory responses, and receives some potential visual and somatosensory inputs from the brainstem superior colliculus and spinal cord somatosensory pathways (Benevento and Fallon, 1975; Apkarian and Hodge, 1989). In general, the nuclei of the pulvinar are known to be most densely connected with association areas of the cerebral cortex, and are thought to be driven mainly by cortical inputs (Sherman, 2007). Additionally, there is evidence that the pulvinar has reciprocal connections with primary and early association sensory areas, and that it might potentially represent a pathway of multisensory convergence between cortical areas (Cappe et al., 2009).

In summary, the combined literature from monkeys and other animals has shown that the group of multisensory nuclei of the posterior nuclei, as well as the medial pulvinar nucleus, have different weightings of somatosensory, visual and auditory responses and project to most areas of the caudal superior temporal gyrus. With respect to the feedforward somatosensory response in CM (Fu et al., 2003), the PO nucleus seems a likely source, in that it receives robust spinal cord input and its main output appears to be the middle cortical layers of CM and RI. However, it is not established to what extent the medial, mainly somatosensory, division of PO projects to CM, or rather is segregated to RI. Other multisensory nuclei and the medial pulvinar also project to CM, and these have at least some inputs from the spinal cord. The MGm has some somatosensory responses and at least some projections to the middle layers. The SG-limitans appears to be more visual and projects more robustly to Ig, and the medial pulvinar is thought to be mainly driven by cortical inputs. Nevertheless, in the absence of direct physiological evidence, these structures cannot be ruled out as sources of somatosensory inputs to CM and other auditory areas. With respect to somatosensory input to A1, the same thalamic nuclei can be listed as potential sources. However, given the evidence that the somatosensory input to A1 targets the superficial layers (Lakatos et al., 2007) it might be proposed that this input comes from the axons of MGm that are superficially distributed. If this is correct, then it might be predicted that the widely distributed MGm axons have a similar role in other rostral and caudal auditory areas.

The functions of the projections from the multisensory thalamic nuclei to auditory cortex are incompletely understood. An early hypothesis was that they might relay noxious stimuli to the cortex, but subsequent recordings in RI/CM found only a small minority of cells that respond to such stimuli (Burton and Jones, 1976; Robinson and Burton, 1980b). In rats, LeDoux and colleagues have demonstrated that these nuclei, along with the adjacent posterior intralaminar nucleus, can relay emotionally salient information to the amygdala (Bordi and LeDoux, 1994). In addition to this direct projection to the amygdala, there is a complementary pathway from these nuclei to auditory cortex, which indirectly relays emotionally salient information to the amygdala (Romanski and LeDoux, 1992). In fact, it has been demonstrated that individual thalamic cells project to both the amygdala and to auditory cortex (Doron and Ledoux, 2000). It is possible that a similar function is present in primates.

2.3. Cortical association areas and somatosensory input to auditory cortex

Area CM has connections with several cortical areas that can be described as either limbic or higher order association areas. While some of these have robust multisensory responses, they are less likely sources of short latency somatosensory input to CM for two reasons. First, in nearly all cases the laminar patterns of these connections suggest that the direction of information flow is feedforward, mainly directed away from auditory cortex. Second, to the extent that response latencies are known, these cortical areas appear to be sites of delayed, or “downstream” information processing.

Among these cortical association areas, STP in the superior temporal sulcus has the most robust connection with auditory belt and parabelt areas as well as area Tpt. While STP has somatosensory responses, their post-stimulus latencies are typically much later than those of auditory cortex (Musacchia and Schroeder, 2009). Additionally, the laminar pattern of connection is consistent with a mainly feedforward connection from auditory areas to STP (de la Mothe et al., 2006a; Smiley et al., 2007).

Other areas that connect at least with caudal areas Tpt and CPB, and probably sparsely with the caudal belt areas, are cortical areas 7a of the inferior parietal cortex, which receives apparent feedforward connections from auditory cortex (Rozzi et al., 2006; Smiley et al., 2007), areas 23 and 31 along the caudal cingulate gyrus that have somatosensory, visual and limbic connections and receive feedforward inputs from auditory cortex (Morecraft et al., 2004) and several areas of the lateral prefrontal cortex that have complex roles in the integration of multisensory information (Romanski et al., 1999a). Additionally, caudal auditory areas have connections with limbic structures including the claustrum, entorhinal cortex, and amygdala (de la Mothe et al., 2006a; Smiley et al., 2007). The claustrum may be a site of multisensory integration, but its inputs are mainly from higher order association areas, and it is unlikely to be the source of short latency somatosensory responses in auditory cortex (see (Smiley et al., 2007)). Overall, most of these association and limbic areas are thought to receive sensory information later than that seen in auditory cortex, and to provide mainly feedback information to auditory cortex.

2.4. Conclusions

The results from tract tracing studies provide a framework of possible links in the multisensory circuitry of auditory cortex. The weight of evidence for some connections is more compelling, based on the information they carry and the anatomical abundance of the connection. On the basis of these methods, likely candidates for feedforward somatosensory input to area CM seem to be the thalamic nucleus PO and the adjacent cortical areas RI and Ig, which have mainly somatosensory responses and have direct projections to CM. At least the projection from PO terminates in the middle layers as expected for a feedforward projection. Similarly, possible candidates for the somatosensory input to A1, which has short stimulus-response latency and is centered in the superficial layers, include the thalamic nucleus MGm, which receives at least some spinal cord inputs and has projections to both the superficial and middle layers of A1 and other auditory areas. Of course, these findings provide only a hypothetical framework, and it is possible that the several other connections of auditory cortex contribute components of somatosensory and other multisensory influences in auditory cortex.

3. Visual inputs to auditory cortex

Several recent studies in nonhuman species have shown that visual stimuli can influence auditory processing across auditory areas, even in primary auditory cortex (Brosch et al., 2005; Ghazanfar et al., 2005; Schroeder and Foxe, 2005; Bizley et al., 2007; Kayser et al., 2007). The functional relevance of visual modulation of auditory cortex is still being explored. In humans, functional imaging has shown activation of the caudal superior temporal plane by simple moving visual stimuli (Martuzzi et al., 2007; Antal et al., 2008) by visual identification of graphically presented phonemes (van Atteveldt et al., 2004; Bernstein et al., 2008) and by viewing speakers during speech perception tasks (Calvert, 2001; Olson et al., 2002; Calvert and Campbell, 2003; Wright et al., 2003; Pekkola et al., 2006).

Using electrode recordings in monkeys, the main response to visual stimuli was a modulation of auditory responses, and visual responses by themselves typically did not drive neuron spiking (Schroeder and Foxe, 2002; Kayser et al., 2008). Analysis of the laminar profile of visual response showed activation of the superficial and deep layers, consistent with a modulatory or feedback type of input (Schroeder and Foxe, 2002). The effect of the visual modulation was complex, in that both suppression and enhancement of activity was found at adjacent recording sites (Kayser et al., 2008). This mixture of suppression and enhancement was found in primary as well as other caudal areas, using either electrode recordings or fMRI (Kayser et al., 2007; Kayser et al., 2008). However, fMRI showed that the visual effects were comparatively less in the core areas, and that they were more pronounced in caudal than rostral auditory areas (Kayser et al., 2007).

In monkeys, Kayser et al (Kayser et al., 2008) found that visual modulation in auditory cortex was sensitive to stimulus asynchrony: i.e., visual stimuli had significant effects only when presented 20–80 msecs before the auditory stimuli. This timing difference is consistent with the delayed processing of visual stimuli compared to other sensory modalities. In macaque monkeys, the visual response latency in V1 is in the range of 20–30 msecs, compared to shortest latencies of about 10 msec in primary auditory cortex, and about 6 msec in primary somatosensory cortex (Schroeder et al., 1998; Schroeder and Foxe, 2002; Musacchia and Schroeder, 2009). Visual responses in association areas STP and the intraparietal sulcus occur only slightly later than V1, at about 25 msec (Schroeder and Foxe, 2002). Thus the approximate time frame of 20–80msec visual-auditory disparity of described by Kayser et al (2008) is at least consistent with the possibility that auditory cortex is modulated by connections with very early stages of cortical visual processing, but it does not exclude the possibility that this input comes from downstream association areas.

As described above for somatosensory connections, the visual inputs to auditory cortex might potentially arise from multiple structures, including cortical association and limbic areas (e.g., STP, 7a, medial parietal areas 23/31, claustrum) or from thalamic nuclei with visual responses (e.g., SG-limitans, MGm). As already discussed, for most of the cortical association areas, the laminar patterns of connections suggest that they supply mainly feedback inputs to auditory cortex, but this does not exclude the possibility that these areas provide visual modulation of auditory cortex. At present, it has not been directly determined which structures provide visual input to auditory areas, or whether there are multiple sources.

One likely source of visual input to auditory cortex is via direct connections with visual areas V2 and prostriata (Falchier et al., 2002; Rockland and Ojima, 2003; Falchier et al., 2009). Area prostriata is a visual area located in the rostral calcarine sulcus directly abutting areas V1 and V2, near the region of their far peripheral visual field representations. While its function in not well documented, it is strongly connected with the peripheral field representations of V1, V2, and MT, and is thus probably involved in dorsal stream processing (Gattas et al., 1997; Palmer and Rosa, 2006). Falchier et al. (2009) used a series of retrograde tracer injections in macaque auditory areas CM, CL, CPB and Tpt, and consistently found retrograde labeled cells in V2 and prostriata. Only one injection was placed in area A1, and it did not reveal a projection from these visual areas. However, it is possible that sparse connections with A1 simply were not detected, and in fact a similar connection to primary auditory cortex was observed in ferrets (discussed below). Inspection of both hemispheres demonstrated that this projection was strictly from the ipsilateral side. Within V2 and prostriata, the labeled cells were nearly all found in infragranular layers, with only a scattering of supragranular cells. Another study in the marmoset did not find auditory connections with V2/prostriata (Cappe and Barone, 2005). However, the tracer injections were mainly restricted to the auditory core, and it is possible that injections of belt and parabelt would reveal visual connections similar to macaques.

Earlier tracing studies in monkeys had documented the reciprocal connection from auditory cortex to early visual areas (Falchier et al., 2002; Rockland and Ojima, 2003). This pathway also involves the peripheral representation of V2 and prostriata, but additionally there are auditory projections to V1, mainly from its peripheral field representation. Unexpectedly, the cells giving rise to both the visual-auditory and the auditory-visual projections are overwhelmingly located in the infragranular layers, and at least anterograde-labeled axons to visual cortex are concentrated in the superficial layers (Falchier et al., 2002; Rockland and Ojima, 2003; Falchier et al., 2009). Thus these experiments document an unusual type of reciprocal connection that has “feedback” anatomical features in both directions. A possible interpretation of this arrangement is that the functions of these connections between early sensory areas are mainly modulatory. For example, as mentioned above, the visual inputs in auditory cortex have modulatory effects. Similarly, Lakatos et al. (Lakatos et al., 2008; Schroeder and Lakatos, 2009) showed that auditory inputs to primary visual cortex do not drive action potentials, but rather are detected as current sources in the superficial layers that can modulate visual responses.

A striking feature of visual connection with auditory cortex is that they arise largely in the far peripheral visual field representation (Falchier et al., 2002; Rockland and Ojima, 2003; Bizley et al., 2007; Falchier et al., 2009). This organization is similar to a variety of cortical pathways that have segregated connections with either peripheral or foveal visual field representations. For example, peripheral but not central representations of areas V1 and V2 are connected with numerous “dorsal stream” visual areas in temporal and parietal cortex (Ungerleider and Desimone, 1986; Colby et al., 1988; Gattas et al., 1997; Ungerleider et al., 2008). The observation that caudal auditory cortex is connected with early peripheral visual representations suggests a “dorsal-stream” type function of these multisensory connections (Falchier et al., 2002). Consistent with this interpretation, previous authors have suggested that the caudal auditory areas may represent a dorsal stream-like pathway, based on their role in spatial localization of sounds and their connections with spatial domains of the frontal lobe (Romanski et al., 1999b; Rauschecker and Tian, 2000; Romanski and Goldman-Rakic, 2002; Rauschecker and Tian, 2004). Thus it appears that early multisensory convergence may have a prominent role in spatial localization, perhaps especially with respect to the peripheral visual field. One possibility is that the early interactions of auditory and visual cortices might be crucial when sensory sources are in the peripheral visual field, where visual detection is less efficient, and auditory information can help to optimize head-eyes orientation (Goldring et al., 1996; Hughes et al., 1998; Giard and Peronnet, 1999; McDonald et al., 2000). This is consistent with experiments in ferrets where visual responses in auditory cortex were clearly tuned to visual stimuli located near the contralateral surface of the head (Bizley et al., 2007). Similarly, in humans, attention to the spatial location of auditory stimuli causes activation of visual cortex at the region of peripheral field representation (Cate et al., 2009).

Visual modulation of auditory cortex has also been demonstrated in non-primates. In ferrets, single unit recordings showed that visual responses are commonly found across auditory areas, including primary auditory cortex (Bizley et al., 2007; Bizley and King, 2008). Tracer injections demonstrated that the primary and other auditory areas receive inputs from ipsilateral visual areas 17, 18, 19, 20 as well as posterior parietal areas. Like monkeys, the visual responses and connections were comparatively more dense in non-primary auditory areas, and the visual inputs were mainly from areas with peripheral visual field representation (Bizley et al., 2007). An earlier study of kittens showed projections from auditory areas AI and AII to visual cortical areas 17 and 18, but similar connections were not found in adult cats, and it was suggested that these early connections were lost during post-natal development (Clarke and Innocenti, 1990). In gerbils, tract tracing showed a significant projection to auditory cortex from Oc2 (the second visual area) (Budinger et al, 2006). In gerbils, the visual projection to auditory cortex was mainly from lower layers, like that of monkeys (Budinger et al., 2006). Besides these visual connections, gerbil primary auditory cortex also had direct connections with primary somatosensory areas, a connection that has not been found at least in monkeys, and that the authors suggest may be of special behavioral significance to gerbils (Budinger et al., 2006). Thus there are clearly some similarities across species, along with distinct differences.

Finally, it should be emphasized that the above described connection from V2/prostriata might be only one of multiple sources of visual input to auditory cortex. For example, Ghazanfar showed a phase coherence of visual responses in auditory cortex and the polysensory area STP (Ghazanfar et al., 2008). While coherence does not prove that STP drives the visual response in auditory cortex, it is at least suggestive of visual information from STP arriving in auditory cortex. Additionally, their finding supports the hypothesis that the STP is a region especially involved in the integration of auditory and visual cues for the perception of speech (Calvert, 2001; Ghazanfar et al., 2005; Ghazanfar et al., 2008). Future experiments are needed to identify precisely the contributions of different potential sources of visual information to auditory cortex.

4. Conclusions

The goal of this review has been to identify the potential sources of somatosensory and visual inputs into monkey auditory cortex demonstrated by anatomical tract tracing. Among the potential sources, some seem more likely because their sensory responses match those seen in auditory cortex, and because they project to the areas and layers of auditory cortex where the multisensory responses occur. Current evidence suggests that somatosensory inputs are more likely to arise from adjacent cortical areas RI and IG, and possibly from thalamic nuclei PO and MGm. Visual inputs are likely to arise from areas V2 and prostriata, but may additionally come from the thalamic nuclei SG-limitans and MGm, as well as STP in the superior temporal sulcus. Other sources cannot be excluded, and multiple parallel sources may be involved. More definitive physiological experiments with multiple recordings from separate areas, and more precise definitions of the relative latencies of responses in different areas, may resolve these ambiguities.

Some general statements can be made about the multisensory inputs to auditory cortex. First, both primates and non-primates have these connections, and while they are broadly similar across species, there are some differences. A striking example is the gerbil, that has direct connections between primary somatosensory and auditory cortex that was not found in monkeys (Budinger et al., 2006). Even between primate species there is some evidence for qualitative differences. Thus certain species may provide especially advantageous experimental models, but extrapolating results across species should be done with caution. Second, even at early stages of sensory processes, cross-modal connections might be conveyed by multiple parallel connections. An example of this is the distinct profiles of somatosensory responses observed in adjacent areas A1 (Lakatos et al., 2007) and CM (Fu et al., 2003). These different channels may be contributing to separate perceptual processes (e.g., spatial versus temporal encoding) or may be sequentially contributing to a single percept. Further exploration of these responses in other auditory areas will help to clarify the diversity of types of multisensory inputs. Third, within auditory cortex, caudal areas typically have more abundant potential multisensory connections than rostral areas, and this is consistent with more pronounced functional activation of the caudal areas by somatosensory and visual stimuli (Foxe et al., 2002; Kayser et al., 2007). However, there may be some exceptions to this observation, and at least the medial pulvinar, the MGm thalamic nucleus, and the multisensory area STP project to rostral areas. Finally, while the focus of this review has been on areas of early sensory processing, it should be emphasized that the abundance of cross-modal connections increases with distance away from primary sensory cortex. Indeed, earlier tract tracing studies concluded that early sensory areas had overwhelmingly local connections, giving rise to the concept of unimodal primary and association sensory areas (e.g., (Kuypers and Lawrence, 1967; Diamond et al., 1968; Pandya and Kuypers, 1969; Jones and Powell, 1970). More recent applications of more sensitive tracers have highlighted exceptions to this concept, but they consistently found at least qualitative evidence for more abundant multisensory connections in higher order association areas. For example, in monkey auditory cortex, the abundance of connections from potential sources of somatosensory (RI/Ig) visual (V2/prostriata) and association/limbic areas (parietal, temporal and frontal cortex, claustrum, entorhinal cortex) typically was greater in areas CPB and Tpt than it was in belt areas (Smiley et al., 2007; Falchier et al., 2009). A similar pattern was also found in ferret auditory cortex (Bizley et al., 2007; Bizley and King, 2008). Thus, it might be concluded that there is substantial validity to the earlier model of late stage sensory integration, but that some cross-modal inputs are additionally present even as early as primary sensory cortex. More refined experiments will clarify to what extent the differences between early and late stage multisensory connections are qualitative as well as quantitative.

Abbreviations

A1

primary auditory cortex

AIP

anterior intraparietal area

CM

caudal medial auditory belt area

CL

caudal lateral auditory belt area

CPB

caudal parabelt auditory area

Ig

granular insula area

Id

dysgranular insular area

Iag

agranular insular area

LSv

lateral sulcus ventral bank

LSd

lateral suclus dorsal bank

Mgad

anterior dorsal nucleus of the medial geniculate complex

MGm

magnocellular nucleus of the medial geniculate complex

MGpd

posterior dorsal nucleus of the medial geniculate complex

MGv

ventral nucleus of the medial geniculate complex

MT

middle temporal visual area

PFC

prefrontal cortex

PM

medial pulvinar nucleus

PO

posterior nucleus of the thalamus

Pro

prostriata visual area

PR

rostral ventral parietal somatosensory area

PV

parietal ventral somatosensory area

R

rostral core auditory area of monkey

RI

retroinsular area

RM

rostral medial belt auditory area

S2

secondary somatosensory area

SG

supragranular nucleus of the thalamus

STG

superior temporal gyrus

STP

superior temporal polysensory area

STSd

dorsal bank of superior temporal sulcus

Tpt

temporoparietal area

V1

primary visual cortex

V2

secondary visual cortex

VIP

ventral intraparietal area

VPI

inferior division of the ventral posterior nucleus

VPS

superior division of the ventral posterior nucleus