Listening to Another Sense: Somatosensory Integration in the Auditory System

Abstract

Conventionally, sensory systems are viewed as separate entities, each with its own physiological process serving a different purpose. However, many functions require integrative inputs from multiple sensory systems, and sensory intersection and convergence occur throughout the central nervous system. The neural processes for hearing perception undergo significant modulation by the two other major sensory systems, vision and somatosensation. This synthesis occurs at every level of the ascending auditory pathway: the cochlear nucleus, inferior colliculus, medial geniculate body, and the auditory cortex. In this review, we explore the process of multisensory integration from 1) anatomical (inputs and connections), 2) physiological (cellular responses), 3) functional, and 4) pathological aspects. We focus on the convergence between auditory and somatosensory inputs in each ascending auditory station. This review highlights the intricacy of sensory processing, and offers a multisensory perspective regarding the understanding of sensory disorders.

1. Introduction

The world around us is complex and dynamic with a host of information detailing its physical arrangement. To make sense of this sea of information, our senses convert external physical stimuli into neural representations, allowing us to interact with these dynamic conditions. The auditory system transduces external pressure oscillations into neural activity in the cochlea, which is then carried to the central nervous system. Communication and localization are two fundamental tasks of the auditory system; without these, quality of life would be severely degraded.

To improve auditory signal processing, the auditory system makes use of multisensory integration, or the process of combining information from multiple sensory modalities in a non-linear fashion (Stein and Meredith, 1990). A simple form of auditory-somatosensory integration is demonstrated via the skin parchment illusion (Jousmaki and Hari, 1998). For this experiment, subjects rubbed their hands together while experimenters recorded the sound produced. When played back, the subjects reported the skin on their hands turning dry as parchment. Furthermore, amplifying just the high frequency components of the playback produced a stronger sensation of roughness. This experiment demonstrated that auditory input can affect other sensory responses, and that altering characteristics of the auditory signal can fine-tunethese responses. These are hall marks of multisensory integration.

Multisensory integration allows for more precise environmental representations than possible with a single sensory system, by enhancing coincident features from each sense (Meredith, 2002). For example, people localize sound faster and more accurately when visual input is coincident with an auditory signal (King, 2009). Another example is the suppression of self-generated sounds such as chewing, which occurs via integration of somatosensory information produced by jaw motion with the sounds produced (Shore and Zhou, 2006). The result is a decrement in neural circuit noise, thereby enhancing the interpretation of externally-generated sounds.

The nervous system is highly adaptive, with reorganization of synaptic connections and strength in response to loss of afferent drive. Cross-modal compensation, in which one sensory system replaces another following loss of input, is one facet of multisensory integration (Bavelier and Neville, 2002). Utilization of dormant neural circuitry can improve the functionality and processing power of the remaining senses. For example, deafened animals show increased visual acuity and touch sensitivity after redirection of somatosensory inputs to auditory cortex (Merabet and Pascual-Leone, 2010). These processes occur at every stage in the auditory system and will be explored in this chapter with an emphasis on the functional aspects of somatosensory-auditory integration and cross-modal reorganization following deafness.

2. Multisensory Integration in the Cochlear Nucleus

The cochlear nucleus (CN) is the first central nervous system station in the auditory system that integrates multisensory information. The functional relevance of this integration can be demonstrated through manipulation of the somatosensory system. In a similar but inverse phenomenon to the parchment skin illusion, jaw maneuvers can alter auditory perception or produce the sensation of tinnitus (W. Hartmann, personal communication) in normal subjects, and can modulate tinnitus in up to 80% of tested subjects (Levine, 1999, Levine, et al., 2003, Sanchez and Rocha, 2011). These perceptions likely arise in part from projections from the trigeminal and dorsal column systems to the CN (Fig. 1). Specific sensory projection neurons to the CN originate in the trigeminal ganglion and the spinal trigeminal nucleus (Sp5) (Haenggeli, et al., 2005, Shore, et al., 2000, Zhou and Shore, 2004), dorsal root ganglion and dorsal column nuclei (Itoh, et al., 1987, Zeng, et al., 2011, Zhan, et al., 2006, Zhou and Shore, 2004), saccule and vestibular nucleus (Barker, et al., 2012, Bukowska, 2002, Burian and Gstoettner, 1988). Most of the projections from non-auditory sensory ganglia and brainstem nuclei terminate in the CN granule cell domain (GCD), but some of them end in magnocellular CN regions (Gomez-Nieto and Rubio, 2011). Anterograde and retrograde tract tracing demonstrate that the trigeminal ganglia directly innervate neurons in the cochlea, middle ear, shell area of the ventral CN (VCN) including the GCD and the fusiform cell layer of the dorsal CN (DCN). Their synapses contain small, spherical vesicles, indicating excitatory transmission (Shore, et al., 2000). The interpolar and caudal Sp5 subnuclei project to many of the same CN regions as the trigeminal ganglion. These nuclei primarily relay pressure and proprioceptive information from the jaw, face and scalp, but not temperature or pain. Electrophysiological studies in DCN have demonstrated the functional activity of this pathway, linking the perception to a plausible neural mechanism (Koehler, et al., 2011, Shore, et al., 2008). It is likely that somatosensory integration also occurs in VCN, but detailed study has been limited (Shore, et al., 2003). Thus, we focus on the projections to the molecular and deep layers of the DCN. These projections likely provide information related to the orientation of the ear relative to the body, as well as the suppression of self-generated sound such as chewing.

Fig. 1. Multisensory integration in the central auditory pathway.

Auditory pathways begin in VCN/DCN: ventral/dorsal cochlear nucleus; The granule cell domain receives a majority of inputs from the somatosensory system (GCD: or marginal cell area of VCN). Somatosensory input is thus “already processed” as it traverses the central auditory pathwyay. Further inputs occur at each separate location. SOC: superior olivary complex; ICC/ICX: central/external nucleus of inferior colliculus; MGv/MGm/MGd: ventral/medial/dorsal nucleus of medial geniculate body; TG: trigeminal ganglion; Sp5: spinal trigeminal nucleus; DRG: dorsal root ganglion; DCoN: dorsal column nucleus; PV: posterior ventral nucleus of thalamus; S1/S2: primary/secondary somatosensory cortex.

2.1. Adaptive filtering in the fusiform cell complex

The DCN processes externally-generated auditory stimuli partly by comparing them with the sounds produced by an animal’s own movements (Roberts and Portfors, 2008). A cerebellar-like circuitry, consisting of a principal output cell, the fusiform cell, and several inhibitory interneurons (Fig. 2) assist in vertical plane sound localization (Nelken and Young, 1997, Neti, et al., 1992), and suppression of self-generated sounds.

Fig. 2. Cerebellar-like fusiform cell complex is a site of multisensory integration.

Fusiform cells (Fu) in the DCN respond to elevation-related acoustic notches via the D-stellate (D-M)/vertical cell (V) circuit: they require orientation information from oro-facial structures to compute this response, which is provided by the cartwheel cell (Ca)-parallel fiber (p.f.) circuitry. Gr: granule cell; m.f.: mossy fiber; a.n.f: auditory nerve fiber; DAS/IAS: dorsal/intermediate acoustic stria.

Multisensory circuits function as adaptive filters

Fusiform and cartwheel cells (Fig. 2) receive excitatory input from unmyelinated axons of granule cells, the parallel fibers (Golding and Oertel, 1997). The granule cells relay non-auditory information in this manner to the DCN. Activation of parallel fibers enhances fusiform cell activity through direct input onto their apical dendrites, but inhibits them via the cartwheel cells, which are inhibitory interneurons that provide inhibitory input to the fusiform somata (Golding and Oertel, 1996).

Adaptive filters adjust their filtering properties in response to dynamically changing signals. An example of an adaptive filter in action is exemplified in the rejection of 60 Hz noise and its harmonics from an electrocardiogram (Fig. 3). Adaptive filters predict the phase and magnitude of the noise and harmonics by calculating an error relative to a reference signal, and subtract the noisy signal. Sensory perception proceeds in a similar manner (von Holst, 1954): To produce a motor action, an organism generates a command to affect the movement, along with a prediction of the movement’s sensory representation. Perception then results from environmentally-produced differences between the afferent inputs and the prediction, a form of error generation like that produced by an adaptive filter. The perception of ticking provides an example of this principle. It is widely known that a person cannot tickle oneself as the nervous system would always predict their tickle, and negate the tickle response.

Fig. 3. Adaptive filters separate signal from dynamically changing noise.

The unprocessed signal (top panel) is corrupted by the presence of variable 60 Hz noise and harmonics (middle panel). An adaptive filter uses information from the movement signal to calculate errors in the unprocessed signal, isolating the relevant signal (lower panel).

Adaptive filtering requires real time dynamic gain control to change outputs in response to variable errors. In cerebellar-like circuits, this is implemented through spike-timing dependent plasticity (STDP)(Bell, 2001, Bell, et al., 2008). Temporal differences between pre- and postsynaptic activity induce long term changes in neural firing rates of the principal cells. Hebbian-like plasticity refers to an enhanced neural response when presynaptic inputs predict (precede) an action potential whereas anti-Hebbian plasticity refers to decreases in neural output when presynaptic inputs fail to predict/precede an action potential.

The electric eel provides an example of a biological adaptive filter with its radar-like electrosensory system (Bell, 2001, Bell, et al., 2008). The eel produces external electrical pulses from a specialized discharge organ that reflect off surfaces in the environment and return to the eel. However, in order to swim the eel must move its body, changing the position of the electroreceptor relative to the electric discharge organ. This change in position introduces error into the sensor that must be accounted for to accurately respond to external signals. This system’s electroreceptive sensors implement anti-Hebbian plasticity at parallel fiber-to-pyramidal cell synapses. The discrepancy between external signals and predictable self-generated signals forms a “negative image” of the self-generated signal (Bell, et al., 1997, Bell, et al., 1999), which is subsequently subtracted from the neural response. Through this process, the eel is able to localize objects in its environment by identifying exclusively external electrical signals. The similarity in structure and physiology between the DCN cell complex and the electric eel sensory organ suggest a similar structure-function relationship (Roberts and Portfors, 2008).

In the mammalian DCN, cartwheel cells inhibit fusiform cells in a feed forward inhibitor model. Anti-Hebbian plasticity at parallel fiber to cartwheel cell synapses regulates the inhibitory gain of the cartwheel cells, providing a mechanism for forming a negative image of self-generated sounds. Plasticity occurs through changes in voltage gated ion channel activity or membrane distributions (Zhang and Linden, 2003), providing a gain-control mechanism intrinsic to the neural circuitry that is input and timing dependent (Kanold and Manis, 1999, Manis, 1990, Tzounopoulos, et al., 2004). These processes reflect STDP (Tzounopoulos, et al., 2004, Tzounopoulos, et al., 2007). Self-generated sounds such as self-vocalizations introduce changes to the response properties of fusiform cells in response to the external sounds from the vocalization. These responses are further altered by somatosensory inputs from the vocal tract to the granule cell domain and subsequently suppressed before being relayed to the inferior colliculus (Koehler and Shore, 2013), thereby increasing the fidelity of signals conveyed to higher cortical structures.

2.2. Deafferentation rewires the auditory pathway

Several research groups have shown that loss of cochlear input drives synaptic reorganization in which inputs from non-auditory structures are up-regulated (Kujala, et al., 2000, Lomber, et al., 2010, Shore, 2011). This cross-modal compensation has been demonstrated in vitro, in vivo and from a systems level perspective across all levels of the nervous system. The wide scope of this phenomenon suggests that a property intrinsic to individual neurons in the neural network drives this process. While originally demonstrated in cortex, these effects are not restricted to cortical structures, nor are they inherently cortical processes: Meredith et al. have proposed that somatosensory invasion of auditory cortex following deafness is driven by up-regulation of somatosensory inputs into lower levels of the auditory pathway, rather than new axon sprouting or re-innervation (Allman, et al., 2009). The first location at which these changes occur is the CN (Shore, et al., 2008, Zeng, et al., 2009, Zeng, et al., 2011).

The vesicular glutamate transporters (VGLUTS) are transporter proteins that pack glutamate into vesicles prior to synaptic release and are traditionally used to track glutamatergic projections. This class of protein is ubiquitously expressed in the central nervous system; however, the differential expression of isoforms may vary according to neural structure. In the CN, VGLUT1 is predominantly associated with auditory nerve fiber terminals (and thus activity), whereas VGLUT2 is associated with somatosensory nuclei and their brainstem projections to the CN granule cell domain and magnocellular regions (Zhou, et al., 2007). Following unilateral deafferentation of the cochlea, ipsilateral VGLUT1 expression in CN regions receiving auditory nerve fiber inputs is decreased significantly, as would be expected following deafening. In contrast, ipsilateral VGLUT2 expression is significantly increased compared to normal and contralateral CN, particularly in regions associated with non-auditory input into the CN (Zeng, et al., 2009). A later study demonstrated that the increase in VGLUT2 reflected an increase in the number of terminals from two somatosensory nuclei, Sp5 and cuneate nucleus (Zeng, et al., 2012). Tying these together is the temporal correlation between spiral ganglion loss and VGLUT2 expression up regulation: animals with two weeks of hearing loss demonstrated lower spiral ganglion cell counts and higher VGLUT2 levels than animals with only a single week of hearing loss.

The rebalance of excitatory inputs to the CN following de-afferentation demonstrates cross-modal compensation at a molecular level (Zeng, et al., 2009). Physiologically, these changes are reflected in increased sensitivity to trigeminal stimulation and enhanced bimodal integration (Shore, et al., 2008). Importantly, the latter study showed that only those DCN cells with connections to the somatosensory system showed increased spontaneous rates after noise damage, a physiologic correlate of tinnitus (Shore, et al., 2008). Thus, increased excitatory drive to the CN from non-auditory sources is a plausible candidate for the increase in spontaneous rates and enhanced stimulus driven activity seen in tinnitus and hearing loss in the CN as well as more central auditory stations (Kalappa, et al., 2014, Kaltenbach, 2007, Kaltenbach and Godfrey, 2008, Vogler, et al., 2014). Noise overexposure is the most commonly reported cause of tinnitus, followed by temporomandibular and other oro-facial somatosensory insults (Levine, et al., 2007). In this model, noise damage leads to diminished auditory drive, which in turn leads to increased excitatory input from other non-auditory regions. Normal functioning DCN auditory-somatosensory interactions are predominantly suppressive but in tinnitus models, enhanced glutamatergic drive from non-auditory structures, as well as synaptic plasticity, alters these interactions to produce enhancement of neural activity (Dehmel, et al., 2012, Koehler and Shore, 2013). The mechanisms by which the CN adaptively filters auditory signals are also altered following noise overexposure and tinnitus: the timing rules that demonstrate this integration are inverted (Hebbian to anti-Hebbian), and are broadened in animals showing behavioral signs of tinnitus (Koehler and Shore, 2013). This enhanced integration window likely plays a role in the increased excitatory drive seen in tinnitus models.

3. Multisensory Integration in the Auditory Midbrain and Thalamus

3.1. Auditory and non-auditory inputs to Inferior Colliculus

Separate processing streams of auditory inputs from the brainstem converge in the inferior colliculus (IC; Fig. 1). The IC consists of central (ICC) and external nuclei (ICX; also referred to as the shell, or pericentral region, and can be subdivided to ventral and dorsal regions). The divisions are identified by neuronal morphology, dendritic branching, inputs, as well as functional properties (Cant and Benson, 2006, Morest and Oliver, 1984, Oliver, 2005). ICC receives direct auditory inputs from DCN and VCN, as well as the superior olivary complex (SOC). The ventral ICX receives inputs primarily from DCN, lateral SOC and local ICC interneurons (Cant, 2013, Coleman and Clerici, 1987, Loftus, et al., 2008). ICC neurons are sharply tuned with short latency responses, whereas ICX units exhibit broad frequency tuning and habituating responses (Calford and Aitkin, 1983). ICX is also a multisensory area. Major projections are received from somatosensory centers, Sp5 and dorsal column nuclei (DCoN (Aitkin, et al., 1981, Li and Mizuno, 1997, Tokunaga, et al., 1984, Zhou and Shore, 2006) and primary somatosensory cortex (Cooper and Young, 1976), as well as visual inputs from the retinal ganglion (Cooper and Cowey, 1990), SC (Adams, 1980, Coleman and Clerici, 1987), V1 (Cooper and Cowey, 1990). Some terminals from the aforementioned areas also sparsely extend to ICC, but here we focus on the bimodal properties of ICX.

3.2. Bimodal responses in ICX

Multisensory inputs to ICX are functionally relevant as ICX neurons respond to somatosensory or visual stimulation. Auditory and visual integration in ICX has been reviewed by (Gruters and Groh, 2012), thus, here we focus on somatosensory inputs. Unimodal electrical stimulation the dorsal column pathway at cervical nerve, C4 and tibial nerves from forelimbs and hindlimbs results in excitation of 20% of ICX units, while 55% of units respond bimodally to combined auditory-somatosensory stimulation (Aitkin, et al., 1978). Of the bimodal units, 2/3 are inhibited and 1/3 excited by paired acoustic and somatosensory stimulation. The strongest somatic influence is derived from the contralateral side, i.e., the origin of the auditory inputs. Furthermore, the somatic field for the entire body is represented in ICX (Aitkin, et al., 1981), albeit with a less defined topographic arrangement than that from DCoN or Sp5.

Functional inputs via the trigeminal pathway (contralateral) to ICX were later confirmed by (Jain and Shore, 2006) using trigeminal ganglion stimulation in the guinea pig. Nearly 65% of ICX units exhibited either bimodal suppression or enhancement, quantified as percent activity increase/decrease from the maximal unimodal sound-evoked activity (Jain and Shore, 2006). Of the bimodal units, 72% showed suppression (Fig. 4) and 28% showed enhancement. The proportion was similar to the cat study by (Aitkin, et al., 1978).

Fig. 4. Auditory-somatosensory integration in ICX is manifested as altered bimodal responses (sound and trigeminal ganglion electrical stimulation).

(A) A representative unit shows no response to unimodal somatosensory stimulation (upper panel), strong responses to sound (middle panel), and suppressed bimodal responses (bottom panel). Blue horizontal bar indicates tone duration. Red vertical bar shows electrical stimulus. (B) Rate-level function showing bimodal suppression at suprathreshold intensities (20–50 dB SPL). Figure adapted from (Jain and Shore, 2006).

3.3. Function of multisensory integration in IC

The representation of sensory space is universal across sensory systems, and many tasks require integration of more than one representation of the sensory stimulus. The superior colliculus (SC) serves as a good example. Primarily a visual structure, SC contains an auditory space map – with SC neurons in different regions responding to acoustic stimuli from different spatial coordinates (Middlebrooks and Knudsen, 1984, Palmer and King, 1982). The SC also contains visual and somatosensory maps (Meredith and Stein, 1986). The convergence of multisensory spatial representations serves higher functions such as orientation and localization. This type of sensory map integration is not restricted to SC, and is also present in the auditory system. ICX contains an auditory space map (Binns, et al., 1992), which encodes localization of sound, as well as a somatotopic map (Aitkin, et al., 1981). Due to the presence of both sensory maps, ICX may be a site of auditory and somatosensory spatial integration. In particular the somatotopic map in ICX may participate in sound localization coding and orientation. This suggestion is strengthened by the demonstration that ICX sends direct output to the SC (Druga and Syka, 1984, Van Buskirk, 1983), where sensory maps are known to converge. Indeed, a later study confirmed that ICX is essential in constructing an auditory space map in SC (Thornton and Withington, 1996). In addition, ICX projects to the somatosensory area of posterior ventral thalamus (Ledoux, et al., 1987) and thus may serve additional purposes in processing ascending somatosensory information.

ICX plays a role in in vocalization behaviors. While ICC neurons respond indiscriminately to self-generated and external sounds, ICX neurons are selectively suppressed during self-vocalization. (Tammer, et al., 2004). In addition, ICX neurons fire in advance of self-produced vocalization signals, whereas ICC neurons do not fire during pre-vocalization (Pieper and Jurgens, 2003). Since vocalization activates the trigeminal pathway (Kirzinger and Jurgens, 1991); it is likely that the pre-vocalization onset activity and suppression of self-generated vocal signals are derived from the somatosensory inputs to ICX (Jain and Shore, 2006, Li and Mizuno, 1997, Zhou and Shore, 2006). This likely reflects the adaptive filtering process that is also evident in DCN (See section 2.1).

3.4. MGm: A multisensory center in the auditory thalamus

All areas of IC project primarily to the ipsilateral medial geniculate body (MGB) and sparsely to the contralateral MGB with both excitatory and inhibitory projections (Ledoux, et al., 1987, Powell and Hatton, 1969, Winer, et al., 1996). MGB contains ventral, dorsal, and medial subdivisions (Jones and Rockel, 1971, Morest, 1975, Winer, et al., 1988). MGv (ventral MGB) receives inputs only from ICC. It is considered to be a relay station due to its shared physiological characteristics with ICC: short latency responses, homogenous neurons, and fine tonotopicity (Aitkin and Webster, 1972). In contrast, MGd neurons respond only weakly to auditory stimuli; their inputs are exclusively derived from the ventral medial edge of ICX (Calford and Aitkin, 1983). MGm, also known as the magnocellular region, is the multisensory division of the auditory thalamus (Aitkin, 1973). It is analogous to ICX with broader tuning properties and reception of non-auditory inputs (Ledoux, et al., 1987, Ryugo and Weinberger, 1978). The auditory inputs to MGm are derived from ICC and ICX (Calford and Aitkin, 1983), as well as a separate, direct pathway from the DCN and CN small cell cap region that bypass IC (Anderson, et al., 2006, Malmierca, et al., 2002, Schofield, et al., 2014). Multisensory inputs to MGm – and partially MGd – include somatosensory afferents from the spinal-thalamic, dorsal column, and the trigeminal pathways (Jones and Burton, 1974, Lund and Webster, 1967a, Lund and Webster, 1967b), as well as visual afferents from SC (Linke, et al., 1999).

Multisensory responses in MGm have been well documented, Strong unimodal responses (excitatory as well as inhibitory) to vestibular, touch, or nociceptive stimulation are observed (Wepsic, 1966), see Table 1. Latency analyses reveal a direct pathway from the vestibular nucleus, which was assumed and later confirmed (Kotchabhakdi, et al., 1980). Due to the absence of bimodal responses in this region, a separate study speculated that MGm may be divided into sensory-specific regions (Love and Scott, 1969). This view was disproven by (Khorevin, 1978), who found 65% of bimodal neurons responding to acoustic and electrical stimulation on the contralateral forelimb. The latency of the somatosensory response was around 14–17 ms, a similar timeframe to the average transmission time from the somatosensory afferents to the posterior-ventral somatosensory thalamus. Even though somatosensory stimulation did not evoke spike activity in MGv, it elicited IPSPs when recorded intracellularly, and inhibited auditory responses to clicks (Khorevin, 1980a). This type of subthreshold processing is common in neurons bordering MGm as well as within MGm (Fig. 5), wherein auditory responses are inhibited by somatosensory stimulation (Donishi, et al., 2011, Khorevin, 1980b).

Table 1.

An early study by (Wepsic, 1966) showed that MGm is a multisensory region. Number represents count of MGm units responding to different sensory stimuli. Adapted from (Wepsic, 1966).

	Increase in SpAc >50%	Decrease in SpAc >50%
Caloric* (Vestibular)	78	28
Click (Auditory)	93	-
Touch (Somatosensory)	55	-
Nociceptive	63	16
Vestibular Nucleus stimulation	73	26

^*Activation of peripheral vestibular nerves and the ocular-motor pathway by cold water applied at the inner ear

Fig. 5. Bimodal suppression of an MGm unit.

When somatosensory stimulation (transdermal electrical pulses on contralateral forelimb) preceded an auditory click stimulus by ~20 ms (trial 3–7), auditory responses (action potentials) were suppressed. Figure adapted from (Khorevin, 1980b).

3.5. Sensory integration and conditioning responses

Neuronal conditioning refers to plastic changes in spike responses during a classical conditioning paradigm. An auditory conditioned response, usually tone-evoked, undergoes potentiation after training with unconditioned stimuli (e.g. paw shocks), which results in physiological responses such as pupil dilation or eye blinks (Maren, 2001). The amygdala plays a crucial role in mediating such processes (Davis, 1992). Although MGm sends excitatory projections to amygdala (LeDoux, et al., 1990), this auditory pathway does not merely relay auditory stimuli to the amygdala and to associated cortical regions for conditioning. MGm is an essential part of the conditioning circuitry (Campolattaro, et al., 2007, Cruikshank, et al., 1992, Rogan and LeDoux, 1995, Ryugo and Weinberger, 1978). MGm, but not MGv, shows increased neural activity in response to training (Ryugo and Weinberger, 1978). This may reflect auditory-somatosensory integration, as much as the intrinsic plasticity of the ICX–MGm synapses, which undergo long-term potentiation (Gerren and Weinberger, 1983, Rogan and LeDoux, 1995). Because electrical stimulation of MGm alone is sufficient to produce conditioned responses as well as fearful behaviors (Cruikshank, et al., 1992) it has become a major player in these behaviors. Recently, it has been suggested that MGm, and not amygdala, is the putative neural center of auditory conditioning, which supported by the fact that MGm receives sufficient multimodal inputs and outputs to all the necessary brain regions to carry out amygdala-related functions (Weinberger, 2011). Indeed, lesion studies reveal that ICX, with similar characteristics of multisensory integration as MGm, is another essential site for conditioning processing (Freeman, et al., 2007, Heldt and Falls, 2003).

4. Multisensory Integration in the Auditory Cortex

Located bilaterally on the upper side of the temporal lobes, the auditory cortex is commonly divided into three distinct regions. The primary auditory cortex (AI), a core area for auditory processing, is surrounded by the secondary auditory cortex (AII) or “belt” region, a first order association area, which is further bordered by second order association areas of the “parabelt” region (Fig. 1).

4.1. Somatosensory projections to the auditory cortex

In primates, A1 receives somatosensory input from somatosensory cortex, S2 (Cappe and Barone, 2005). In addition, the caudomedial (CM) and caudolateral (CL) auditory belt areas receive direct projections from the retroinsular (RI) and granular insula (Ig) areas of S2 (de la Mothe, et al., 2006a, Hackett, et al., 2007). Somatosensory information is also relayed by the thalamico-cortical connections to the auditory cortex. The belt, parabelt and more sparsely A1 areas receive projections from the MGm and MGd (Hackett, et al., 2007, Smiley and Falchier, 2009). The dominant thalamic projection to CM is provided by MGm and the anterior dorsal division of the medial geniculate complex (MGad) while the posterior division targets the rostromedial (RM) auditory belt area (de la Mothe, et al., 2006b). Additional somatosensory projections to the auditory belt and parable regions are mediated by projections from suprageniculate and limitans (Sg/Lim) and medial pulvinar (PM) nuclei (Hackett, et al., 2007).

4.2. Mechanisms and functional implications of auditory-somatosensory integration in the auditory cortex

In cortex, as in subcortical structures (see sections 1 and 2), multisensory integration is mediated by neurons independently activated by more than one sensory input. In addition, neurons from a specific cortical area may be activated by a single modality but their responses are significantly modulated, i.e. enhanced or suppressed, by the input of a second modality. Examples include subthreshold modulation of auditory cortex responses by somatosensory and visual inputs(Allman and Meredith, 2007, Dehner, et al., 2004, Meredith and Allman, 2009), subthreshold auditory and visual input to prefrontal cortex (Sugihara, et al., 2006), and subthreshold modulation in subcortical areas (see sections 1 and 2). This suggests that this mechanism may be one of general importance for multisensory processes throughout the brain. In the auditory cortex, neurons from AI (Banks, et al., 2011), and the auditory cortical field of the anterior ectosylvian (FAES) (Meredith and Allman, 2009), resposes to auditory stimuli are significantly modulated when auditory and visual or somatosensory stimuli, respectively are combined. As the degree of multisensory integration depends on the synaptic strength of the neural connections mediating various modalities, subthreshold multisensory processing can be viewed as an intermediary stage between unimodal and multi-modal responses. Therefore, this type of processing may be common to other fields of the auditory cortex (Bizley and King, 2009).

Synaptic plasticity

One mechanism that significantly impacts multisensory integration is synaptic plasticity. Plasticity induction can be achieved by a) combining two or more auditory stimuli with specific temporal and frequency characteristics or b) combining auditory and somatosensory (or other sensory) stimuli in similar configurations to the ones used in (a).

Auditory-auditory stimulation

Changes in cortical frequency representations can be induced by presenting asynchronous auditory stimuli with specific temporal and frequency characteristics. Such stimulation protocols can be designed to be consistent with in vitro stimulation configurations that induce STDP. Dahmen et al. demonstrated that plastic changes in the receptive fields of neurons in A1 can be induced by best frequency tones presented in close temporal proximity with tones at a “non-preferred” frequency (Dahmen, et al., 2008) (Fig. 6). For that study, the frequency tuning curve of the neuron was determined and the best frequency and “ non-preferred” frequency (i.e. outside the tuning curve) were selected for the stimulation protocol (Fig. 6 A). Repetitive pairing of tones at best and non-preferred frequencies (Fig. 6 B) was presented with specific time intervals between the consecutive tones. Depending on the order of the stimuli, shifts in the best frequency occurred (Fig. 6 C and D). These changes were most effective at intervals of 8 and 12 ms, consistent with Hebbian learning rules observed in STDP studies (Fig. 6 E). The shifts in best frequency representations were accompanied by changes in firing rate and tuning widths of the auditory responses (Fig. 6 F). These findings emphasize plastic properties of the A1 responses and the importance of millisecond scale timing of the sensory input in shaping the auditory neural function.

Fig. 6. Stimulus-timing dependent plasticity of auditory cortical representations.

(A) Iso-intensity tuning curve. Two frequencies (preferred frequency at best frequency (PF) and non-preferred Frequency at the edge of the tuning curve (NPF)) are selected to construct the stimulation protocol. (B) Examples of stimulation protocol trains of positive conditioning when NPF tones precedes PF tones and negative conditioning when PF precedes NPF tone presentation. (C–D) Examples of shifts in the tuning curve observed in response to positive (C) and negative (D) conditioning. (D) Tuning curve shifts for the first and second stimuli blocks are displayed for negative conditioning in light and yellow green and their average in red. The raw tuning curve and its Gaussian fit before conditioning are shown in solid and dashed black lines. (E) Percent shifts in best frequency are presented as a function of conditioning interval. Negative intervals correspond to negative conditioning and positive intervals correspond to positive conditioning. (F) Changes in the firing rate and tuning width for the same neural cell population as in (E). Figure adapted from (Dahmen, et al., 2008).

Auditory-somatosensory stimulation

A recent study (Basura, et al., 2012) investigated the effects of multisensory plasticity inducing stimulation on the bimodal integration of neurons in DCN and A1. The bimodal stimulation protocol consisted of auditory stimuli followed or preceded by electrical stimulation of a somatosensory nucleus (Sp5). The difference between the onset of auditory and electrical stimulation, called a bimodal interval, was between −40 to 40 ms. where negative values indicate auditory stimulation followed by Sp5 stimulation and positive bimodal intervals indicate Sp5 followed by auditory stimulation. Bimodal stimulation resulted in long-lasting effects in the form of facilitation or suppression for up to an hour (Fig. 7). An example of a persistent change (increase) in the response of a single unit in A1 is shown in Fig. 7 A, in comparison with the response of a fusiform cell in DCN (Fig. 7 B), which shows suppression. As illustrated in Fig. 7 C and D for the same bimodal stimulation interval, in both DCN and A1, some neurons responded with facilitation while others respond with suppression. In cortex however, the facilitation effects were stronger.

Fig 7. Bimodal stimulation results in persistent changes in A1 (A,C) and DCN (B,D).

Upper panels (A and B) show peri-stimulus time histograms (PSTHs) demonstrating the persistent 25 enhancement (A) and suppression (B) of somatosensory-auditory stimulation. Pre-bimodal responses to sound alone are shown in blue; post-pairing responses to sound alone are shown in red. Pairing interval: 10 ms. Black bar under PSTHs indicates tone duration. Histograms for A1 (C) and DCN (D) demonstrate the percentage of units showing persistent changes in firing rates following bimodal stimulation at a pairing interval of 10 ms. Responses to the right of zero are facilitated, while those left of zero are suppressed. Figure adapted from (Basura, et al., 2012).

4.3. Subcortical induced cortical reorganization

Thalamo-cortical (TC) projections provide the main ascending auditory input to the auditory cortex. Plasticity of the synapses mediating this communication were thought to be limited to the early stages of development but recent research indicates that the TC projections to the auditory cortex can exhibit long-term potentiation at various stages of development, but become “gated” with age (Chun, et al., 2013). For instance, cholinergic activation can release presynaptic gating through muscarinic receptors M(1) that down-regulate adenosine inhibition of neurotransmitter release (Blundon, et al., 2011). Once presynaptic gating is released, the TC synapse can express long term potentiation through metabotropic glutamate receptors postsynaptically (Blundon, et al., 2011). Interestingly, consecutive sound presentations can induce changes in the firing pattern of the thalamic neurons. In vitro studies showed that these neurons respond with a burst to brief depolarizing pulses mimicking the input of a brief sound stimulation. When a second pulse is added within hundreds of milliseconds, the thalamic neurons respond only with a single spike. These Ca(v)3.1 mediated switches from bursting to tonic firing depress the TC synapses contributing to forward suppression of auditory cortex activity (Bayazitov, et al., 2013).

The ventral and dorsal divisions of the MGB, which comprise the lemniscal and nonlemniscal thalamic auditory nuclei, mediate different types of plasticity in the auditory cortex (Ma and Suga, 2009). Electrical stimulation of the narrowly frequency tuned MGv neurons `evokes a shift of A1 frequency tuning curves toward the tuning curve of the MGv neurons but preserves the tuning curve width. In contrast, electrical stimulation of the broad frequency tuned MGm neurons broadens the tuning curve of the auditory neurons but it doesn’t mediate a frequency shift. These findings suggest that MGv neurons are more likely to mediate tone-specific plasticity in AI neurons, thus adjusting frequency related auditory signal processing while the MGm neurons facilitate a non-specific plasticity, increasing the sensitivity of the AI cortical neurons (Ma and Suga, 2009).

The pedunculopontine tegmental nucleus is an important brainstem cholinergic nucleus involved in learning and plasticity. Pairing electrical stimulation of this region with a tone induces major changes in frequency tuning of the A1 neurons, shifting the best frequency of the neurons towards the frequency of the paired tone (Luo and Yan, 2013).

4.4. Neuromodulation

Basal forebrain cholinergic input to the auditory cortex (Bajo, et al., 2014, Mesulam, et al., 1983) can modulate sensory processing and stimulus-specific plasticity depending on the behavioral state of the subject. For instance, paired auditory stimulation with electrical stimulation of the nucleus basalis induced a significant reorganization of AI in the adult rat as reflected by the reshaping of the receptive fields, changes that mirror the remodeling of the receptive fields as a result of certain types of behavioral training (Edeline, et al., 2011, Kilgard and Merzenich, 1998). Selective loss of cholinergic input provided by nucleus basalis reduces sound localization accuracy and prevents adaptive reweighting of the auditory localization cues in response to chronic occlusion of one ear (Leach, et al., 2013). Furthermore, non-specific augmentation of cortical responses to auditory stimuli may be mediated by the histaminergic system (Ji and Suga, 2013), suggesting that together these two neuromodulators may mediate differential gating of cortical plasticity.

Noradrenaline can also modulate cortical encoding of auditory stimuli (Manunta and Edeline, 1999). Pulses of noradrenaline paired with tone presentations at frequencies close (within ¼ of an octave) to the best frequency of the cortical auditory neurons, can induce long lasting, selective changes in their receptive fields opposite to those induced by acetylcholine neuromodulation (Manunta and Edeline, 2004).

An interesting neuromodulatory effect on multisensory integration in the auditory cortex was demonstrated in female mice with pups (Cohen, et al., 2011). In vivo recordings revealed that exposure to pups’ body odor reshaped the neuronal responses to pure tones and natural auditory stimuli. Neurons from lactating mothers were also more sensitive to sounds. Together these uni- and multisensory cortical modulation effects may facilitate the detection and facilitation of pup distress calls.

4.5. Multiplexing stimulus information

Multiplexing is employed in multisensory auditory neurons to co-represent and bind multimodal stimuli of relevance. One form of multiplexing is mediated by precise spike timing relative to the phase of low-frequency network oscillations, which may convey additional information about sensory stimuli. This mechanism has been observed in many brain areas including visual cortex where spike timing relative to the phase of delta band oscillations (1–4Hz) carries information about natural visual stimuli (Montemurro, et al., 2008), the hippocampus where it mediates memory function (Manns, et al., 2007, Rutishauser, et al., 2010) and prefrontal cortex where it encodes reward expectancy (van Wingerden, et al., 2010). In A1, spike timing relative to the theta band (4–8Hz) of local field potential (LFP) is informative about the type of sounds presented to awake, passively listening monkeys (Kayser, et al., 2009). Both spike rate and local field oscillations of the auditory cortical neurons encode natural sound stimuli such as animal vocalizations, environmental sounds or segments of speech. However, when these sounds were corrupted by noise, the information encoded by these response features decreased with increasing noise while the information gain in the nested spikes in LFP phase increased with increasing noise. This suggests that spike timing relative to the slow rhythmic neural activity may serve to stabilize the sensory representation against the adverse effects of sensory noise. Investigations of somatosensory-auditory interactions in the macaque A1 revealed that somatosensory inputs appear to reset the phase of ongoing neuronal oscillations such that concurrent auditory inputs arrive during an ideal, high-excitability phase and produce an amplified neuronal response. In contrast, auditory inputs arriving during the low-excitability phase tend to be suppressed (Lakatos, et al., 2007).

4.6. Functional implications of auditory-somatosensory integration

Audiotactile interactions

The human brain utilizes inputs from different senses to construct perceptual representations of objects and events. Interactions between the auditory and somatosensory system play an important role in enhancing the human haptic experience during dynamic contact between the hands and the environment. At least two neurophysiological mechanisms may mediate these interactions: 1) Crossmodal modulation of cortical activity depending on the salience of the stimuli mediates preferential responses of the cortical area processing the more salient stimuli and inhibition of the cortical activity in the area processing the less salient stimuli. For instance, when combining auditory and tactile stimuli, for salient stimuli, MEG responses indicate auditory cortex activation and suppression of the secondary somatosensory cortex (S2) (Lutkenhoner, et al., 2002). In contrast, when tactile stimuli carry more salience, activation of the somatosensory cortex is accompanied by suppression of the auditory responses (Gobbele, et al., 2003). 2) Vibrotactile stimuli and tactile pulses without vibration activate the auditory belt area in animals and humans (Foxe, et al., 2002, Fu, et al., 2003, Schroeder, et al., 2001, Schurmann, et al., 2006) with supra-additive integration (Kayser, et al., 2005), suggesting a specific functional specialization of this brain region.

The auditory -tactile interactions play a particularly important role in music perception (Huang, et al., 2012) and their neural representation is more robust in humans with musical training (Kuchenbuch, et al., 2014).

Suppression of self-generated sounds

While a large research effort has been dedicated understanding how the auditory cortex processes externally-generated sensory signals; significantly less is known about the processing of internally generated sounds. EEG studies aimed to better understand these responses (van Elk, et al., 2014a, van Elk, et al., 2014b) reported reduced N1 auditory components when participants listened to heart beat-related sounds compared with externally generated sounds (van Elk, et al., 2014a), and when participants listened to sounds generated by their own limbs (van Elk, et al., 2014b). The robust and persistent effect that the brain automatically differentiates between the interoceptive and externally generated sounds suggests that a predictive mechanism could be at play, comparable to similar mechanisms mediating sensory suppression of self-generated actions (Blakemore, et al., 1998). Section 1 proposes a mechanism whereby distinguishing self-generated from external stimuli is initiated subcortically through Hebbian plasticity. Whether this process is conserved across central stations or further modified is yet to be determined.

4.7. Pathological cross-modal reorganization and compensation in the auditory cortex

Evidence from deaf, hearing impaired, and cochlear implanted animal models and patients

The development of multisensory neural systems and the ability to integrate crossmodal information requires maturation and a rich sensory experience to achieve its full capabilities (Wallace and Stein, 2007, Yu, et al., 2010). Thus, sensory deprivation may alter or impair multisensory processing, particularly in earlier stages of development with possible permanent effects (Polley, et al., 2013, Whitton and Polley, 2011). One mechanism that can mediate such changes could be crossmodal plasticity, i.e. a substitution of the neural representations of a damaged sensory system by the input and representations of a different sensory modality. For instance, early life or congenital auditory deprivation induces a remodeling of auditory cortex representations. However, both early hearing-impaired (Meredith and Allman, 2012) as well as adult deafened ferrets (Allman, et al., 2009) demonstrate significant crossmodal reorganization of AI as indicated by previously absent, robust responses to somatosensory stimulation.

In congenitally deaf humans, the auditory association area (supratemporal gyrus) is activated by sign language (Nishimura, et al., 1999). Investigations aimed to understand whether this plasticity is dependent on the extent of hearing loss revealed that while the auditory association areas are activated in subjects with total or partial hearing loss, AI was activated only in the subjects with total hearing loss (Lambertz, et al., 2005). Interestingly, studies in subjects with temporary auditory deprivation that was restored by cochlear implants showed that temporary deafness can impair multisensory (audio-tactile) integration (Landry, et al., 2013, Nava, et al., 2014). More specifically, the subjects were tested using the audiotactile illusory-flash effect (Hotting and Roder, 2004) in which simultaneous presentation of a somatosensory stimulus with a large number of successive non-speech sounds can lead to as many tactile as auditory perceptions in normal hearing individuals. Temporarily auditory deprived patients failed to perceive this illusion, thereby unmasking a failure of auditory tactile integration processing.

5. Concluding Remarks

Multisensory integration occurs throughout the auditory system. The CN, the first processing station in the ascending auditory pathway, already processes converging somatosensory inputs for adaptive filtering (Section 2.1), which is important for sound localization and suppression of self-generated signals. Bimodal neurons receiving somatosensory projections, including those showing subthreshold responses, are found in each ascending station: IC, MGB, and AC. Neurons in ICX not only discriminate self-generated from external auditory stimuli, but they form a somatic spatial response map, to aid in orientation to sounds (Section 2.3). MGm, a multisensory area in the auditory thalamus, is a core structure for sensory conditioning, indicating higher level cross-modal syntheses (Section 2.5). Neurons in which multisensory integration occurs commonly demonstrate cross-modal compensation after sensory deprivation. The rebalancing of sensory inputs after deafness (reduced auditory and increased somatosensory) is observed in CN and AC (Section 2.3 and 4.7). A similar mechanism may be expected, although has not yet been observed in IC or MGB.

This sequential multisensory processing across structures – from CN, IC, MGB, towards AC raises the question of whether multisensory inputs processed independently in each ascending station, or is processed information relayed to the next station (i.e. integration in IC merely reflects integration in CN)? Anatomical connections suggest independent processing, as separate terminals from Sp5 and DCoN are found in CN, IC, and MGB. However, a recent study on cross-modal compensation in AC could not attribute observed increases in somatic representation to altered cortical connectivity (Allman, et al., 2009), suggesting that the process of integration and compensation may have originated in lower structures. This substantiates the notion that some already-processed multisensory integration from lower structures is carried to the next. In addition, there is overlapping functional significance within the ascending pathway. For instance, both CN and IC mediate adaptive filtering. If multiple stations are serving a single function, it is more reasonable to assume that the system would avoid redundancy and adapt toward serial rather than parallel processing. Future studies are needed to confirm this hypothesis.

In this chapter we have discussed multisensory integration from the perspective of an ascending system, but there are significant descending projections from AC to MGB, IC, and CN (Schofield, 2011, Schofield and Coomes, 2006, Winer, et al., 2002, Winer and Larue, 1987). In the visual system, descending projections from the visual cortex modulate subcortical bimodal responses in SC and inactivation of cortical projection renders SC neurons incapable of synthesizing multisensory inputs (Stein, et al., 2002). Whether a parallel exists in the auditory system is yet to be elucidated.