Crossmodal Plasticity in Hearing Loss

Abstract

Crossmodal plasticity is a textbook example of the ability of the brain to reorganize based on use. We review evidence from the auditory system showing that such reorganization has significant limits, is dependent on pre-existing circuitry and top-down interactions, and that extensive reorganization is often absent. We argue that evidence does not support the hypothesis that crossmodal reorganization is responsible for closing critical periods in deafness, rather it represents a neuronal process that is dynamically adaptable. We evaluate the evidence for crossmodal changes in both developmental and adult-onset deafness, which start as early as mild-moderate hearing loss and show reversibility with hearing restoration in some cases. Finally, crossmodal plasticity does not appear to affect the neuronal preconditions for successful hearing restoration and given its dynamic and versatile nature, we describe how it can be exploited for improving clinical outcomes after neurosensory restoration.

Crossmodal Plasticity in the Auditory System

Crossmodal plasticity (see Glossary) is an adaptive change in the drive of neurons from one (deprived) sensory input towards another (non-deprived) sensory input, i.e. it is a form of between-sensory-systems (intermodal) plasticity [1]. Crossmodal plasticity represents a textbook example of the brain’s capacity for plastic changes.

In early studies in animal models, a massive crossmodal visual takeover of the auditory cortex has been found when early deafness was combined with aspiration of the auditory midbrain: such combined intervention led to reorganization of anatomical inputs to the thalamus and a large-scale remapping of the auditory cortex by visual inputs [2,3]. These studies demonstrated probably the maximal possible functional adaptation of the auditory cortex to novel input. What was subsequently often overlooked was that this extensive reorganization was made possible by aspiration of the auditory midbrain that allowed the visual afferents to target auditory thalamus (in addition to visual thalamus) during development. That is, anatomical re-routing of visual information to the auditory thalamus (that does not normally accompany sensory deprivation) was a precondition for this massive reorganization. Many textbooks subsequently suggested that in the case of auditory deprivation the entire auditory cortex becomes a battlefield between sensory systems and each of its areas can be recruited for a new sensory function (Fig. 1). This is often considered a key reason for developmental critical periods for sensory restoration: such “colonization” of the auditory system would preclude the processing of auditory inputs after neurosensory restoration, e.g. via cochlear implants. Therefore, such crossmodal reorganization would be a key reason why critical developmental periods close. This has important clinical consequences: To prevent fostering this effect it was traditionally recommended not to use visual communication (sign-language) before cochlear implantation. However, as our review will show, the afore-mentioned concept of a massive crossmodal reorganization and its negative influence is not consistent with recent findings and requires modification.

Fig. 1: Schematic illustration of approximate locations of sensory brain areas in the cat and their reorganization in deafness.

Blue – visual; red – auditory; green – somatosensory; orange – motor; grey – association areas. Mixture of colors at sensory borders (appears as violet, light brown/golden and deeper green) depicts bimodal responsiveness (~ two colors). A) AES, the area of the anterior ectosylvian sulcus, together with area In correspond to human insular cortex [4]. The ectosylvian sulcus (divided into dorsal area, ED, intermediate area, EI, and ventral area, EV) correspond to human superior temporal gyrus [5]. Feline prefrontal cortex is likely multisensory based on tracer studies [6]. Rostral lateral sulcus (rLS) and/or anterior ectosylvian sulcus (AES) allow multisensory integration in the superior colliculus [7]. “Parietal association cortex” are two separate areas within Brodmann area 7 [8,9]. Visual responsiveness was observed in the posterior belt in ED [5], but also in DZ. Auditory and visual responses have been described around the cruciate sulcus in M1. In the pericruciate association cortex visual responses and auditory responses were reported, here shown as grey circles within the motor cortex (orange). Cingulate cortex on the medial hemisphere and ventral limbic areas (entorhinal cortex and parahippocampal gyri) are not shown. B) A putative cortical map under the assumption of massive crossmodal reorganization in congenital deafness. C) An approximate representation of actual cortical organization in congenitally deaf cats according to the evidence as reviewed in the text.

We propose that crossmodal reorganization is a dynamically modified process initiated already in mild or unilateral hearing loss continuing into complete deafness, occurring both in congenital and adult-onset hearing loss (albeit with significant differences). It is a primarily top-down driven process which exploits pre-existing neuronal connections. In its essence, we argue, crossmodal reorganization is not detrimental for neurosensory restoration, rather it is compensatory and can be used to enhance communication before therapy, or aid in enhancing real world speech perception.

Crossmodal effects in deafness

Crossmodal ‘supranormal’ enhancements of visual and somatosensory modalities have been extensively documented in subjects that were deaf from childhood [10–18]. The extensively-studied visual effects have been related to enhanced visual motion detection and localization abilities [12,19–23], better visual change detection [24] or faster reactions to visual stimuli in deaf subjects [25,26]. But not all visual functions show such supranormal performance [27]: there was e.g. no change in visual acuity, brightness discrimination, contrast sensitivity, or visual motion direction sensitivity. Mainly those functions that the visual and auditory systems have in common (like localization, movement detection and change detection, i.e., ‘supramodal’ functions [28]) have the capacity to become compensated by crossmodal plasticity. This specificity of cross-modal reorganization raises the question on the extent and limits of reorganization of the brain with respect to cross-modal plasticity.

In developmental deafness, in both humans and animals models, enhanced visual spatial localization and visual movement detection was associated with cortical auditory areas underlying localization and movement (congenitally deaf cats, CDCs [28]; early deafened cats [29]; humans [23]). In humans, face processing was observed in the temporal voice area of pre-lingually deaf humans [30]. Auditory cortex contribution to face processing was also noted in postlingually deaf adults [31]. Given these findings, it appears that crossmodal reorganization switches the sensory, but not behavioral roles of the auditory cortex [23,28,29]. Supporting this, a recent human study emphasized the overlap of the crossmodally reorganized region on to the previously observed region of auditory motion detection in hearing subjects [32]. This is compatible with a change in driving sensory input in crossmodal reorganization – from auditory to visual, but principally preserving the subsequent corticocortical processing.

The normal auditory system, characterized by exceptional timing precision, provides key “calibration” timing information to other sensory systems (similarly to the way the visual system is key for spatial information due to its high spatial acuity) [33,34]. Indeed, in cases of conflict in timing, hearing can “override” visual information (e.g., auditory capture effect, [35,36]), whereas vision similarly “overrides” hearing in spatial location discrepancy (e.g., ventriloquist effect, [37]). Therefore, visual and somatosensory temporal processing are negatively affected by deafness [33,38]. It is interesting to note that deaf individuals, when tested under well-controlled conditions, may infer timing properties also from spatial cues, and in these situations, only if both are correlated they perform well [39]. Auditory feedback of motor actions through secondary auditory cortex has also been demonstrated to be key for action timing [40]. Consequently, hearing loss may have adverse effects in the spared sensorimotor functions.

Causality and Specificity in Crossmodal Reorganization

Causal evidence of crossmodal reorganization is difficult to obtain. Studies that directly test causality of crossmodal reorganization have been conducted primarily in animal models, where invasive studies are feasible, for instance in the congenitally deaf cat (CDC), an important model of pediatric congenital deafness [41–43]. CDCs show supranormal performance in visual localization and visual motion detection compared to normal hearing cats [28]. Using cooling deactivation of specific auditory areas, activity in these areas can be reversibly silenced and behavior can be assessed to test for potential causal links between deactivated areas and their behavioral functions. Regions found to be subserving supranormal visual performance were posterior auditory field and dorsal auditory cortex (dorsal zone), but primary fields A1 and AAF were not involved [28]. The CDC replicates crossmodal effects observed in humans and allows exact identification of the cortical regions responsible for the behavioral effects. Taken together, the CDC data and the afore-mentioned human studies implicate that crossmodal plasticity shows a strict areal specificity and that crossmodal plasticity switches the sensory but not behavioral roles, i.e., supranormal visual behavior in deafness is subserved by auditory cortical regions underlying the same auditory behavior.

Areas involved in this crossmodal plasticity, when studied using retrograde tracers to reveal their anatomical connections (fiber tracts), generally showed only a small percentage of new (ectopic) connections to non-auditory areas. That means that the connectivity patterns to other sensory systems were similar for CDCs and hearing cats, only individual connection strengths were changed in CDCs (dorsal cortex: [44], posterior auditory field: [45]; for later deafening in cats see [46,47]). The overall effect size of ectopic connections (which were a small percentage of all connections) did not explain the behavioral outcomes of supranormal visual motion detection and visual localization ability in CDCs. The few ectopic projections observed are unlikely to be a consequence of new axonal sprouting targeting new cortical areas, but rather may represent developmental exuberant projections [48] that are normally pruned by experience but were preserved in congenital deafness [44,49]. These exuberant connections are often formed between anatomically-bordering areas and may provide these with a higher susceptibility to crossmodal change than distant areas. Only in the anterior auditory field AAF of early deafened cats the data are not completely consistent with this concept – here more anatomical reorganization was observed [47]. However, area AAF is special because of the large number of somatosensory areas and fibers closely neighboring it, which are potential candidates for tracer pickup and require further study.

However, this does not mean that there is no effect of congenital auditory deprivation on the auditory cortex (reviewed in [43]). Two consequences of deafness in CDCs are relevant in the present context: (i) a specific reduction of corticocortical interactions [50], with (ii) reduction in functional intrinsic cortical connectivity between supragranular and infragranular layers [51]. These deficits could be related to a specific reduction of top-down interactions between secondary and primary auditory cortices [52]. Infragranular layers are the source of top-down interactions. Indeed, dystrophic effects (reduced thickness) were observed in infragranular and granular but not supragranular layers in areas A1, DZ and All [53]. Reduced activity in infragranular layers of area A1 has also been observed [54]. This all suggests that top-down interactions (that are known to develop after bottom-up interactions [55]) are more sensitive to developmental alteration of hearing than bottom-up interactions. A change in balance between bottom-up and top-down interactions may play an important role in crossmodal plasticity [53,56]. Reduced functional connectivity (measured by spike-field coherence) between superficial and deep cortical layers appears a key element responsible for integration of cortical column and contribute to the reduced top-down interactions between primary and secondary cortical areas in deafness [51]. Taken together, the cortical column gets reorganized in congenital deafness, potentially to change the balance between thalamocortical and corticocortical inputs towards corticocortical inputs. It is important to emphasize that both the thalamic input and bottom-up interactions were not eliminated in CDCs ([52]; posterior auditory field: [50]), explaining why auditory responsiveness is preserved when stimulated with a cochlear implant (ibid.).

The timeline of feline auditory cortical synaptogenesis covers the first 1-2 months of life, followed by synaptic pruning [57]. In CDCs this process was delayed by 1 month and the pruning was significantly increased [57]. It is likely that such pruning predominantly eliminates existing inactive cortical synapses, which may involve the majority of auditory corticocortical synapses in deafness. The more active ‘visual’ synapses, on the other hand, may survive and get stabilized in deafness. These visual synapses are likely those that come from other cortical areas and from multimodal thalamic regions like lateral posterior nucleus. These may convey multimodal interactions in the auditory cortex of hearing animals.

Taken together, these lines of evidence indicate that it is not the reorganized fiber tracts but predominantly the synapses, their number and synaptic efficacy, that convey crossmodal effects. Compared to adult-onset deafness, congenital deafness has additional effects in preventing multimodal integration and potentially preserving exuberant heteromodal connections (see below). Future studies of functional connectivity would provide clearer insights into the changes related to sensory deprivation and crossmodal reorganization (see Outstanding Questions).

Outstanding Questions.

• What are the exact molecular mechanisms of gain change in cross-modal plasticity, and are they the same for the different cortical areas involved? Can they be leveraged clinically? Identifying such molecular mechanisms would open a wide field of pharmacological modulation of cortical plasticity.

• Given its top-down nature, is cross-modal reorganization associated with up- or down-regulation of cognitive reserve in age-related hearing loss, and does it play any role in the link between hearing loss and cognitive decline?

• How can multimodal representations be established at a later age in congenital deprivation? Are there ways of extending critical periods by behavioral or new molecular approaches?

• What is the therapeutic potential of sensory substitution in hearing impaired (e.g. glasses translating speech into written text in real time) for auditory cortical representations? How effective are these approaches for clinical applications - do they support or prevent adaptations to the newly provided auditory input?

• How different is cross-modal plasticity between sensory systems? A different developmental sequence and different roles of the sensory systems for cognition suggest that differences in cross-modal plasticity across sensory systems may exist. Addressing this question is challenging, partly because reversibility of total deprivation is best clinically feasible in the auditory system; it is often an elusive goal in the other systems. New methodological approaches would be required for progress in this area.

• Should oral language learning for deaf children with cochlear implants be multimodal, exploiting all cross-modal adaptations, or should it be primarily auditory-focused to prevent excessive cross-modal reliance?

• Is cross-modal reorganization reversible even after very long periods of age-related hearing, given that typically older adults receive hearing aids after 10+ years of hearing loss onset?

Multimodal Integration and its Relation to Crossmodal Effects

The high specificity of the crossmodal effects described above suggests a tight relation of crossmodal plasticity to regions that receive heteromodal (and even multimodal) inputs in hearing (Fig. 2, [58–61]). Heteromodal influences in sensory cortices are modulating rather than driving [62,63], often involving phase effects on oscillations [61,64,65] and reflecting behavioral low-dimensional effects [66]. Multimodal integration requires higher associative areas with large functional interareal connectivity [67] that provide the top-down heteromodal inputs to sensory areas [68].

Fig. 2: Schematics of connections between sensory systems in hearing subjects, simplified.

Auditory system shown in red, non-auditory in blue, multisensory in grey. Connections are differentiated into driving (shown as straight lines), defined as connections able to elicit action potentials in absence of other active connections, and modulatory (shown as curved dashed lines), which affect the activity of neurons, but fail to cause postsynaptic action potentials in absence of other inputs (for details, see text). Connections between primary areas not shown since these differ between different sensory cortices. Multisensory information is observable in all cortical areas (filled background), but mainly as a modulatory influence. The main driving input comes from within the sensory system (the adequate input).

Heteromodal influences are more pronounced in higher-order auditory areas [64,69,70]. It has been in part related to attention and motor activity, including corollary influence of movement [66,71]. Their effect is often inhibitory in nature, possibly with the aim to suppress responses to self-produced or otherwise predicted sounds [71,72]. This all makes clear why a sensory system cannot be completely encapsulated or isolated in the brain: there is a need for the ability to modulate it depending on behavioral context. Overall, heteromodal influences in auditory cortex are not driving but modulatory in a “normal” brain, and they increase with increasing level of “cortical hierarchy”.

Visual influences in auditory cortex have been observed in subjects with normal hearing when visual and auditory inputs were coherent, as e.g. during lipreading [73–75]. Similar effects were further documented in hearing subjects using sign language [14]. Pre-existing visual influences have been found in areas of hearing animals that are known to undergo crossmodal plasticity (visual responses in cats: [76], visual responses in mice: [70], somatosensory responses in ferrets: [77]; visual responses in ferrets: [78]). The role of these visual inputs was to modulate an auditory response [62], including inhibitory effects [79].

These effects were sometimes reminiscent of the influence of attention on oscillatory responses [64]. The effect has been stronger in higher-order auditory areas than in primary areas [80]. Similar subthreshold influences have also been described in higher order somatosensory and multisensory regions [81]. Heteromodal top-down influences may be interpreted in the sense of prediction error hypothesis, where predictions penetrate through the network hierarchically from top to bottom [82,83] and modulate the information flow in the reverse direction.

Cortical multimodal structures are extensively connected with sensory areas by top-down modulations [67,84] and thus provide multimodal and hetermodal inputs to the auditory cortex (Fig. 2). Interestingly, in early deafness, higher-order auditory cortex additionally takes over functions related to cognitive processes [85–87]. In age-related hearing loss, cognitive processes compensate for degraded auditory input [88]. Reduced auditory acuity may reorganize the interaction between sensory and higher order cortices, such that cognitive and crossmodal processes are upregulated to compensate for effortful listening, depleting cognitive reserve [89] and possibly underlying the association between hearing loss and cognitive decline in adults [90].

The existing evidence thus suggests that the source of heteromodal activity in deafness is resting on pre-existing modulatory influences on neurons, partly caused by top-down inputs from associative (multisensory) areas (Fig. 3). Thus, the specificity of crossmodal plasticity with respect to cortical areas is likely due to fiber tracts that normally connect these areas to heteromodal and multimodal areas and provide heteromodal information also in hearing animals. Their role is stronger, accentuated and becomes driving only when these cortical areas are deprived of their dominant driving input.

Fig. 3: Schematics of the crossmodally reorganized hearing-impaired auditory system.

The main reorganization is the change of the modulatory heteromodal inputs into driving inputs, both from the secondary heteromodal areas as well as from multimodal areas. The effect is assumed to rest on resetting the working point of neurons in the auditory cortex. The absent auditory (adequate) input is shown by dashed lines.

Studies on hearing restoration following congenital deafness additionally document issues with multimodal binding, particularly fusion of visual with auditory inputs; processing of multimodal stimuli is instead dominated by the visual inputs if hearing restoration is later in life [91]. Correspondingly, in CDCs the neurons in the cross-modally reorganized area dorsal cortex did not show evidence of audiovisual fusion [49]. Corresponding outcomes have been recently reported in congenital blindness [92]. This confirms studies showing that early multimodal experience is essential for the development of multimodal processing capabilities [93–95]. The full potential of multisensory rehabilitation cannot be harvested if sensory restoration occurs too late in congenital deafness. In humans, the timeline for full effectiveness in restoration interventions is probably the first two years of life [91]. These considerations also explain why developmental deprivation has more severe consequences compared to late (adult-onset) deprivation.

Crossmodal Plasticity: A Dynamic, Flexible and Reversible Process?

There is growing evidence of crossmodal effects in mild and aging-related hearing loss. This is underscored by the high incidence of hearing loss in aging and the key importance of hearing loss in the pathophysiology of age-related cognitive decline [90]. In rats, crossmodal visual plasticity has been observed in secondary auditory areas following partial hearing loss in adulthood [96]. This suggests that even partial adult hearing loss can induce crossmodal plasticity. Indeed, in a series of studies in people with mild-moderate aging-related hearing loss, crossmodal effects have been described [89,97–102]. Recent evidence shows that crossmodal reorganization appears early, within 3 months of adult-onset hearing loss [89] and can be reversed with as little as 6 months of treatment with hearing aids [102]. Adding to its versatility, crossmodal plasticity reported in unilateral hearing loss can be reversed after cochlear implant use, although somatosensory cross modal effects appear to reverse more completely than visual effects [103], which is consistent with a continued reliance on the visual modality for disambiguating the auditory signal through a hearing aid or cochlear implant [91]. However, reversibility is more limited with congenital deafness compared with adult-onset of hearing loss [104], with an abnormal dominance of visual inputs and absence of multisensory fusion if congenital deafness is restored after the second year of life [91].

These studies demonstrate that crossmodal reorganization is a dynamic and rather fast process. It does not require structural changes and comes and vanes with mild sensory deprivation. The same neuronal circuitry can thus be used for crossmodal effects as well as for physiological processing of multimodal inputs.

We propose that the first neuronal step in fast crossmodal changes is a change in overall neuronal responsiveness (Fig. 4). This is finely regulated in cortical neurons, and in experimental models, changes in excitability of neurons can be induced through various paradigms, e.g. using acetylcholine, which can enhance intrinsic excitability [105] or manipulations that cause changes in excitatory-inhibitory balance [106,107]. In deafness, the main driving input to auditory neurons is eliminated, and in auditory regions, heteromodal inputs are typically too weak to drive neurons. That means that as a first step, the neurons are silenced, leading to an inevitable sequence of adaptation counteracting this effect. Homeostatic plasticity [107–110] adapts the working point of synapses and the spiking threshold of neurons, and synaptic changes affect excitatory-inhibitory balance to allow neurons to generate action potentials. In our view, reduced neuronal input will cause adaptation of neuronal responsiveness by these cellular and network mechanisms. In the absence of the adequate sensory input it causes increased sensitivity to the remaining inputs. This cannot fully compensate for the non-existing auditory inputs, but heteromodal inputs may transition from an originally modulatory to a driving role, and activate deprived cortical areas.

Fig. 4: Suggested mechanism of dynamic crossmodal plasticity.

In a hearing auditory cortex (A), the heteromodal inputs are only modulatory and are therefore dependent on the driving auditory input. Provided this is present, the responses can be significantly modified by heteromodal inputs. In hearing loss (B), there is reduced or no driving input and thus homeostatic plasticity may increase excitability to such an extent that the previously weak modulating input becomes driving. Thereby both the heteromodal response and neuronal sensitivity increase, i.e. the spiking thresholds to an input is decreasing. After hearing restoration (C), the gain is reduced due to the restoration of a strong driving input. The heteromodal input becomes modulating again. Since hearing restoration is rarely complete, the gain of cortical neurons is in between the one of deaf and hearing subjects. In the illustrated schematics, the changes are modeled only by an effect on spiking threshold and response increase. This is a simplification of the multiplicity of homeostatic processes present physiologically.

Upregulation of excitability is well documented in the cortex of deaf cats [57,111], reviewed in [42]) and of hearing-impaired rodents [96,112,113]. Provided that inputs to neurons include heteromodal sensory information, a previously modulatory effect may change to driving input. Such responses will be consistent with the auditory responses observed normally in these regions and can be well-processed by the same auditory circuitry. In milder hearing impairment, these changes will provide a-priori information (in the Bayesian sense) and by that will be naturally adaptive. In complete deafness, crossmodal effects will only affect functions that the auditory system has in common with other sensory systems and that are normally used to form multimodal representations.

In this concept of a dynamic adaptive crossmodal plasticity, homeostatic excitability adaptations will serve like a volume knob on a HiFi music audio system: when input is reduced, excitability will be pulled up to guarantee that all information that is available, including non-auditory information, will be optimally exploited for behavioral advantage and functional connectivity can be strengthened by synaptic plasticity. When input to the auditory cortex is appropriately restored via hearing aids and/or cochlear implants, then excitability is dynamically downregulated reversing the crossmodal changes to some extent.

A direct consequence of such adaptation in neuronal responsiveness is a change in functional connectivity. Appearance of heteromodal responses provides the window of opportunity for strengthening functional interactions to other regions [114,115] that are structurally connected but normally functionally coupled only in specific conditions (multimodal stimulation). Neuronal oscillations ubiquitously present due to properties of neuronal membranes and environmental or self-produced rhythms [116] allow an effective mechanism of long-distance coupling by synchronized activity. Increased responsiveness also facilitates it under unimodal stimulation in the spared modality. Phase synchronization of such oscillations has been demonstrated in between-modality coupling in hearing subjects [61,116]. Phase synchrony between primary and secondary auditory areas have been observed during top-down interactions in CDCs [51,52]. This mechanism is thus a plausible candidate for crossmodal reorganization in hearing loss: oscillatory synchronization, particularly in the form of phase reset, is under top-down control (review in [116]). Dynamic crossmodal plasticity based on increased gain in the impaired sensory input allows exploitation of top-down influences from multisensory and cognitive centers to synchronize increased heteromodal responses with activity from the spared modalities. This leads to the picture of a cross-modally reorganized functional connectome. Increased functional coupling can lead to new axon collaterals, increase the synaptic counts and synaptic efficacies for heteromodal inputs in the sensory-deprived areas and explain the easier retrograde tracer pickup in anatomical tracer studies, thus explaining more abundant (“stronger”) existing connections but no ectopic connections as observed in cats.

Functional connectivity can be studied using many methods (Box 1), including some that are applicable in humans. Functional connectivity analyses thus allow studying crossmodal effects in humans both in resting state as well as in response to a stimulus (see reviews, [114,117,118]).

BOX 1: Connectivity quantification.

The connectome, defined here as the totality of all synaptic connections between neurons in the brain, can be studied using many different methods [150]:

1. Structural connectivity defines the totality of fiber tracts connecting the brain. It can be analyzed using diffusor tensor imaging in humans, or anatomical tracers in animals. Fiber tracts are established early in development, driven mainly by genetic makeup. Fiber tracts are a precondition for functional interactions, but structural and functional connectivity correlate only weakly [151]. Functional connectivity additionally depends on synaptic counts, synaptic efficacies, and excitatory-inhibitory balance.

2. Functional connectivity (FC) defines statistical dependencies of neural activity. It can be subdivided into stimulus-related and stimulus-independent functional connectivity. Effective connectivity refers to the influence that one system exerts over the other (is directed).

   1. Stimulus-independent FC can be assessed by:

      1. Direct stimulation of one neural structure (electric or optogenetic) and recording in the other. In addition to fiber tracts (structural connectivity) it also depends on synaptic organization, their efficacy and receptiveness of the target structure [52,152].
      2. Ongoing activity reveals different brain networks activated at rest or attentive condition like default mode network [118]. This method provides general information on the brain networks and its hubs.
   2. Stimulus-dependent connectivity includes the effects of stimuli on the networks. The overall brain networks can reconfigure depending on stimulation (review in [118,153]). Stimulus-dependent connectivity must disentangle common input from direct interactions. It allows conclusions on how stimuli propagate in the brain.

Several measures allow quantifying functional connectivity:

1. Correlations [154] have the disadvantage to be dependent on the temporal function of the response, but historically allowed pioneering insights into brain networks.

2. Granger causality [155,156] quantifies effective ongoing and stimulus-dependent connectivity. It provides directionality of the connection, but is influenced by signal power and requires longer time windows for calculation.

3. Mutual information and transfer entropy [157,158] directly quantify information transfer. They are influenced by signal power and require longer time windows for calculation.

4. Phase-based measures like phase coherence, pairwise phase consistency or spike-field coherence [159,160] are power independent and when computed on induced activity also independent of common input. Their standout advantage is the short time window required for quantification, however, they do not provide directionality.

5. A combination of several methods. For example, separating the time-frequency bands in which phase-based methods detect connectivity at high temporal resolution allows subsequent extraction of directional information (in the corresponding bands) using effective connectivity measures (e.g. Granger causality) [52].

A number of publications revealed increased functional connectivity between visual, multisensory and auditory cortex following hearing loss [119–121], with contribution of multisensory parietal areas [122]. A decrease in intramodal connectivity and increase in crossmodal connectivity has been reported [120]. Crossmodal reorganizations were susceptible to compensation of hearing loss by hearing aids [123] and predictive or related to outcomes after hearing compensation [121,124,125]. These findings support the concept of dynamic flexible cross-modal plasticity that appears with hearing loss and partially subsides after hearing therapy.

In CDCs, reduced activity in the ongoing alpha-band has been found in the cross-modally reorganized posterior auditory area but not in area A1 [50] (for similar visual effects in congenitally blind humans, see [126]). Alpha oscillations have been additionally related to excitability changes of neurons [115]. At the same time, there was a loss of top-down interactions between secondary and primary auditory cortex in CDCs prominent in the alpha band [52]. This could indicate that loss of alpha power reflects loss of top-down influence on early areas rather than crossmodal reorganization per se. Similarly, alpha oscillations convey top-down interactions in humans [127], and their power has been related to speech intelligibility in hearing subjects [128,129]. Adult onset of hearing loss may provide also a different oscillatory signature than congenital hearing loss (e.g. theta oscillations [130] or delta oscillations [131]). Methodologically, oscillatory activity has the advantage of allowing to focus on different oscillatory bands, thus separating parallel neural processes [117], and further allowing separation of common input from corticocortical interaction by calculating induced oscillations (Fig. 5, [114,116,132]). This is essential for studying corticocortical interactions in the absence of common thalamic input, to a large extend reflected in the traditional evoked response components. Current neuroscience techniques provide the tools required for more detailed analysis of these effects in the future (Box 1).

Fig. 5: Functional connectivity related to an auditory stimulus determined from oscillatory activity.

(A) Primary auditory cortex response to an auditory stimulus (click train, 3 clicks, train duration 0.006 sec.) as observed in local field potentials recorded with a microelectrode. Top: 30 trials of the stimulus presented at 0 sec. results in a response that is reproducible (phase-locked) in each trial (within 0-0.1 sec. post stimulus) and a response that jitters in time from trial to trial (0.2-0.6 sec.). Bottom: Time-frequency representation (TFR) allows to compute the mean total power, revealing both responses that cover all relevant frequency ranges. (B) Computing an average LFP in time domain preserves mainly the phase-locked response (0-0.1 sec.). TFR reveals the evoked power. (C) Subtracting the average from each trial better isolates the non-phase-locked induced activity, well-preserved in the TFR where the response does not disappear during averaging due to averaging power not the amplitude. (D) To compute connectivity between two oscillators, the trial-to-trial stability in phase differences of oscillations quantifies the coupling strength (spring stiffness). This is reflected in θ_j for trial j. (E) When the phase differences in two different trials (denoted j, k) are plotted on the unit circle, the projection of one on another reveals their stability. This is achieved by calculating the sum all mutual dot products of the unitary vectors for each trial combination, normalized to the number of trial combinations. PPC = pairwise phase consistency. (F) Example of two sites recorded in primary and secondary auditory cortex of a hearing cat, demonstrating a PPC increase in the time between 0.2 and 0.5 sec. after presentation of the brief click train. Using this approach, the synchrony of the activity between cortical sites can be determined with specificity to frequency and with ms precision. Using induced activity for PPC computation additionally eliminates common input. Data from [52].

Crossmodal plasticity due to gain change and subsequent effects on functional connectivity, while adaptive, has limitations: in congenital deafness it cannot provide the substrate for multisensory integration and cannot guarantee normal development of the auditory system. Critical auditory periods are not due to colonization by the visual or somatosensory systems from crossmodal processes; they are rather the consequence of absence of adequate sensory input, consequent abnormal development and pronounced loss of cortical auditory synapses (reviews in [42,43]). Therefore, predominantly corticocortical interactions are affected by early deprivation [50]. Only some auditory areas overtake specific visual and somatosensory functions. Reduced thickness in deep cortical layers (a predominant source of top-down interactions) throughout all studied auditory cortical areas [53] suggests that despite crossmodal reorganization in congenital hearing loss, massive alterations in the cortical microcircuitry are observable [50,52]. Furthermore, dystrophic changes in deep layers were observed in all studied auditory areas, including those not involved in crossmodal reorganization. This means that the effect is due to auditory deprivation per se, and is not compensated by crossmodal reorganization. In vision loss, processing within auditory cortex is strengthened in the bottom-up direction, consistent with a shift in the sensory balance in favor of hearing [56,79]. This supports the present concept and represents another interesting aspect of future cross-modal research in hearing loss.

There are also limitations regarding the compensatory nature of this plasticity: Visual stimuli may help (in the Bayesian sense) disambiguate the auditory inputs. Visual stimulation alone, however, obviously cannot compensate the auditory experience nor can it negatively interfere after early restoration of hearing. Finally, the early developmental auditory effects of deafness are subject to critical periods, and this may compromise the ability to form multimodal representation by use of the deprived modality.

Clinical implications

A dynamic flexible crossmodal plasticity has several clinical implications. In adult onset of hearing loss, cross-modal plasticity can be adaptive and compensatory in many ways. When it’s not possible to restore deprived input, a ‘sensory substitution’ approach is used to transform information from the spared modality (within certain limits using its specific properties) to compensate for deficits in the deprived modality [133]. Such an approach has been reported in blind subjects using acoustic sonification of the visual information stream [134,135]. Similarly, eyeglasses which convert speech to subtitles for persons with hearing loss have recently became technically possible and represent a future commercial cross-modal application. In hearing loss, lipreading which activates auditory cortex is used routinely for rehabilitation [104,136–141]. Somatosensory enhancement of speech perception via cochlear implants was reported to aid speech understanding in deaf patients [142,143]. Thus, crossmodal stimulation can be harvested for clinical applications using the spared modalities [144].

In prelingual (congenital) deafness, including visual cues in communication is adaptive and may be leveraged clinically, particularly in the time before intervention. The disputed maladaptive consequences of visual cross-modal plasticity are in our view unlikely to be an issue if restoration of hearing is provided as soon as feasible (within the critical period) to assure a functional auditory system and multisensory interactions that include the auditory modality.

Consistent Observations in Other Sensory Systems

The notions discussed here are further supported by observations from other sensory systems. In blindness, rewiring does not appear to be necessary for the crossmodal effects being reported [145], however, response properties and implicitly functional interactions in occipitotemporal network are affected by blindness [146–148]. Similarly, somatomotor reorganization following amputations is rather limited and corresponds to the present considerations [149]. Taken together, it appears as if the textbook examples of brain plasticity in function loss need a revision: the crossmodal adaptations do not rest on the ability of the brain to take over (any) functions, but rather use preexisting circuitry that is modified depending on the overall input to the given areas, governed by E-I balance and homeostatic plasticity. In such perspective, crossmodal plasticity is exploitable for therapeutic approaches and allows to use these as an objective monitoring of efficacy of neurosensory restoration and subsequent rehabilitation.

Concluding Remarks

The brain can make use of the same neuronal circuitry for qualitatively different functions by dynamically adapting synaptic gain (synaptic rescaling) and inhibition, causing stimulus- and condition-dependent functional connectivity reorganization in absence of structural changes. This is particularly advantageous in case of reduced sensory input. The evidence discussed in this article is consistent with the notion that the substrate of crossmodal reorganization is essentially synaptic and functional, is operating at the microscopic scale and leads to functional connectivity changes in absence of large-scale rewiring of the brain. Auditory responsiveness is generally preserved in absence of hearing experience, both in primary and higher-order auditory cortical areas. Crossmodal reorganization is likely supported by (i) few exuberant connections formed during development and abnormally persisting into adulthood in congenital hearing loss and (ii) local increases in axonal collaterals and synaptic efficacy of heteromodal inputs, facilitated by (iii) increased sensitivity of target neurons deprived of their natural (adequate) auditory input, attentional modulation and (iv) their influences on neuronal oscillations and their interareal coupling. Heteromodal inputs originate predominantly from top-down interactions that can similarly affect neuronal oscillations. This explains why even adult-onset mild-to-moderate hearing loss can facilitate crossmodal reorganization that is reduced or reversed after hearing restoration. Understanding the neuronal mechanisms of how functional connectivity and thus flow of information can be reversed from bottom-up to top-down will provide further insight into constraints of crossmodal plasticity.

Highlights.

Crossmodal plasticity is a textbook example of the ability of the brain to reorganize based on use. In the auditory system, crossmodal plasticity is evident in all degrees of auditory deprivation and may be reversed in some cases of adult onset hearing loss.

Neither developmental nor adult crossmodal plasticity appear to involve extensive reorganization of structural connectivity. Instead, they typically involve strengthening and weakening of existing connections, often top-down heteromodal projections.

The auditory system is a model system for neurosensory restoration with neuroprostheses, and the dynamic and versatile nature of crossmodal plasticity can be exploited clinically for improving outcomes of neurosensory restoration.

Glossary:

Crossmodal reorganization / plasticity

A change in properties of neurons leading to stronger responsiveness to stimuli of the non-deprived modality. In the present text, the term “crossmodal” is reserved for conditions with hearing loss. This could refer to structural reorganization which involves formation of new fiber tracts, or, a functional reorganization referring primarily to changes from latent unmasking of existing pathways.

Multimodal reorganization / plasticity

A change in properties of neurons due to multimodal stimulation in subjects without sensory deprivation.

Multimodal areas

Cortical areas responsive to stimulation of more than one modality; the neurons in these areas can even be non-responsive to stimulation of a single modality. Multimodal responses are classified into subadditive, additive and superadditive depending on the amount of change caused by adding another modality.

Heteromodal inputs

Inputs into a given sensory area originating from another modality; i.e. visual or somatosensory inputs into the primary or secondary auditory cortex.