Experience-induced Malleability in Neural Encoding of Pitch, Timbre, and Timing

Abstract

Speech and music are highly complex signals that have many shared acoustic features. Pitch, Timbre, and Timing can be used as overarching perceptual categories for describing these shared properties. The acoustic cues contributing to these percepts also have distinct subcortical representations which can be selectively enhanced or degraded in different populations. Musically trained subjects are found to have enhanced subcortical representations of pitch, timbre, and timing. The effects of musical experience on subcortical auditory processing are pervasive and extend beyond music to the domains of language and emotion. The sensory malleability of the neural encoding of pitch, timbre, and timing can be affected by lifelong experience and short-term training. This conceptual framework and supporting data can be applied to consider sensory learning of speech and music through a hearing aid or cochlear implant.

Introduction

From the cochlea to the auditory cortex, sound is encoded at multiple locations along the ascending auditory pathway, eventually leading to conscious perception. While there is no doubt that the cortex plays a major role in the perception of speech, music, and other meaningful auditory signals, recent studies suggest that subcortical encoding of sound is not merely a series of passive, bottom-up processes successively transforming the acoustic signal into a more complex neural code. Rather, subcortical sensory processes dynamically interact with cortical processes, such as memory, attention, and multisensory integration, to shape the perceptual system’s response to speech and music.

In the last two decades there has been a surge in research devoted to how musical experience affects brain structure, cortical activity, and auditory perception. These three lines of research have uncovered several interesting byproducts of musical training. Musicians have brain structural differences not only in the motor cortices—the parts of the brain controlling hand/finger movement and coordination—but also in the auditory cortices.^1,2 In addition to structural differences, musicians show different patterns of neural activation. For example, musicians show stronger responses to simple, artificial tones and heightened responses to the sound of their own instrument compared to other instruments.^3–7 Interestingly, such cortical differences can be seen as early as 1 year after the onset of musical training⁸ and extend to speech signals.^9,10 Recently, this line of research has moved to subcortical levels. This work, along with supporting data, will be presented here within the pitch, timbre, and timing conceptual framework. In the final section of this review, we will switch the focus to cochlear implants and apply this conceptual framework to consider sensory learning of speech and music through an implant.

Conceptual Framework for Studying Subcortical Responses: Pitch, Timbre, and Timing

Work from our laboratory^d points to pitch, timbre, and timing as having distinct subcortical representations which can be selectively enhanced or degraded in different populations.

Pitch, as defined by the Standard Acoustical Terminology of the Acoustical Society of America, is “that attribute of auditory sensation in terms of which sounds may be ordered on a scale extending from low to high” S12.01, P.34.¹¹ For pure tones, the frequency, or cycles per second of the waveform, is the physical correlate of pitch; however when considering more complex sounds, pitch corresponds, in part, to the lowest resonant frequency, also known as the fundamental frequency (F₀).^e For speech, F₀ is dictated by the rate of vocal fold vibration and for music it depends on the instrument. For example, the reed is the source of F₀ vibration for the oboe and clarinet, whereas the string is the source for the violin and guitar. For the purposes of this review, we use the word pitch as shorthand for referring to the information carried by the F₀, and so in this context, pitch and F₀ are synonymous.

Timbre, also referred to as “sound color,” enables us to differentiate two sounds with the same pitch. Timbre is a multidimensional property resulting from the interaction of spectral and temporal changes associated with the harmonics of the fundamental along with the timing cues of the attack (onset) and decay (offset). Together this gives rise to the characteristic sound quality associated with a given instrument or voice. Timbre is also an important cue for distinguishing contrastive speech sounds (i.e., phonemes). As the vocal tract is shaped by the movement of the articulators during speech production, the resonance structure of the vocal tract changes and certain harmonics are attenuated while others are amplified. These amplified harmonics are known as speech-formants and they are important for distinguishing phonemes. Our focus here is on the harmonic aspects of timbre and the corresponding subcortical representation.

Timing refers to the major acoustic landmarks in the temporal envelope of speech and music signals. For speech, timing arises from the alternating opening and closing of the articulators and from the interplay between laryngeal and supralaryngeal gestures. Timing also includes spectrotemporal features of speech, such as time-varying formants. As such, timing arises from the interplay between the actions of the source (glottal pulse train) and filter (articulators). For music, timing can be considered in conjunction with the temporal information contributing to timbre perception. Likewise, on a more global scale, it refers to the duration of sounds and their subsequent perceptual groupings into rhythm. For the purposes of this review, we will focus on the neural representation of transient temporal features, such as onsets and offsets occurring as fast as fractions of milliseconds.

The Auditory brain stem Response

The auditory brain stem, an ensemble of nuclei belonging to the efferent and afferent auditory systems, receives and processes the output of the cochlea en route to higher centers of auditory processing. The auditory brain stem response (ABR), a highly replicable far-field potential recorded from surface electrodes placed on the scalp, reflects the acoustic properties of the sound stimulus with remarkable fidelity. In fact, when the electrical response is converted into an audio signal, the audio signal maintains a striking similarity to the eliciting stimulus.¹² Because of the transparency of this subcortical response, it is possible to compare the response timing and frequency composition to the corresponding features of the stimulus (Fig. 1). Timing features (including sound onsets, offsets, and format transitions) are represented in the brain stem response as large transient peaks, whereas pitch (F₀) and timbre (harmonics up to about 1000 Hz) information is represented as interspike intervals that match the periodicity of the signal, a phenomenon known as phase locking.^f By means of commonly employed digital signal processing tools, such as autocorrelation^g and Fourier analysis,^h features relating to stimulus pitch and timbre can be extracted from the response. As a consequence of being such a highly replicable measure, incredibly subtle differences in the timing and phase locking of the ABR are indicative of sensory processing malleability and abnormality.

Figure 1.

Schematic representation of timing, pitch, and timbre in the stimulus (black) and brain stem response (gray) waveforms. Top: The full view of the time-domain stimulus waveform “da.” The temporal features of the stimulus, including the sound offset and onset, are preserved in the response. The gray box demarcates six cycles of the fundamental frequency (F₀); a blowup of this section is plotted in the middle panel. Middle: Major waveform peaks occur at an interval of 10 ms (i.e., the periodicity of a 100-Hz signal). This stimulus periodicity, which elicits the perception of pitch, is faithfully represented in the response. Bottom: The left panel shows a closeup of an F₀ cycle. The harmonics of the stimulus are represented as small-amplitude fluctuations between the major F₀ peaks in the stimulus and response. In the right panel, the stimulus and response are plotted in the spectral domain. Frequencies important for the perception of pitch (100 Hz) and timbre (frequencies at multiples of 100 Hz) are maintained in the brain stem response.

Subcortical Representation of Pitch

Musicians have extensive experience manipulating pitch within the context of music. Work by the Kraus Laboratory^9,13,14 shows that lifelong musical training is associated with heightened subcortical representations of both musical and linguistic pitch, suggesting transfer effects from music to speech processing.

Musacchia et al.¹⁴ employed an audiovisual (AV) paradigm to tap into the multisensory nature of music. Given that music performance involves the integration of auditory, visual, and tactile information, we hypothesized that lifelong musical practice would influence AV integration. Subcortical responses were compared in three conditions: AV, auditory alone (A), and visual alone (V). In the AV condition, subjects watched and listened to a movie of a person playing the cello or saying “da.” In the A condition, no movie was displayed, and in the V condition, no sounds were presented. For both musicians and nonmusicians, the pitch responses to both speech and music were larger in the multimodal condition (AV) compared to unimodal A condition. However, musicians showed comparatively larger pitch response in both A and AV conditions (AV responses are plotted in Fig. 2), and more pronounced multimodal effects, that is, greater amplitude increase between A and AV conditions. In addition, pitch representation strongly correlated with years of musical practice, such that the longer a person had been playing, the larger the pitch response (Fig. 3, top). When the cortical responses to the AV condition were examined, this pitch representation was positively correlated with the steepness of the P1–N1 slope, such that the sharper (i.e., more synchronous) the cortical response, the larger the pitch representation.⁹ Other aspects of these multisensory responses will be explored in the sections relating to subcortical representation of timbre and timing. Taken together these data indicate that multisensory training, such as is acquired with musical experience, has pervasive affects on subcortical and cortical sensory encoding mechanisms for both musical and speech stimuli and leads to training-induced malleability of sensory processing.

Figure 2.

Grand average brain stem responses to the speech syllable “da” for both musician (red) and non-musician (black) groups in the audiovisual condition. Top: Amplitude differences between the groups are evident over the entire response waveform. These differences translate into enhanced pitch and timbre representation (see bottom panel). Auditory and visual components of the speech stimulus (man saying “da”) are plotted on top. Middle: Musicians exhibit faster (i.e., earlier) onset responses. The grand average brain stem responses in the top panel have been magnified here to highlight the onset response. The large response negativity (shaded region) occurs on average ~0.50 ms earlier for musicians compared to nonmusicians. Bottom. Fourier analysis shows musicians to have more robust amplitudes of the F₀ peak (100 Hz) and the peaks corresponding to the harmonics (200, 300, 400, 500 Hz) (left). To illustrate frequency tracking of pitch and harmonics over time, narrow-band spectrograms (right) were calculated to produce time–frequency plots (1-ms resolution) for the musician (right top) and non-musician groups (right bottom). Spectral amplitudes are plotted along a color continuum, with warmer colors corresponding to larger amplitudes and cooler colors representing smaller amplitudes. Musicians have more pronounced harmonic tracking over time. This is reflected in repeating parallel bands of color occurring at 100 Hz intervals. In contrast, the spectrogram for the nonmusician group is more diffuse, and the harmonics appear more faded (i.e., weaker) relative to the musician group. (Adapted from Musacchia et al.⁹,¹⁴) (In color in Annals online.)

Figure 3.

Neural enhancement varies according to the extent (top) and onset (bottom) of musical practice. Top: The number of years (over the last 10 years) of consistent practice is correlated with the strength of subcortical pitch encoding. Thus, the longer an individual has been practicing music, the larger the F₀ amplitude. (Adapted from Musacchia et al.¹⁴) Bottom: The precision of brain stem pitch tracking is associated with the age that musical training began. Subjects who started earlier show a higher degree of pitch tracking. [N.B.: “Perfect” pitch tracking (i.e., no deviation between the stimulus pitch trajectory and response pitch trajectory) would be plotted as a 1 along the y-axis.] (Adapted from Wong et al.¹³)

In music and language, pitch changes convey melodic and semantic or pragmatic information. Recently, a number of studies have looked at the representation of linguistic pitch contours (i.e., sounds which change in pitch over time) in the brain stem response. In Mandarin Chinese, unlike English, pitch changes signal lexical semantic changes. Compared to native English speakers, Mandarin Chinese speakers have stronger and more precise brain stem phase locking to Mandarin pitch contours, suggesting that the subcortical representation of pitch can be influenced by linguistic experience.^15,16 Using a similar paradigm, we explored the idea that musical pitch experience can lead to enhanced linguistic pitch tracking.¹³ ABRs were recorded to three Mandarin tone contours: tone 1 (level contour), tone 2 (rising contour), and tone 3 (dipping contour). Musically trained native English speakers, with no knowledge of Mandarin or other tone languages, were found to have more accurate tracking of tone 3 (Fig. 4), a complex contour not occurring at the lexical (word) level in English.¹⁷ In addition, we found that the accuracy of pitch tracking was correlated with two factors: years of musical training and the age that musical training began (Fig. 3, bottom). The differences between musicians and nonmusicians were less pronounced for tone 2 and not evident for tone 1. In contrast to tone 3, which only occurs at the phrase level in English, tones 1 and 2 are found at the word and syllable level. Taken together with the finding that musicians exhibit distinctive responses to emotionally salient pitch cues¹⁸ and enhanced pitch elements in musical chords²³ (reviewed below), we concluded that musical training alters subcortical sensory encoding of dynamic pitch contours, especially for complex and novel stimuli.

Figure 4.

Pitch tracking plots from a musician (left) and nonmusician (right). The thin black line represents the pitch contour of the stimulus (Mandarin tone 3), and the thick gray line represents the extracted pitch trajectory of the brain stem response. The musician’s brain response follows the pitch of the stimulus more precisely, a phenomenon known as pitch tracking. (Adapted from Wong et al.¹³)

The studies reviewed above investigated the effects of lifelong auditory (linguistic and musical) experience on the subcortical representation of pitch. Recent work from Song et al.¹⁹ suggests that lifelong experience may not be necessary for engendering changes in the subcortical representation of pitch. In fact, we found that as few as eight training sessions (30 mins each) can produce more accurate and more robust subcortical pitch tracking in native-English-speaking adults. Interestingly, improvement occurred only for the most complex and least familiar pitch contour (tone 3).

Unlike musicians who have heightened pitch perception,^20,21 some individuals with autism spectrum disorders (ASD) are known to have issues with pitch perception in the context of language. For example, these individuals often cannot take advantage of the prosodic aspects of language and have difficulty distinguishing a question (rising pitch) from a statement (level or falling pitch). Russo et al.²² explored whether this prosodic deficit was related to subcortical representation of pitch. We found that a subset of autistic children showed poor pitch tracking to syllables with linearly rising and falling pitch contours. Given that the subcortical representation of pitch can be enhanced with short-term linguistic pitch training and lifelong musical experience, this suggests that some children with ASD might benefit from an auditory training paradigm that integrates musical and linguistic training as a means of improving brain stem pitch tracking.

Subcortical Representation of Timbre

A growing body of research is showing that musicians represent the harmonics of the stimulus more robustly than their nonmusician counterparts.^9,18,23 This is evident for a whole host of stimuli including speech and emotionally affective sounds as well as musical sounds. Lee et al.²³ recorded brain stem responses to harmonically rich musical intervals and found that musicians had heightened responses to the harmonics, as well as the combination tonesⁱ produced by the interaction of the two notes of the interval. In music, the melody is typically carried by the upper voice and the ability to parse out the melody from other voices is a fundamental musical skill. Consistent with previous behavioral and cortical studies,^24–27 we found that musicians demonstrated larger subcortical responses to the harmonics of the upper note relative to the lower note. In addition, an acoustic correlate of consonance perception (i.e., temporal envelope) was more precisely represented in the musician group. When two tones are played simultaneously, the two notes interact to create periodic amplitude modulations. These modulations generate the perception of “beats,” “smoothness,” and “roughness,” and contribute to the sensory consonance of the interval. Thus by actively attending to the upper note of a melody and the harmonic relation of concurrent tones, musicians may develop specialized sensory systems for processing behaviorally relevant aspects of musical signals. These specializations likely occur throughout the course of musical training—a viewpoint supported by a correlation between the length of musical training (years) and the extent of subcortical enhancements.

The link between behavior and subcortical enhancements is also directly supported by Musacchia et al.,⁹ who found that better performance on a timbre discrimination task was associated with larger subcortical representations of timbre. Timbre was also an important distinguishing factor for separating out musicians from nonmusicians. As a group, the musically trained subjects had heightened representation of the harmonics (Fig. 2, bottom). Furthermore, when the subjects were analyzed along a continuum according to the age musical training began, subjects who started at a younger age were found to have larger timbre representations compared to those who began later in life. In addition, a correlation was found between cortical response timing and subcortical timbre encoding, which may be indicative of cortical structures being active in the processing of more subtle stimulus features.

Subcortical Representation of Timing

Timing measures provide insight into the accuracy with which the brain stem nuclei synchronously respond to acoustic stimuli. The hallmark of normal perception is an accurate representation of the temporal features of sound. In fact, disruptions on the order of fractions of milliseconds are clinically significant for the diagnosis of hearing loss, brain stem pathology, and certain learning disorders. Compared to normally hearing nonmusicians, musicians have more precise subcortical representation of timing, resulting in earlier (i.e., faster) and larger onset peaks^14,18 (Fig. 2, middle). Furthermore, the results of these studies suggest an intricate relationship between years of musical practice and neural representation of timing. Taken together, the outcomes of our correlational analyses show that subcortical sensory malleability is dynamic and continues beyond the first few years of musical training.

Summary: Music Experience and Neural Plasticity

Transfer Effects

By binding together multimodal information and actively engaging cognitive and attentional mechanisms, music is an effective vehicle for auditory training.^29,30 By showing that the effects of musical experience on the nervous system’s response to sound are pervasive and extend beyond music,^{9,13,14,18,31} work from our laboratory fits within the larger scientific body of evidence. We find transfer effects between the musical domain and the speech domain resulting in enhanced subcortical representation of linguistic stimuli.^9,13,14 However, these enhancements are not only specific to musical and linguistic stimuli, but also occur with non-linguistic emotionally rich stimuli as well. Strait et al.¹⁸ (also appearing in this volume^31a) recorded ABRs to the sound of a baby’s cry, an emotionally laden sound. Compared to the nonmusician cohort, musicians showed enhanced pitch and timbre amplitudes to the most spectrally complex section of the sound, and attenuated responses to the more periodic, less complex section. These results provide the first biological evidence for enhanced perception of emotion in musicians^32,33 and indicate the involvement of subcortical mechanisms in processing of vocally expressed emotion. Another compelling finding is that extensive auditory training can lead to both enhancement and efficiency (i.e., smaller amplitudes are indicative of allocation of fewer neural resources) of subcortical processing, with both enhancement and economy being evident in the subcortical response to a single acoustic stimulus. This finding reinforces the idea that subcortical responses to behaviorally relevant signals are not hardwired, but are malleable with auditory training.

The multisensory nature of music may also have an impact on vocal production by engaging auditory/vocal-motor mechanisms. Stegemöller and colleagues³¹ recorded speech and song samples from musicians and non-musicians. Vocal productions were analyzed using a statistical analysis of frequency ratios.³⁴ The vocal productions (speech and music) of both groups showed energy concentrations at ratios corresponding to the 12-tone musical scale. However, musicians’ samples were smoother and had fewer deviant (i.e., non 12-tone ratio) peaks (Fig. 5), showing that musicians had less harmonic jitter in their voices. This pattern was apparent even in the speech condition, where nonmusicians were found to differ from the vocally trained subjects in the musician group. This suggests that musical vocal training has an impact on vocal tract resonance during speech production. Also notable is that the musicians who did not undergo vocal training (instrumentalists) had smoother spectra for the song samples. Therefore, exposure to the 12-tone scale through instrumental training can be seen to influence vocal production, indicating a transfer from the auditory to the motor modalities.

Figure 5.

Normalized spectra of speech (top two traces) and song (bottom two traces) tokens for non-musicians and vocalists. Prominent peaks in the spectra correspond to the intervals of the 12-tone scale. Unison, Perfect 4th, Perfect 5th, Major 6th, and Octave are labeled and represent the most well-defined spectral peaks in the speech and song tokens. Compared to nonmusicians, vocalists and professional musicians (not plotted) have smoother normalized spectra which include fewer unexpected (non–12-tone interval) peaks. The encircled portion of (A) is magnified in (B) to show the decrease in the number of unexpected peaks from speech to song, and from no musical experience to trained vocal experience. (Adapted from Stegemöller et al.³¹)

Subcortical Enhancements and the Interaction of Top-down Processes

At first blush, it would appear that musical training is akin to a volume knob, leading to musicians’ processing sounds as if they were presented at a louder decibel level. While it is clear that musicians show subcortical enhancements for pitch, timbre, and timing, a simple stimulus-independent gain effect cannot explain all of the results reviewed above. A better analogy is that musical training helps to focus auditory processing, much in the same way that glasses help to focus vision, and that this leads to clearer and more fine-grained subcortical representations. If only a gain effect was operative, we might expect all stimuli and all stimulus features to show more or less equivalent enhancements. However, available data do not support this stimulus-independent view. What we find instead is that only certain stimuli¹³ or certain aspects of the stimuli are enhanced in musicians.^14,18,23 So while musical training might help focus auditory processing at a subcortical level, it does not do so blindly. Instead the behavioral relevance and complexity of the stimulus likely influences how the sensory system responds. This suggests that higher-level cognitive factors are at play. In order to obtain auditory acuity, musicians actively-engage top-down mechanisms, such as attention, memory, and context, and it is this binding of sensory acuity and cognitive demands that may in fact drive the subcortical enhancements we observe in musicians. Our findings suggest that higher-order processing levels (i.e., cortical) have efficient feedback pathways to lower-order (i.e., brain stem) processing levels. This top-down feedback is likely mediated by the corticofugal pathway, a vast track of efferent fibers that link together the cortex and lower structures.^35–38 While the corticofugal system has been extensively studied in animal models, the direct involvement of this efferent system in human auditory processing has also been demonstrated by Perrot and colleagues.³⁹ In the animal model, the corticofugal system works to fine-tune subcortical auditory processing of behaviorally relevant sounds by linking learned representations and the neural encoding of the physical acoustic features. This can lead to short-term plasticity and eventually long-term reorganization of subcortical sound encoding (for a review see Suga et al.³⁵). Importantly, corticofugal modulation of specific auditory information is evident in the earliest stages of auditory processing.⁶ It is therefore our view that corticofugal mechanisms apply to human sensory processing, and can account, at least in part, for the pattern of results observed in musicians. Consistent with this corticofugal hypothesis and observations of experience-dependent sharpening of primary auditory cortex receptive fields,^7,40 we maintain that subcortical enhancements do not result simply from passive, repeated exposure to musical signals or pure genetic determinants. Instead, the refinement of auditory sensory encoding is driven by a combination of these factors and behaviorally relevant experiences, such as lifelong music making. This idea is reinforced by correlational analyses showing that subcortical enhancements vary as a function of musical experience^{9,13,14,18,23} (Fig. 3).

When Auditory Processing Goes Awry

Impaired auditory processing is the hallmark of several clinical conditions, such as auditory-processing disorder (APD), a condition characterized by difficulty perceiving speech in noisy environments. Work from our laboratory has shown that a significant subset of children with language-based learning problems, such as dyslexia, where APD is common, show irregular subcortical representations of timing and timbre (harmonics), but not pitch.^28,41 This pattern is consistent with the phonological processing problems inherent in reading disorders. Our research into the subcortical representation of speech in the learning-impaired population has been translated into a clinical tool, BioMARK (Biological Marker of Auditory Processing; see Clinical Technologies at http://www.brainvolts.northwestern.edu/). This test provides a standardized metric of auditory encoding and can be used to disentangle roles of pitch, timbre, and timing in normal and disordered auditory processing.

For a significant number of children with reading disabilities, sound is atypically encoded at multiple levels of the auditory system—the auditory brain stem,^28,41–44 the auditory cortex^45–47 or both^48–50—suggesting a complex interaction between subcortical and cortical levels. Thus, the deficits we find in language impairment, such as developmental dyslexia^28,48 (Fig. 6) and ASD,²² might be the consequence of faulty or suboptimal corticofugal engagement of auditory activity.

Figure 6.

Brain stem responses from a child with reading difficulties (top), a young adult with typical hearing (middle) and a professional musician (bottom). Note the differences in waveform morphology, with the musician having larger and more defined (sharper) peaks.

Further evidence for the dynamic nature of subcortical auditory processing can be found by studying the effects of short-term training in children. After undergoing an 8-week commercially available auditory training program, children with language-based learning impairments showed improved subcortical response timing for speech signals presented in background noise.⁵¹ Because the auditory training was not specific to speech perception in noise, it raises the possibility that training-induced brain stem plasticity was mediated by top-down, cortically driven processes, a conclusion also supported by work from de Boer and Thornton.⁵²

Cochlear Implants and Music Perception

Cochlear implants (CIs) have proven to be enormously successful in engendering speech perception, especially in quiet settings, yet music perception is still below par. This is perhaps not surprising given that CI processing strategies are primarily designed to promote speech perception and thereby provide only a rough estimation of spectral shape, despite comparably fine-grained temporal resolution. While both speech and music have spectral and temporal elements, the weighting of these elements is not the same: speech perception requires more temporal precision whereas music perception requires more spectral precision.⁵³ The CI user’s poor performance on musical tasks can be explained in large part by this underlying CI processing scheme and the acoustic differences between speech and music.

Real-world music listening requires the integration of multiple cues including pitch, timing (e.g., tempo and rhythm), and timbre (e.g., instrument identification). For research purposes, music can be analytically decomposed into perceptual tasks that tap into each individual element. The pitch, timbre, and timing model that we employ in our laboratory for studying brain stem responses is also a useful trichotomy for assessing CI performance on musical tasks. With respect to timing tasks, the general consensus in the CI literature is that CI users and normal-hearing listeners have nearly comparable performances, yet the CI users perform far below average on timbre and pitch tasks.^54–60 On timbre tasks, CI wearers often have a difficult time telling two instruments apart.^{54–56,58,60} However, despite this well-documented performance, Koelsch and collegues⁶¹ have demonstrated that timbral differences can elicit subliminal cortical responses. This suggests that even though many CI users cannot formally acknowledge differences in sound quality, these differences may in fact be registered in the brain.

When it comes to pitch perception, CI users could be described as having an extreme form of amusia (tone deafness). For example, whereas normally hearing adults can easily tell the difference between two adjacent keys on a piano (i.e., 1 semitone difference), for the average postlingually implanted CI wearer, the notes must be at least 7 keys apart.⁵⁴ However, even if implantation occurs later in life, recent work by Guiraud and colleagues,⁶² indicates that CIs can help reverse the effects of sensory deprivation by reorganizing how spectral information is mapped in the cortex.

For CI users, rehabilitative therapy has traditionally focused on improving speech perception and production. Despite numerous anecdotal and case reports showing that music therapy is being integrated into the rehabilitative process, the effects of musical training after CI implantation have garnered little scientific attention. Nevertheless, two known published reports reinforce the idea that focused short-term training can improve timbre and pitch perception.^54,63

While vocoded sounds—sounds that have been manipulated to simulate the input that CI users receive—cannot fully mimic the CI acoustic experience, they serve as a useful surrogate for studying how the nervous system deals with degraded sensory input before and after training. Studies are currently under way in our laboratory to explore how the normal hearing system encodes pitch, timbre, and timing features of speech and musical stimuli, and their vocoded counterparts. Special attention will be paid to the relationship between musical experience and how vocoded and more natural conditions are differentially represented at subcortical and cortical levels.

Because of magnetic and electromagnetic interference from the CI transmitter, magnetoencephalography and magnetic resonance imaging cannot be performed while a person is wearing a CI. Although an electrical artifact can plague electrophysiological recordings from CI wearers, techniques have been developed to minimize these effects in cortical potentials.^64,65 ABRs to speech and music have the capacity to be a highly objective and revealing measure of auditory processing in normal subjects listening to vocoded sounds, and with technological advances speech- and music-evoked ABRs may eventually be recorded in CI users. This work would complement the existing literature that has documented the integrity and plasticity of the CI user’s subcortical auditory pathways using simple click stimuli.^66,67

Furthermore, in order to promote large scale and cross-laboratory/cross-clinic comparisons there is a need for standardized measurements of electrophysiology (equivalent to BioMARK) and music perception in this population (for three examples of music tests, see Nimmons et al.,⁶⁸ Cooper et al.,⁶⁹ and Spitzer et al.⁷⁰). The benchmark of an effective test is one that can track changes before and after training, and is also sensitive enough to keep up with advancing CI technologies.

Speech and music perception are without question constrained by the current state of CI technology. However, technology alone cannot explain the highly variable performance across implantees, including the exceptional cases of children and adults who demonstrate near-normal pitch perception and production.^71,72 These “super-listeners” serve as beacons for where commonplace CI performance can aspire in the near future.

While most CI wearers have limited musical experience before implantation,⁷³ a growing number of trained musicians are receiving implants. These individuals seem to have an advantage when it comes to music perception through a CI, especially for pitch perception. This underscores the important role that music experience plays in shaping sensory skills and lends further support for experience-dependent corticofugal (top-down) modulation of cortical and subcortical auditory pathway.^13,35,39,74 Through the use electrophysiology and standardized music tests, we will gain better insight into the biological processes underlying super-listeners and ordinary listeners, which will ultimately lead to more refined CI technology and improved music enjoyment among CI users.

Conclusion and Future Outlook

Subcortical auditory processes are dynamic and not hardwired. As discussed here, auditory sensory processing interacts with other modalities (e.g., visual and motor influences) and is influenced by language and music experience. The role of subcortical auditory processes in perception and cognition is far from understood, but available data suggest a rich interplay between the sensory and cognitive processes involved in language and music, and a common subcortical pathway for these functions. It appears that in the normal system, music and language experience fundamentally shape auditory processing that occurs early in the sensory processing stream.^{13–16,18,19,23} This top-down influence is likely mediated by the extensive corticofugal circuitry of descending efferent fibers that course from the cortex to the cochlea.⁷⁵ In order to facilitate sensory learning, the impaired system can capitalize on the shared biological resources underlying the neural processing of language and music, the impact music has on auditory processing and multisensory integration, and the apparent cognitive-sensory reciprocity.