Looking for Mickey Mouse™ But Finding a Munchkin: The Perceptual Effects of Frequency Upshifts for Single-Sided Deaf, Cochlear Implant Patients

Abstract

Purpose

Our aim was to make audible for normal-hearing listeners the Mickey Mouse™ sound quality of cochlear implants (CIs) often found following device activation.

Method

The listeners were 3 single-sided deaf patients fit with a CI and who had 6 months or less of CI experience. Computed tomography imaging established the location of each electrode contact in the cochlea and allowed an estimate of the place frequency of the tissue nearest each electrode. For the most apical electrodes, this estimate ranged from 650 to 780 Hz. To determine CI sound quality, a clean signal (a sentence) was presented to the CI ear via a direct connect cable and candidate, and CI-like signals were presented to the ear with normal hearing via an insert receiver. The listeners rated the similarity of the candidate signals to the sound of the CI on a 1- to 10-point scale, with 10 being a complete match.

Results

To make the match to CI sound quality, all 3 patients need an upshift in formant frequencies (300–800 Hz) and a metallic sound quality. Two of the 3 patients also needed an upshift in voice pitch (10–80 Hz) and a muffling of sound quality. Similarity scores ranged from 8 to 9.7.

Conclusion

The formant frequency upshifts, fundamental frequency upshifts, and metallic sound quality experienced by the listeners can be linked to the relatively basal locations of the electrode contacts and short duration experience with their devices. The perceptual consequence was not the voice quality of Mickey Mouse™ but rather that of Munchkins in The Wizard of Oz for whom both formant frequencies and voice pitch were upshifted.

Supplemental Material

https://doi.org/10.23641/asha.9341651

At fitting and sometimes for weeks or months after, a common description of the sound quality of a cochlear implant (CI) is “like Mickey Mouse™,” that is, sounds are higher in frequency than normal. Other descriptions are “like Donald Duck™” or “like the Chipmunks™.” For this report, we tested single-sided deaf listeners fit with CI (SSD-CI) within 6 months of device activation and asked them to create signals for their normal-hearing ear that matched the sound quality of their CI. Our aim was to make audible for normal-hearing listeners the Mickey Mouse™ sound quality of CIs often found following device activation.

The Value of SSD-CI Patients

SSD-CI patients allow researchers to objectively determine the sound quality of an implant. This is because SSD-CI patients are the first CI patients with a normal-hearing ear and can rate the similarity of a clean signal (a sentence in our experiments) presented to the CI ear via a direct connect cable and candidate, CI-like signals presented to the ear with normal hearing via an insert receiver (Dorman, Natale, Butts, Zeitler, & Carlson, 2017). In our experiments, a candidate signal is constructed for the normal-hearing ear and the listener is asked to rate the similarity of that signal to the clean signal in the CI ear. If the sound quality is close, then the experimenter continues to modify the signal using the same parameter set. If the match is not close, a different type of modification is implemented. The final match is, most often, a combination of several acoustic modifications of a clean signal.

Background

Our first experiment using this paradigm (Dorman et al., 2017) established that the sound quality of a CI is not that of a noise vocoder—which is the most common “simulation” of a CI. In that study, noise vocoders with four to 12 channels received median ratings between 1.5 and 1.9 on a 10-point scale where 0 indicated that the signal was not at all like the CI and 10 indicated that the signal created for the normal-hearing ear matched exactly the sound quality of the CI. Sine output vocoders fared a little better with median ratings between 2.7 and 3.0.

If speech heard by a CI listener is not similar to that of a vocoder, what does it sound like? Dorman et al. (2017) reported that muffling was a component of the CI percept for three young SSD-CI patients. In the next section, we discuss a factor, electrode insertion angle (or the lowest frequency stimulated in the spiral ganglion), which may play a role in Mickey Mouse™ voice quality.

Electrode Insertion Angle and Frequency Upshift

In a CI, energy from a given frequency region of a speech signal is delivered to a cochlear place that is higher in frequency than the input frequency (e.g., Landsberger, Svrakic, Roland, & Svirsky, 2015). Consider the data from a patient with a 16.5-mm electrode array and an electrode array angular depth of 419°. Table 1A shows the angular depth of each electrode in degrees, the frequency of the nearest tissue in the spiral ganglion for each electrode (Noble, Labadie, Gifford, & Dawant, 2013), the CI filter center frequency (cf), and the ratio of nearest tissue to filter cf.

Table 1.

Upward shifts in speech component frequencies for two patients with different electrode arrays.

A. Data from a patient with a 16.5-mm electrode array
Electrode	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Angular depth	419	370	328	298	273	247	226	204	176	164	145	122	95	69	47	29
Nearest tissue (Hz)	620	750	1040	1300	1490	1940	2300	2730	3240	3700	4310	5110	5950	8700	11350	13460
Filter cf (Hz)	333	455	540	642	762	906	1076	1278	1518	1803	2142	2544	3022	3590	4264	6665
Nearest tissue/filter cf	1.9	1.6	1.9	2.0	1.9	2.1	2.1	2.1	2.1	2.0	2.0	2.0	2.0	2.4	2.7	2.0
B. Data from a patient with a 28-mm electrode array
Electrode	1	2	3	4	5	6	7	8	9	10	11	12
Angular depth	525	434	382	337	281	244	210	161	119	81	42	18
Nearest tissue (Hz)	390	590	720	960	1430	1980	2630	3770	5210	7190	12010	14800
Filter cf (Hz)	120	235	384	579	836	1175	1624	2222	3014	4084	5507	7410
Nearest tissue/filter cf	3.3	2.5	1.9	1.7	1.7	1.7	1.6	1.7	1.7	1.8	2.2	2.0

Note. cf = center frequency.

At each electrode along the array, energy from the input signal (filter cf) is delivered to a spiral ganglion region that has a higher place frequency than the input signal. The upshift varies from approximately 300 Hz at the apical end of the array to 6000 Hz at the basal end. Overall, signal frequencies are delivered to place frequencies in the spiral ganglion that are approximately double the input frequency (mean across electrode positions = 2.03 × input frequency). In this example, the lowest place frequency stimulated is near 600 Hz.

Consider now, in Table 1B, similar data for a patient with a 28-mm electrode array and an angular depth of 525°. Once again, at each electrode along the array, energy from the input signal is delivered to a spiral ganglion region that has a higher place frequency than the input signal. The mean upshift is similar to that in the previous example (M = 1.97 × input frequency), although details vary. Critically, the lowest frequency region stimulated in the spiral ganglion is 390 Hz—lower than the 620-Hz value in the previous example.

Because our goal in this article was to make Mickey Mouse™ voice quality audible, we chose to work with patients with a relatively shorter electrode array and within a few months following fitting. As illustrated in the examples above, the most apical electrode in shorter arrays stimulates higher frequencies than the most apical electrode in longer arrays, and it is reasonable to suppose this high “starting frequency” and basally shifted signal frequencies could play a role in the Mickey Mouse™ voice.

The 1939 movie, The Wizard of Oz, exposed listeners to the perceptual effect of an upshift in speech signal frequencies. In that movie, Dorothy finds herself in the land of the Munchkins. The Munchkins were played by “little people” actors. However, the actors did not speak or sing their lines. The acoustic signals were created by typical-sized actors and “slowing the tape down during recording and playing it back at normal speed” (OzWiki, 2019). Although vague, the description is consistent with an upward frequency shift of not only voice pitch (fundamental frequency [F0]) but also formant frequencies (also a decrease in signal duration). The voice quality of the Munchkins can be heard in the following clip: https://www.youtube.com/watch?v=6KSiyaqnZYs.

Based on the upward shift of speech component frequencies for the CI listeners shown in Table 1 and the frequency upshifted signals used to create the Munchkin voices, it is not unreasonable to suppose that what we call Mickey Mouse™ voice might sound like the Munchkin voices in The Wizard of Oz. To find out, we tested three SSD-CI patients who were within 6 months of initial fitting and asked them to create signals for their normal-hearing ears that matched the sound quality of their CI. The hypothesis to be tested was that one component of the match would be an upshift in both formant frequencies and F0.

Method

Subjects

The listeners were three single-sided deaf women with normal-hearing thresholds in the ear contralateral to the CI. The listeners ranged in age from 37.8 to 48.5 years. Duration of deafness ranged from 1.7 to 10.2 years. The time from fitting the CI to the time of testing ranged from 0.4 to 0.6 years. Demographic data are summarized in Table 2.

Table 2.

Demographic data for the three patients.

Subject ID	AzBio Q	AzBio +5	Vowels	Consonants	Localization accuracy	Age	Duration deafness	Time since fitting
2460	77%	11%	71%	72%	280	48 yr	3.5 yr	0.5 yr
2461	18%	0%	25%	29%	540	59 yr	10.2 yr	0.6 yr
2465	76%	36%	83%	94%	250	38 yr	1.7 yr	0.4 yr

Note. AzBio Q = sentence score in quiet; yr = years.

The listeners were implanted with a 16.5-mm electrode array (Advanced Bionics) and used a Naida Q90 processor. All listeners received a postimplant, high-resolution computerized tomography scan at St. Joseph's Hospital and Medical Center in Phoenix, Arizona. The data from those scans were evaluated at Vanderbilt University using the Oto-Pilot software program (Noble et al., 2013). Table 3 shows, for each patient and electrode, the electrode angular depth in degrees and the nearest neural tissue (Hz) in the spiral ganglion. The table also shows the filter cf for each electrode and the ratio of nearest tissue to filter cf.

Table 3.

For each patient: angular depth for each electrode in degrees, the nearest tissue to each electrode, the center frequency of the filter that outputs to each electrode (filter cf), and the ratio of tissue stimulated to filter cf.

Parameter	Electrode
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Patient 2460
Angular depth	419	370	328	298	273	247	226	204	176	164	145	122	95	69	47	29
Nearest tissue (Hz)	620	750	1040	1300	1490	1940	2300	2730	3240	3700	4310	5110	5950	8700	11350	13460
Filter cf (Hz)	333	455	540	642	762	906	1076	1278	1518	1803	2142	2544	3022	3590	4264	6665
Nearest tissue/filter cf	1.9	1.6	1.9	2.0	1.9	2.1	2.1	2.1	2.1	2.0	2.0	2.0	2.0	2.4	2.7	2.0
Patient 2461
Angular depth	361	330	299	275	247	211	181	148	117	88	60	41	25	19	8	4
Nearest tissue (Hz)	780	1020	1280	1490	1940	2580	3120	4230	5210	6300	9750	12010	13980	14800	15970	16580
Filter cf (Hz)	333	455	540	642	762	906	1076	1278	1518	1803	2142	2544	3022	3590	4264	6665
Nearest tissue/filter cf	2.3	2.2	2.4	2.3	2.5	2.8	2.9	3.3	3.4	3.5	4.6	4.7	4.6	4.1	3.7	2.5
Patient 2465
Angular depth	409	365	342	305	263	240	226	204	179	158	138	120	97	77	52	34
Nearest tissue (Hz)	650	770	930	1230	1640	2060	2300	2680	3180	3850	4560	5110	5950	7620	10720	12960
Filter cf (Hz)	333	455	540	642	762	906	1076	1278	1518	1803	2142	2544	3022	3590	4264	6665
Nearest tissue/filter cf	1.9	1.7	1.7	1.9	2.1	2.3	2.1	2.1	2.1	2.1	2.1	2	2	2.1	2.5	1.9

Note. cf = center frequency.

For Patients 2460 and 2465, the most apical electrodes were near the 620- and 650-Hz place frequencies, respectively; for Patient 2461, the most apical electrode was near the 780-Hz place frequency. The ratio of place frequency to filter cf for Patients 2460 and 2465 was 2.05 and 2.04, respectively; for Patient 2461, it was 3.2.

Speech Understanding and Sound Source Localization

As shown in Table 3, using a direct connect cable for audio input to the CI, two of the three listeners (Patients 2460 and 2465) had better than average scores on tests of speech understanding, for example, 76%–77% correct on AzBio sentences in quiet, whereas one had a poor score of −18% correct. Gifford et al. (2018) report a mean score of 63% correct for the AzBio sentences in quiet for a sample of 453 CI patients. The two patients with good speech understanding scores were able to locate a wide band noise on the horizontal plane with accuracy appropriate for an SSD-CI listener, that is, 25°–28° of error (Dorman, Loiselle, Cook, Yost, & Gifford, 2016; see Yost, Loiselle, Dorman, Burns, & Brown, 2013, for a description of the test environment). The remaining listener showed poor localization ability.

Test Signals

The sentence “Do you like camping?” from the City University of New York corpus was used for testing. The sentence was first resynthesized using the STRAIGHT (Kawahara, Estill, & Fujimura, 2001) algorithm so that other manipulations, for example, pitch and formant shifting, could be implemented.

Modifications of Clean Signals

Custom-built software allowed changes in the following acoustic characteristics of clean speech signals.

• Signals could be low, high, and bandpass filtered using sixth-order (36 dB/octave) Butterworth filters. The corner frequencies were variable. Low-pass and bandpass filtering produces a muffled sound quality. Bandpassed signals commonly sound as if they are farther away than wide band signals.

• Spectral peaks could be broadened and spectral peak-to-valley differences could be reduced in a simulation of the effects of poor frequency selectivity (our algorithm was modeled after Baer & Moore, 1993). The results of spectral smearing are shown in Figure 1. For a synthetic vowel test signal, with smear = 0, the F1 peak-to-valley amplitude difference was 23.9 dB; with smear = 5, the difference was 17.1 dB; and with smear = 10, the difference was 11.8 dB. At high levels of broadening (at Level 10 on a 1–10 scale), a low level of a static-like sound was introduced. Spectral smearing, like filtering, produces a muffled sound quality.

• Formant frequencies could be altered over the range −300 to +1000 Hz. In our implementation using STRAIGHT, the distance between formants was maintained and the spectrum as a whole was shifted up or down in frequency. A pilot experiment with nine normal-hearing undergraduates indicated that a 400-Hz upshift was sufficient to elicit a reliable Munchkin-like percept. We note that this alteration is not the same as doubling the frequency of all components of a speech signal, as discussed in the introduction.

• The mean voice pitch (F0) could be increased or decreased. An increase in pitch per se did not elicit a Munchkin-like percept but rather a Mickey Mouse™–like percept.

• Speech signals could be made to sound more or less metallic by altering resonances and ring times. Since struck metal objects often have inharmonically related resonant frequencies and long ring times, a filter was constructed using a bank of sharp, inharmonically related resonances in combination with a bandpass. The resonance frequencies were f = [442 578 646 782 918 1054 1257 1529 1801 2141 2549 3025 3568 4248]. The filter used to match the metallic aspect of Patient 2465's CI sound quality is shown in Figure 2.

• A slight pitch and amplitude shift over time was implemented by a flanging operation. The flange effect was implemented by introducing a fixed delay of 0.01% to the signal and adding it back into the unaltered signal. The perceptual result is similar to a “whooshing” sound. One of our first patients, who played the guitar, suggested a flange-like effect was a part of his CI percept.

• Noise and sine vocoders could be implemented with four to 12 channels. A noise vocoder has a hissy voice quality, and a sine vocoder has an electronic “whine.” Neither has a strong impression of pitch.

Figure 1.

Linear predictive coding spectrum of synthetic vowel test signal when smear = 0, 5, and 10.

Figure 2.

Filter applied to clean signal to achieve metallic sound quality.

Procedure

The procedure to obtain sound quality matches for the CI is illustrated in Supplemental Material S1. The matching procedure was controlled by the experimenter who operated a software mixing console with sliders for the dimensions listed above. Signals were delivered to the CI via a direct connect cable and were delivered to the normal-hearing ear via an insert receiver (ER3-A). A clean signal was delivered to the CI first and then to the normal-hearing ear. The patient was asked how the signal to the normal-hearing ear should be changed to sound like the signal to the CI ear. If the patient said, for example, that the signal to the CI ear was “higher” than the signal to the normal-hearing ear, then the experimenter increased the formant frequencies and/or the F0 in the signal until the patient said that the frequency upshift was about right. The patient was then asked what else needed to be changed. This continued until the patient said that the match was very close or until the experimenter had exhausted the parameter sets. At this point, the patient was asked to rate the similarity of the signal presented to the normal-hearing ear to that of the CI on a 10-point scale, with 10 being a complete match.

Results

The results of the matching experiment are shown in Table 4 where changes in F0, formant frequencies, and other dimensions are listed, as well as the match rating. Also, for reference, the insertion angle, most apical tissue stimulated, and percent words correct are listed. Sound files for both the clean signal and the matches for all patients are found in Supplemental Material S2.

Table 4.

Outcomes of voice quality matching procedure.

Patient	Insertion angle	Most apical tissue	Percent words correct AzBio Q	Voice quality match parameters			Match rating
				F0	Formant frequency	Other
2460	4190	620 Hz	77	+10 Hz	+800 Hz	Metallic	9.7
2461	3610	780 Hz	18	−15 Hz	+300 Hz	LP 0.3 kHz; Smear 10; flange; metallic; sine vc @ −20 dB	8.0
2465	4090	650 Hz	76	+80 Hz	+600 Hz	BP 0.4–4 kHz; Smear 3; flange; metallic	9.25

Note. AzBio Q = sentence score in quiet; F0 = fundamental frequency; LP = low-pass filter; vc = vocoder; BP = bandpass filter.

The closest matches to the sound quality of the CI were obtained for Patients 2460 and 2465 who rated the matches as 9.7 and 9.25, respectively. To make a match, both patients required (a) an upshift in pitch, +10 and +80 Hz, respectively; (b) an upshift in formant frequencies, +800 and +600 Hz, respectively; and (c) a metallic sound quality. Patient 2465 also needed (a) a small amount of muffling of sound quality as indicated by the bandpass filter settings and spectral smearing and (b) the slight pitch and amplitude changes over time produced by flanging.

Patient 2461 ranked the match as an 8. This patient, like the others, needed (a) an upshift in formant frequencies (+300 Hz) and (b) a metallic sound quality. However, unlike the others, this patient needed a small decrease in pitch (15 Hz) and needed the signal to be extremely muffled—as indicated by the low-pass filter at 300 Hz and maximum spectral smearing. These manipulations are consistent with the very poor speech understanding found for this patient (cf, Warren, Bashford, & Lenz, 2017). In addition, the patient needed (a) a sine vocoder signal to be added at a low level, which produced a ringing quality, and (b) a flanger. The flanger fulfilled her request to add “something extra” that trailed behind the end of each word in the sentence.

Discussion

The aim of this project was to make audible for normal-hearing listeners the Mickey Mouse™ voice quality often heard by CI patients in the period immediately following implantation. We chose as listeners individuals implanted with a 16.5-mm electrode array who had less than 6 months of experience with their CI. We reasoned that (a) a relatively high “starting frequency” and basal shift for electrical stimulation due to the relatively short electrode array and (b) a short period of adaptation would give us the best chance to find this voice quality.

All three patients needed, for their normal-hearing ear, an upshift in formant frequencies and a metallic sound quality to match the sound quality of their CI. For two of the three, an upshift in F0 and muffling were needed. The high match rating scores for two of the three patients suggest that the matches captured much of the sound quality experienced by these patients. As we noted in the introduction, the voices of the Munchkins in The Wizard of Oz were created with both formant and pitch upshifts. Because a formant shift was a principal component of the match, in contrast to only a pitch shift, we propose that the voice quality we captured in this project be called Munchkin voice.

We had expected that the amount of upshift in formant frequencies would be related to the lowest place frequency stimulated; that is, the patient with the shortest insertion would need the largest upshift. However, this was not the case—that patient needed the smallest upshift. However, the issue is clouded by the very poor sound quality of the CI for this patient, which may have made it difficult to attend to the frequency shifts.

How Far Can We Generalize This Outcome?

In our sample, patients had at least several months to adapt to electrical stimulation. It remains to be seen whether patients during fitting or immediately after fitting hear the same voice quality as found in this study. Patient report suggests that they do not. Commonly, the percept on the day of fitting is different—very much poorer—than the percept a few months later. However, it is reasonable to suppose that increases in F0 and formant frequencies as well as muffling are a part of the initial percept.

Clinical reports indicate that some patients with long arrays, for example, 28 mm, report a higher-than-normal voice “pitch” during the initial fitting process. This is not unreasonable as signals are, in fact, upshifted (see Table 1b). For some patients, this percept changes to a lower pitch by the end of the fitting session or within days. It remains to be seen whether this voice quality—for patients with longer electrodes—is similar to the voice quality we report here and whether the changes are in F0, formant frequencies, or both. We note that formant upshifts and a metallic sound quality were not reported by the three patients in Dorman et al. (2017) who were implanted with longer electrodes.

Acoustic Versus Electric Changes to the Signal

For the patients in this experiment, a small number of acoustic alterations to a clean signal captured much of the sound quality of speech elicited by electrical stimulation. For the perceptual dimension of muffling (which has been a common outcome in published and unpublished studies from our laboratory), the acoustic operations, per se, are of less interest than the perceptual dimension. For example, muffling can be created by filtering, smearing, and, most likely, other acoustic operations. It is likely that a similar situation exists for electrical stimulation. Patient 2461 needs a low-pass filter set to 300 Hz to capture the degree of muffling she experienced with CI stimulation. Determining the distortions in the electrically evoked representation of speech that produces an effect similar to, for example, acoustic low-pass filtering is the next step in our research program.

Funding Statement

Author J. H. N. was supported by NIH R01DC014037, authors M. F. D. and S. C. N. by a grant from MED EL, and author D. M. Z. by a grant from Advanced Bionics.