Binaural pitch fusion: Binaural pitch averaging in cochlear implant users with broad binaural fusion

Abstract

Objectives:

Individuals who use hearing aids (HAs) or cochlear implants (CIs) can experience broad binaural pitch fusion, such that sounds differing in pitch by as much as 3-4 octaves are perceptually integrated across ears. Previously, it was shown in HA users that the fused pitch is a weighted average of the two monaural pitches, ranging from equal weighting to dominance by the lower pitch. The goal of this study was to systematically measure the fused pitches in adult CI users, and determine whether CI users experience similar pitch averaging effects as observed in HA users.

Design:

Twelve adult CI users (Cochlear Ltd, Sydney, Australia) participated in this study: six bimodal CI users, who wear a CI with a contralateral HA, and six bilateral CI users. Stimuli to HA ears were acoustic pure tones, and stimuli to CI ears were biphasic pulse trains delivered to individual electrodes. Fusion ranges, the ranges of frequencies/electrodes in the comparison ear that were fused with a single electrode (electrode 22, 18, 12, or 6) in the reference ear, were measured using simultaneous, dichotic presentation of reference and comparison stimuli in opposite ears, and varying the comparison stimulus. Once the fusion ranges were measured, the fused binaural pitch of a reference-pair stimulus combination was measured by finding a pitch match to monaural comparison stimuli presented to the paired stimulus ear.

Results:

Fusion pitch weighting in CI users varied depending on the pitch difference of the reference-pair stimulus combination, with equal pitch averaging occurring for stimuli closer in pitch and lower pitch dominance occurring for stimuli farther apart in pitch. The averaging region was typically 0.5 – 2.3 octaves around the reference for bimodal CI users and 0.4 – 1.5 octaves for bilateral CI users. In some cases, a bias in the averaging region was observed toward the ear with greater stimulus variability.

Conclusions:

Fusion pitch weighting effects in CI users were similar to those observed previously in HA users. However, CI users showed greater inter-subject variability in both pitch averaging ranges and bias effects. These findings suggest that binaural pitch averaging could be a common underlying mechanism in hearing-impaired listeners.

INTRODUCTION

Binaural fusion is the perceptual integration of different stimuli across ears into a single auditory object, analogous to the fusion of visual stimuli to the two eyes into a single image. Binaural fusion has been most often investigated in the context of spatial fusion, as fusion of inputs across the two ears is thought to be important for the spatial localization of sounds based on interaural timing and intensity differences using brainstem circuitry. However, binaural fusion has been under-studied in the context of pitch fusion, or the fusion of stimuli that evoke different pitches across ears, with a single resulting pitch.

Previous studies have shown that in normal hearing (NH) listeners, binaural pitch fusion typically occurs only for tone stimuli with small frequency differences of less than 0.1-0.2 octaves across the ears (Thurlow and Bernstein, 1957; Van den Brink et al., 1976; Reiss et al., 2017). However, recent studies show that binaural pitch fusion is often broadened in hearing-impaired (HI) listeners. Bilateral hearing aid (HA) users can exhibit abnormally broad binaural pitch fusion, fusing tones with interaural frequency differences up to 1-4 octaves (Oh and Reiss 2017; Reiss et al., 2017), compared to the fusion limits of <0.1-0.2 octaves of frequency difference in NH listeners. Similarly, CI users can also exhibit broad pitch fusion compared to NH listeners. In bimodal CI users, who wear a CI in one ear and a HA in the other ear, a single electrode in the CI ear can be fused with contralateral acoustic tones differing by as much as 3-4 octaves in pitch from that elicited by the electrode (Reiss et al., 2014). Similarly, in bilateral CI users, who wear CIs in both ears, a single electrode can be fused with the wide range of the electrode arrays in the opposite CI ear (Van Hoesel et al., 1993; Van Hoesel and Clark, 1997; Long et al., 2003; Kan et al., 2013; Reiss et al., 2018).

Previously, it was shown in both bilateral HA and bimodal CI users that pitch fusion leads to a percept that is a weighted average of the original monaural pitches (Reiss et al., 2014; Oh and Reiss 2017). A systematic measurement of the perceived pitch of the fused stimulus conducted in bilateral HA users showed that the weighting varied from equal weighting between the two monaural pitches to domination by the lower pitch of the pair (Oh and Reiss, 2017).

Such pitch averaging can lead to detrimental averaging of spectral information between the ears in hearing-impaired listeners. Binaural speech interference, or specifically worsened vowel discrimination with two ears than one ear alone, has been demonstrated in both bilateral HA and bimodal CI users with broad fusion (Reiss et al., 2016). Further, recent studies suggest that broad fusion is associated with greater difficulty in using voice gender cues to separate a target voice from other voices (Oh et al., 2018). In fact, subjects often reported that voices fused together and that they heard new sound blends that were not part of the actual set of words presented, suggesting that the difficulties with noise were directly due to broad fusion and pitch averaging.

The current study was designed to systematically measure these pitch averaging weightings as a function of stimulus pair in CI listeners. This was conducted for both bimodal CI and bilateral CI users with broad fusion, in order to determine if similar pitch averaging trends occur in CI users as for HA users.

MATERIALS AND METHODS

Subjects

These studies were conducted according to the guidelines for the protection of human subjects as set forth by the Institutional Review Board (IRB) of Oregon Health & Sciences University (OHSU), and the methods employed were approved by that IRB. Six adult bimodal CI subjects with residual hearing in the contralateral, non-implanted ear and six adult bilateral CI subjects (total twelve subjects with two males and four females in each group) participated in the experiments. All subjects were screened for normal cognitive function using the Mini Mental Status Examination with a minimum score of 25 out of 30 required to qualify (MMSE; Folstein et al., 1975; Souza et al., 2007). In addition, all CI subjects were required to have binaural fusion ranges broad enough (greater than 1-octave ranges) to allow measurement of binaural pitch averaging trends across frequency and electrode for the bimodal and bilateral CI subjects, respectively. Note that the fusion range in an acoustic octave-scale for the bilateral CI subjects was converted from the scale in mm based on the observed mostly linear relationship of cochlear place to log frequency above the 1000 Hz cochlear place for both the organ of Corti and spiral ganglion maps (~1 octave per 5 mm; Greenwood, 1990; Stakhovskaya et al., 2007). Three bimodal CI subjects (CI24, CI25, and CI56) were sampled from a larger pool of subjects in a previous study (N=21, Reiss et al., 2014) and three new bimodal CI subjects (CI53, CI68, and CI73) were recruited specifically for the current study. All bilateral CI subjects were also sampled from a recent study (N=18, Reiss et al., 2018).

All subjects had CIs made by Cochlear Ltd. (Sydney, Australia), which consist of 22 intra-cochlear stimulation electrodes. The subjects’ demographic data including ages, gender, duration of severe/profound deafness, duration of CI use, CI internal device, and reference ear are shown in Table 1. In addition, individual audiograms are shown for both bimodal CI and bilateral CI listeners in Figure 1. Unaided audiometric thresholds for non-implanted ears (dotted line in Figure 1.A.) were obtained for the bimodal CI user group, and sound field thresholds for CI ears were presented for both CI user groups. Note that the unaided thresholds for the implanted ears were not measured due to the assumption of a total loss of hearing. All CI subjects in the study had at least one year of experience with their CI and/or HA devices and used their hearing devices on a regular basis.

Table 1.

Demographic information for all subjects: age, gender, duration of severe to profound hearing loss, duration of CI use, CI internal device, and reference ear used in testing. Avg=average; Std=standard deviation; L=left; R=right; S/P HL=severe to profound hearing loss; n/a=not applicable. CI24R, CI24RE, and CI512 are all pre-curved modiolar arrays. CI24M and CI422 are all straight lateral wall arrays.

Subject ID	Age (years)	Gender	Duration S/P HL (years)	Duration CI Use (years, L; R)	CI Internal Device (L; R if diff.)	Reference Ear
Bimodal CI (N=6)
CI24	57	M	12	4	CI512	R
CI25	58	F	17	4	CI512	R
CI53	61	F	15	4	CI24RE	R
CI56	64	M	27	14	CI24R	R
CI68	54	F	54	3	CI422	L
CI73	48	F	38	6	CI512	L
Avg	57.0	n/a	27.2	5.8	n/a	n/a
Std	5.6		16.2	4.1
Bilateral CI (N=6)
BI03	46	F	42	6;6	CI512; CI24RE	R
BI04	56	M	25	5;5	CI512	R
BI06	49	F	8	2;1	CI24RE	R
BI09	18	M	18	17;11	CI24R; CI24M	L
BI14	52	F	33	7;13	CI24R; CI24RE	L
BI15	51	F	40	2;2	CI24RE	L
Avg	45.3	n/a	27.8	6.5; 6.3	n/a	n/a
Std	13.8		13.1	5.5; 4.8

Fig. 1.

Audiograms for the subjects in this study. A. Unaided audiograms for non-implanted ears (dotted lines) and sound field thresholds for implanted ears (solid lines) of the bimodal CI subjects. B. Sound field thresholds for the bilateral CI subjects, with solid and dashed lines showing individual thresholds for the left and the right ears, respectively. Note that all sound field thresholds were measured with subject’s own CI processors.

All subjects were paid an hourly wage ($25/hour) and completed all experiments in between three to seven sessions of 3-4 hours each. No prior experience with psychophysical research was required for participation; however, practice tutorials (20-30 minutes) were provided to all subjects in order to assure familiarity with the procedures.

Stimuli and procedures

All experiments were conducted in a double-walled, sound attenuated booth. For CI stimulation for both CI user groups, electric stimuli were delivered via computer directly to the CI using NIC2 CI research software (Cochlear Ltd, Sydney, Australia) with L34 speech processor(s) via the programming pod interface. Note that the current study used the direct electrical stimulation through the research interface, which allows us to selectively stimulate one electrode at a time. Only even-numbered electrodes were used for the experiment in order to limit the number of trials. All subjects had all activated electrodes from electrode 1 to 22 and used the standard frequency-to-electrode allocations in their clinical programs, except for one bilateral CI subject (BI14) who had one deactivated odd-numbered electrode (electrode 19) in the left CI and shifted frequency-to-electrode allocations between electrode 14 to 18 in the clinical program.

Direct stimulation of each electrode consisted of a pulse train of either 25-μs or 37-μs biphasic pulses presented at 1,200 pulses per second (pps) / electrode, above the temporal rate limit (200 – 300 pps) for pitch in most CI users (Kong et al., 2009), although some exceptional users can discriminate pitch changes at far higher rate, up to 900 pps (Kong and Carlyon, 2010). Almost all of the subjects used the 900 pps with 25-μs biphasic pulses in their everyday programs, with exception of one bimodal CI subject (CI24) and two bilateral CI subjects (BI06 and BI09) who used 37-μs biphasic pulses. Only one bilateral CI subject (BI09) used the 1,200 pps in the left ear for the everyday program. The electrode ground was set to monopolar stimulation with both the ball and plate electrode active (MP1+2). For acoustic stimulation for the bimodal CI subjects, acoustic stimuli were delivered to the non-implanted ear using an ESI Juli sound card, TDT PA5 digital attenuator and HB7 headphone buffer, and Sennheiser HD-25 headphones. Each headphone’s frequency response was equalized using calibration measurements obtained with a Brüel & Kjær sound level meter with a 1-inch microphone in an artificial ear. All acoustic stimuli consisted of sinusoidal pure tones with 10-ms raised-cosine onset/offset ramps. It should be noted that timing cues of all stimuli were controlled by computer and bilaterally synchronized.

The signal stimuli were loudness balanced before the experiment by using a method of adjustment. First, the level of the electric stimulation for each electrode in the reference CI ear was initialized to a “medium loud and comfortable” current level corresponding to 6 or “most comfortable” on a visual loudness scale from 0 (no sound) to 10 (too loud). Then, for bimodal CI subjects, acoustic tones were presented to the contralateral, non-implanted ear and set to “medium loud and comfortable” levels again using the same loudness scale as for electric levels. Tone frequencies that could not be presented loud enough to reach a “medium loud and comfortable” level due to too much hearing loss at those frequencies were excluded; this determined the upper limits of the loudness balanced frequency range of the acoustic ear. For bilateral CI subjects, all contralateral CI electrodes were loudness-balanced sequentially with the reference electrodes. The electrodes/frequencies and order of presentation were randomized to minimize the effect of biases such as time-order error and underestimation or overestimation of the loudness (Florentine et al., 2011). This loudness balancing procedure was performed to minimize use of loudness-difference cues and maximize focus on pitch differences as the decision criteria.

Stimuli were presented binaurally in a set of three experiments: dichotic fusion range measurement, interaural pitch matching, and fusion pitch matching.

For the bimodal CI subjects, the reference electrode for the CI ear, designated the reference ear, was fixed at a single electrode (electrode 22, 18, or 12), and comparison tone frequencies presented to the contralateral ear were varied in ¼ octave steps between 0.125 kHz and the upper frequency limits of the loudness balanced frequency range: 0.75 kHz for CI56, 2 kHz for CI24, CI53, and CI 68, 3 kHz for CI25, and 4 kHz for CI73. Apical electrodes were used in bimodal CI users because of the limited frequency range of the acoustic hearing needed to make pitch comparisons. Measurements were repeated for all three reference electrodes for each subject.

For the bilateral CI subjects, the ear with worse within-ear electrode pitch discrimination was designated as the reference ear with a fixed reference electrode (electrode 18, 12, or 6 to span the electrode array), and all even-numbered electrodes on the contralateral ear used as comparison electrodes. The worse ear was assigned to be the reference ear so that the resolution of comparison electrode testing would be maximized using the better ear, instead of limited by the worse ear. The worse ear was chosen based on a global pitch (electrode) ranking score rather than a local (per electrode) measurement, as described previously in Reiss et al. (2018). Briefly, electrode discrimination was measured using a pitch ranking test in which all possible pairs of electrodes were presented twice, in a two-interval, two-alternative forced choice procedure. In each trial, 500-msec pulse trains were delivered sequentially within one ear, with a 500-msec interstimulus interval with randomized interval order. The pitch ranking score was computed by indicating which stimulus had the higher pitch. Reference ears are shown in Table 1. The contralateral ear was the comparison ear. Measurements were repeated for all three reference electrodes for each subject.

All experiments used a constant-stimulus procedure to compute the 25%, 50%, or 75% point on subject’s response functions. The results for all experiments were averaged with two separate runs for each reference electrode in conditions as described briefly below (for additional details for bimodal CI: Reiss et al., 2014; bilateral CI: Reiss et al., 2018).

1. Dichotic fusion range measurement

Dichotic fusion range measurements were performed under interaurally simultaneous presentation to measure the fusion range, which is either the range of frequencies for bimodal CI subjects or the range of electrodes for bilateral CI subjects in the comparison ear that were fused with a single electrode in the reference ear. Both reference and dichotic comparison stimuli were presented simultaneously in a 1500-ms single interval (see schematic of the stimuli in Fig. 2A), in which subjects were asked to indicate whether they heard a single fused sound or two different sounds in each ear. Within each run, the reference stimulus (black in Fig. 2A) was fixed, and the comparison stimuli (light gray in Fig. 2A) were pseudo-randomly varied at each trial, with six repeat presentations of each comparison stimulus. The comparison stimuli in the contralateral ear were acoustic tones for bimodal CI subjects, and electrodes in the contralateral CI for bilateral CI subjects. Subjects were asked to indicate on a touchscreen whether they heard one or two sounds, indicated as buttons. The subject’s responses averaged over the repeat presentations formed the fusion function, with a value of 0 assigned to comparison frequencies over which two sounds were heard, and a value of 1 assigned to comparison frequencies over which one sound was heard.

Fig. 2.

Schematics of the stimuli used for the three different tasks: (A) Dichotic fusion range measurement, (B) Interaural pitch matching task, and (C) Fusion pitch matching task. The reference ear stimulus (black) is shown at top, and the contralateral ear stimulus (light gray: comparison stimulus; dark gray: paired dichotic stimulus) is shown at bottom for each task, representing stimuli presented in a single trial presentation.

All subjects understood the concept of fusion as assessed during the task familiarization, where subjects were given practice trials with feedback before the fusion range measurement. This was verified by repeatable fusion function shapes over repeat runs.

2. Interaural pitch matching

Interaural pitch matches were calculated from a pitch discrimination task using interaurally sequential presentation to find the perceived pitch of an electrode relative to tones or electrodes in the other ear. A 2-interval, 2-alternative constant-stimulus paradigm was used to estimate a psychometric function. Reference and comparison stimuli were each 500 ms in duration and separated by a 500-ms interval (see schematic of the stimuli in Fig. 2B), with interval order randomized. The reference electric stimulus (black in Fig. 2B) was held constant in the reference ear, and the comparison stimulus (light gray in Fig. 2B), which was either an acoustic tone for bimodal CI subjects or electric stimulation for bilateral CI subjects, was sequentially presented in the contralateral ear with pseudorandom sequences across trials counterbalanced to average out sequence effects as in Reiss et al. (2012). In each trial, subjects were asked to select which interval had the higher pitch using a touchscreen with two buttons. The next trial would start immediately after the subject responded, and response feedback was not given. The psychometric functions were generated from the average of the responses to six repeated presentations of each comparison stimulus.

3. Fusion pitch matching

Fusion pitch matches were conducted using both simultaneous dichotic and sequential monaural presentation to find the pitch of a fused dichotic stimulus. The procedure for the fusion pitch matching measurement was a 2I-2AFC task similar to that used for the interaural pitch matching measurement, except that a fused dichotic stimulus was presented as the reference stimulus (see schematic of the stimuli in Fig. 2C). That is, the reference stimulus in this task consisted of a “reference stimulus” in the reference ear (black in Fig. 2C) presented simultaneously with a “paired dichotic stimulus” in the contralateral ear (dark gray in Fig. 2C) that was found to be fused together in the dichotic fusion range task. For each “reference stimulus”, fusion pitch matching measurements were conducted for multiple “paired dichotic stimuli”. These “paired dichotic stimuli” (tone frequencies for bimodal CI subjects or electrodes for bilateral CI subjects) were selected to sample approximately linearly spaced intervals within the subject’s fusion range (four to eight inharmonic frequencies for bimodal CI subjects, or electrodes for bilateral CI subjects), and outside ±1 standard deviation of the 50% point of the pitch match. This reference stimulus, i.e. the reference and paired dichotic stimulus, was fixed for each run. A third stimulus, the comparison stimulus (light gray in Fig. 2C), was presented to the contralateral ear sequentially with the reference stimulus, with the comparison stimulus (frequency for bimodal subjects or electrode for bilateral CI subjects) varied pseudo-randomly across trials. All stimuli were 500 ms in duration, and the reference and comparison stimuli were separated by a 500-ms inter-stimulus interval, with interval order randomized. Subjects were asked to choose the stimulus with the higher pitch using a touchscreen with two buttons. Psychometric functions were generated similarly as for the interaural pitch matching measurement. All subjects performed the fusion pitch match tests with all three reference electrodes, with the exception of one bilateral CI subject (BI06), who did not have broad enough fusion at electrode 6 to conduct fusion pitch matching.

All statistical analyses were conducted on octave-scale data in SPSS (version 25, IBM). Due to the small sample sizes (N=6 for each group), non-parametric tests (Kruskal-Wallis H test) were performed to compare fusion ranges and pitch averaging ranges between two CI user groups. In addition, the two groups used different electrodes as references (bimodal CI: 22, 18, and 12; bilateral CI: 18, 12, and 6) so that equivalent electrodes 18 and 12 were only considered as independent variables for statistical comparison.

RESULTS

1. Binaural pitch fusion ranges

Figure 3 shows individual fusion range results as thin vertical lines. Each panel represents results for different reference electrodes for bimodal CI users (Fig. 3A) and for bilateral CI users (Fig. 3B).

Fig. 3.

Individual fusion range and interaural pitch match results for bimodal CI users (A) and bilateral CI users (B). Each panel shows results for one reference electrode: for bimodal CI users, electrodes 22, 18, and 12, and for bilateral CI users, electrodes 18, 12, and 6. The numbers in the x-axes indicate subject ID. Thin vertical lines indicate fusion ranges, and the open squares at the top and bottom of each line indicate that fusion ranges may exceed the upper or lower limits of the range tested. Thick vertical lines and filled circles show 25-75% ranges and 50% points of pitch/electrode match, respectively. Horizontal shaded areas in the panel (A) represent the frequencies-to-electrode allocations for each reference electrode for bimodal CI users, and horizontal dashed lines in the panel (B) represent each reference electrode for bilateral CI users.

All bimodal CI subjects sampled in this study exhibited broad fusion ranges across all reference electrodes (Fig. 3A; mean and std = 1276 ± 1041 Hz or 2.71 ± 0.9 octaves, 1238 ± 552 Hz or 2.61 ± 0.87 octaves, and 1578 ± 908 Hz or 3.16 ± 0.93 octaves for reference electrode 22, 18, and 12 , respectively), as desired for the experiment. Those broad fusion ranges did not differ statistically across the reference electrodes (H(2)=3.066, p=0.216). For most bimodal CI subjects, either the lower or upper boundaries of the fusion ranges were the maximum frequency limits (open squares) used in the experiments for at least one reference electrode, with the exception of subject CI73 who had the fusion range boundaries within the maximum frequency limits tested across all reference electrodes.

Bilateral CI subjects sampled in this study similarly had broad binaural fusion ranges (Fig. 3B; mean and std = 14 ± 5 electrodes or 7.38 ± 3.12 mm, 16 ± 3 electrodes or 8.68 ± 1.87 mm, and 12 ± 7 electrodes or 6.44 ± 3.35 mm for reference electrode 18, 12, and 6, respectively). The fusion range results in units of millimeters were calculated using the following inter-electrode distances: approximately 0.75 mm for Cochlear internal device CI24M, and variable distances between 0.4 and 0.81 mm for Cochlear CI24R/CI24RE/CI512 and between 0.85 and 0.95 for Cochlear CI422 arrays. Distances in mm were then converted to an acoustic octave scale by using an approximation of 5 mm/octave observed for both the organ of Corti and spiral ganglion for 1000 Hz and up (Greenwood, 1990; Stakhovskaya et al., 2007). Hence, these fusion ranges in mm correspond to 1.48 ± 0.62 octaves, 1.74 ± 0.37 octaves, and 1.29 ± 0.67 octaves for reference electrode 18, 12, and 6, respectively. Again, those broad fusion ranges did not differ statistically across the reference electrodes (H(2)=1.234, p=0.540). In addition, the electrode numbers in Cochlear devices in Figure 3B are numbered from the base (high frequency region) to the apex (low frequency region), and so are plotted in reverse order on the y-axis to represent low to high frequency regions from bottom to top, consistent with the bimodal CI plots.

2. Interaural pitch matches

Figure 3 shows individual interaural pitch match results for bimodal CI users (Fig. 3A) and for bilateral CI users (Fig. 3B) with interaural pitch match 50% points as solid dark circles and 25%-75% ranges overlaid as thick dark vertical lines.

For bimodal CI subjects, the 50% points of the pitch matches (solid dark circles) did not align with the programmed frequency-to-electrode allocations (horizontal bars) in many cases (mean and std = 452 ± 332 Hz, 501 ± 379 Hz, and 722 ± 568 Hz for reference electrodes 22, 18, and 12, respectively). Note that all bimodal CI subjects used the standard frequency-to-electrode allocations in their clinical program, and the reference electrode 22, 18, and 12 were allocated to the frequency ranges of 188-313 Hz, 688-813 Hz, and 1563-1813 Hz, respectively. For reference electrode 22, the 50% interaural pitch matches were close to the programmed frequencies in subjects CI24, CI25, and CI56, but higher in subjects CI53, CI68, and CI73. For other reference electrodes (electrodes 18 and 12), most subjects had lower interaural pitch matches than the programmed frequencies, with the exception of subject CI53. Relative to the geometric center frequency of the corresponding frequency-to-electrode allocation, averaged pitch mismatches were calculated as 0.5 ± 1.12 octaves, −0.91 ± 1.06 octaves, and −1.56 ± 1.08 octaves for reference electrodes 22, 18, and 12, respectively.

For bilateral CI subjects, the 50% points of the pitch matches (solid dark circles) were varied depending on the reference electrodes (mean and std = 18 ± 2 electrode, 15 ± 4 electrode, and 10 ± 3 electrode for reference electrode 18, 12, and 6 respectively). Relative to the reference electrode, averaged pitch mismatches were calculated as −0.19 ± 1.37 mm, −2.07 ± 2.13 mm, and −2.27 ± 1.67 mm for reference electrode 18, 12, and 6 respectively. By the same conversion method as used in the fusion range results, averaged interaural pitch mismatches in mm correspond to −0.04 ± 0.27 octaves, −0.41 ± 0.43 octaves, and −0.45 ± 0.33 octaves for reference electrodes 18, 12, and 6, respectively.

3. Binaural pitch averaging within the fusion range

The fusion pitch matching procedure was used to measure the pitch evoked by each fused stimulus pair, by pitch-matching to comparison stimuli presented monaurally to either ear. Recall that each fused stimulus pair refers to a reference electrode + paired contralateral tone for bimodal CI users, or to a reference electrode + paired contralateral electrode for bilateral CI users; hence pair tone frequency refers to the frequency of the contralateral tone for bimodal CI users, and pair electrode refers to the contralateral electrode for bilateral CI users. Fusion pitch matching was conducted for all pair stimuli that fused with the reference electrode. From this series of measurements, summary fusion pitch averaging plots were generated as shown in Figure 4.

Fig. 4.

Example fusion pitch match results. A. Fusion pitch matches for electrode 12 for a bimodal CI user. Each vertical gray bar represents the 25-75% range on the fusion pitch match psychometric function (inset) for a specific pair stimulus indicated on the x-axis. The horizontal line in the middle of each gray bar represents the fusion pitch match, defined as the 50% point on the psychometric function (inset). The wide solid horizontal line across the plot shows the reference electrode interaural pitch match 50% point, with standard deviations shown as error bars at the right end of the horizontal line. The diagonal dashed line indicates where a 1:1 relationship between the pair frequency and pitch-matched frequency would occur. B. Fusion pitch matches for a bilateral CI user, plotted as in A. C. Fusion pitch index (FPI, Equation 1) curves for the bimodal CI user in A, indicating the weighting of the fusion pitch averaging. Vertical dashed dotted lines and the dark gray shaded colorbar both indicate the boundaries of the pitch averaging range, defined as FPI values between 0.2-0.8. The lower-pitch dominance region, defined as FPI values <0.2 or >0.8, is shown in the colorbar as light gray. The dotted vertical line and small white bar in the colorbar both indicate the 50% point of the reference electrode pitch match. D. Fusion pitch indices for the bilateral CI user in B, plotted as in C.

As an example, Figure 4A shows fusion pitch match results for a fixed reference electrode (electrode 12) as a function of paired tone frequency for a bimodal CI user (CI68). Each vertical gray shaded bar on this plot indicates the 25-75% range for a fusion pitch match psychometric function (inset) measured for the reference electrode paired with a specific tone frequency indicated on the x-axis. The horizontal line in the middle of each gray bar represents the fusion pitch match, defined as the 50% point on the psychometric function (inset). The error bars at the top and bottom of each gray bar represent the standard deviations of the 25-75% points across repeat trials. The wide horizontal line in Fig. 4A indicates the original pitch match for the reference electrode alone, so that points on this line indicate that the fusion pitch match was equal to (dominated by) the reference electrode pitch (e.g. gray bars for pair frequencies above 1.5 kHz). The diagonal dashed line in Fig. 4A indicates a 1-1 relationship of fusion pitch match to the tone pair frequency, so that points on this line indicate that the fusion pitch match was equal to (dominated by) the pair tone frequency (e.g. gray bars for pair frequencies below 0.5 kHz). Points intermediate between the horizontal and diagonal line indicate that the fusion pitch match was a weighted average of the pitch evoked by the reference electrode and paired tone (e.g. gray bars for pair frequencies between 0.5-1.5 kHz). Representative results are shown similarly in Fig. 4B for a bilateral CI user (BI06) as a function of paired electrode).

Similar fusion pitch averaging trends were observed for the representative subjects of both listener groups shown in Figure 4. When the paired stimulus was distant from the reference electrode pitch (distant from the point of intersection between diagonal dashed line and horizontal line), the fusion pitch match 50% points were close to either the paired stimulus pitch (diagonal dashed line) or the reference electrode pitch (horizontal solid line). In these cases, the fused binaural pitch was always more strongly weighted toward or dominated by the lower pitch stimulus component.

When the paired stimulus was closer to the reference electrode pitch, either just above or below the reference electrode pitch, the fusion pitch match 50% points were intermediate between the paired stimulus pitch (diagonal dashed line) and the reference electrode pitch (horizontal solid line). That is, for these paired stimuli, the fused binaural pitch was an equally weighted average of the original pitches perceived in either ear alone. Individual fusion pitch match results for all subjects and reference electrodes are shown in Fig. S1–S6, Supplementary Digital Content 1.

The weighting of the fused binaural pitch was quantified by a fusion pitch index (FPI), which is defined as the absolute value of the difference between the fused binaural pitch 50% points and the paired stimulus, normalized by the difference between the interaural pitch match 50% point and the paired stimulus (Reiss et al., 2014; Oh and Reiss, 2017):

$$\begin{gathered} \mathit{FPI}\triangleq \left|\frac{{\mathit{Fusion\; Pitch}}_{50\%}-\mathit{Paired\; Stimulus}}{{\mathit{Pitch\; Match}}_{50\%}-\mathit{Pairerd\; Stimulus}}\right| \\ =\{\begin{array}{ll}1,& {\mathit{if\; Fusion\; Pitch}}_{50\%}\approx {\mathit{Pitch\; Match}}_{50\%}\\ 0,& {\mathit{if\; Fusion\; Pitch}}_{50\%}\approx \mathit{Paired\; Stimulus}\end{array} \end{gathered}$$ (1)

where Fusion Pitch_50%, Pitch Match_50%, and Pair Stimulus were either in the unit of frequency in Hz or electrode for bimodal or bilateral CI subjects, respectively. The FPI approaches a value of 0 when the fused binaural pitch is close to that of the paired stimulus, which indicates that the fused binaural pitch is dominated by the paired stimulus. Conversely, the FPI approaches a value of 1 when the fused binaural pitch is close to the interaural pitch match, which indicates that the fused binaural pitch is dominated by the reference electrode pitch. FPI values intermediate between 0 and 1 indicate pitch averaging between ears. Thus, the FPI is a measure of how much the fused pitch is weighted toward the pitch of the paired stimulus (0) or the reference stimulus (1). Figures 4C and D illustrate how the FPI functions quantify the weightings for the representative subjects shown in Figs. 4A and B, respectively. For instance, the FPI is at 0 when the fusion pitch is aligned with the paired stimulus along the diagonal line, gradually increases in the intermediate pitch averaging region where the fusion pitch is between the diagonal and horizontal line, and is at 1 when the fusion pitch is aligned with the reference stimulus pitch along the wide horizontal line.

In both subjects, the FPI functions monotonically increased as the paired stimulus pitch increased. The 20% to 80% points on the FPI functions (indicated by vertical dashed-dotted lines in Fig. 4C–D) were chosen to quantitatively measure the boundaries of this averaging region. Pitch averaging by this definition (FPI between 0.2-0.8) occurred within the range of 595 Hz and 1165 Hz of the paired tone frequency for the bimodal CI subject (Fig. 4C) and between electrodes 17 and 9.8 (as determined by interpolation) for the bilateral CI subject (Fig. 4D). Outside of these pitch averaging ranges, FPI values <0.2 indicated pitch dominance by the pair stimulus and FPI values >0.8 indicated pitch dominance by the reference, i.e. dominant weighting by the lower pitch component of the pair. These pitch averaging ranges are also indicated by the shading of the horizontal colorbars above each plot, with dark gray shading indicating the pitch averaging range and light gray shading indicating lower pitch dominance outside of the pitch averaging range.

Further, asymmetric pitch averaging was observed in both subjects. The center of the pitch averaging range (region between FPI values of 0.2-0.8, between the vertical dash-dot lines in Fig. 4C–D) was shifted toward the paired stimulus above the pitch match 50% point (vertical dotted line or white bar in Fig. 4C–D; 634 Hz for CI68 and electrode 17.06 for BI06), rather than centered around the pitch match 50% point.

Individual pitch averaging ranges (range of pair stimuli with FPI values between 0.2-0.8) for all bimodal and bilateral CI subjects are shown as vertical lines in Figure 5A and 5B, respectively. This weighting bias in pitch averaging range toward fused pitches above the pitch match 50% point (filled circle) of the reference stimulus alone (offset of vertical line relative to filled circle) was observed in many subjects, particularly for more apical electrodes. However, some subjects had either the opposite weighting bias toward the upper boundaries (bimodal CI subjects CI53 and CI56 for the reference electrode 12, and bilateral CI subjects BI14 and BI15 for the reference electrode 18 and 6, respectively) or relatively unbiased weighting of their pitch averaging (BI15 and BI09 for the reference electrode 12 and 6, respectively). The pitch averaging region was 0.5 – 2.3 octaves around the reference for bimodal CI users (mean and std = 685 ± 599 Hz or 1.51 ± 0.78 octaves, 672 ± 420 Hz or 1.19 ± 0.52 octaves, and 517 ± 249 Hz or 1.03 ± 0.57 octaves for reference electrodes 22, 18, and 12, respectively) and 0.4 – 1.5 octaves for bilateral CI users (mean and std = 7 ± 2 electrodes or 0.79 ± 0.22 octaves, 10 ± 2 electrodes or 1.07 ± 0.27 octaves, and 7 ± 4 electrodes or 0.85 ± 0.49 octaves for reference electrodes 18, 12, and 6, respectively). Those pitch averaging ranges did not differ statistically between the two listener groups when matched by electrode (H(1)=0.712, p=0.399).

Fig. 5.

Individual pitch averaging range and interaural pitch/electrode match results plotted by subject are shown for bimodal CI users (A) and bilateral CI users (B). Each panel shows results for one reference frequency/electrode. The numbers in the x-axes indicate subject ID. Thin vertical lines with the open diamonds at the top and bottom indicate the pitch averaging ranges (the range of pair stimuli in Fig. 4C/D for which FPI is between 0.2-0.8). The filled circles show the 50% points of the reference electrode pitch match. Horizontal shaded areas in the panel (A) represent the frequencies-to-electrode allocations for each reference electrode for bimodal CI users, and horizontal dashed lines in the panel (B) represent each reference electrode for bilateral CI users.

Population scatterplots (Fig. 6A and 6B) of the centers of the pitch averaging ranges versus the reference pitch matches show that the centers tend to be higher than the pitch matches (offset relative to diagonal line), confirming that the upward weighting bias in pitch averaging occurs for the majority of listeners and electrodes in both groups. Again, all individual FPI functions and quantitative 20% to 80% pitch averaging boundaries are shown in Fig. S1–S6 in Supplementary Digital Content 1.

Fig. 6.

Population summary of pitch averaging centers as a function of reference electrode pitch match are shown for bimodal CI users (A) and bilateral CI users (B), representing the offset between the pitch averaging ranges and pitch matches shown in Fig. 5. Points on the diagonal dotted lines indicate symmetry of the pitch averaging range around the reference electrode pitch match.

DISCUSSION

In this study, as expected, both bimodal and bilateral CI users showed pitch averaging weighting of fused binaural stimuli similar to those seen in bilateral HA users (Oh and Reiss, 2017); the fused pitch was a weighted average of the two monaural pitches, ranging from equal weighting to dominance by the lower pitch component of the pair. Pitch averaging has also been observed in NH listeners over the limited fusion range observed in that group (Van den Brink et al., 1976). However, the CI users, especially bimodal CI users, showed slightly broader pitch averaging regions (bimodal CI users: 1.24 ± 0.24 octaves; bilateral CI users: 0.9 ± 0.15 octaves; averaged over all electrodes) than HA users with broad fusion (0.54 ± 0.09 octaves, Oh and Reiss, 2017). Taken together, the findings indicate that pitch averaging occurs independent of type of hearing stimulation, likely governed by global principles that generally govern binocular and multisensory integration.

While both bimodal and bilateral CI users demonstrated fusion pitch averaging in all three reference electrodes, the fusion pitch averaging region was slightly broader in bimodal CI users (0.5 – 2.3 octaves) than bilateral CI users (0.4 – 1.5 octaves). However, this is a small sample, with significant variability in the bimodal group, and differences were not significant when matched by electrode (H(1)=0.712, p=0.399). Another caveat is that the conversions of electrode distances to mm in the bilateral CI subjects is approximate, due to variations in electrode-to-modiolar distance that will affect the effective inter-electrode distance in mm along the organ of Corti or spiral ganglion (Landsberger et al., 2015).

As found in the previous study in bilateral HA users, the weighting of pitch averaging was biased toward the ear used to make the pitch comparisons in their fused binaural pitch results, i.e. the ear with greater stimulus variability (Oh and Reiss, 2017). Possible causes of the weighting biases include non-sensory factors. For instance, applying different procedures including randomization or interleaving of the reference ear and stimulus set across ears to balance variability across ears have been shown to reduce these pitch averaging range biases (Oh and Reiss, 2017). Stimulus variability may serve to orient attention that might affect the relative weighting between ears (Posner, 1980; Asbjørnsen and Bryden, 1996; Thompson et al., 2011). In addition, in CI listeners, tasks for finding a pitch match are even more difficult due to the variable timbre of the electric pitch, and biases might also depend on the stimulus factors such as the frequency range of the comparison stimuli (Carlyon et al., 2010; Goupell et al., 2019). While the weightings obtained in the present study show bias, the finding of pitch averaging itself is still robust and likely valid because the bias simply shifts the pitch averaging ranges from the unbiased presentation (Fig. 7 in Oh and Reiss, 2017). The greater bias of the pitch averaging ranges observed for apical compared to basal electrodes is interesting, and we can speculate that this could be due to factors such as perceptual differences in weighting between the apex and base that may affect fusion pitch averaging. Further study is needed to better understand the effects of attention and other factors that can shift weighting.

However, CI users showed greater inter-subject variability in both pitch weighting ranges and weighting biases compared to HA users. This variability may be due to the inherent variability of electrode insertion depths and distances from the modiolus, which lead to greater variability in the neural populations and corresponding frequency regions that are effectively stimulated (Landsberger et al., 2015). Another factor may be the plasticity that has been observed in pitch-electrode relationships (e.g. Reiss et al., 2007; Svirsky et al., 2015), as well as potentially with fusion ranges (Reiss et al., 2014), which has not been observed in HA users and could lead to greater variability in both pitch weighting and biases.

Recently it was shown that the presence of broad binaural fusion and the associated pitch averaging might cause binaural speech interference in bilateral HA users and bimodal CI users at the phoneme level, i.e., worsened vowel discrimination with two ears than one ear alone (Reiss et al., 2016). In that experiment, vowel continua were created by varying the first formant (F1) between two vowels while keeping the second formant (F2) and other parameters constant. For different synthetic vowel continua such as /I/-/i/ and /α/-/Λ/, bimodal CI users showed various binaural vowel integration patterns, including dominance by one ear, averaging of both monaural patterns, and binaural speech interference in which the binaural discrimination function was shallower than that of either ear alone. Ear dominance was seen for individuals with narrow fusion, while all three patterns were seen for individuals with broad fusion, and could vary within individuals depending on the specific vowel pair and formant frequencies. The variation in binaural pitch averaging weightings with frequency or electrode observed in this study may explain the variation in these binaural vowel integration patterns. For instance, one subject, CI25, was also tested in this study, and showed binaural speech interference for /I/-/i/ and dominance of vowel perception by the CI ear for /α/-/Λ/ and /ε/-/I/. The binaural speech interference for /I/-/i/ may occur because the lower first formant frequency range for that pair (265-388 Hz) falls into the frequency allocation range for electrode 22, which fuses and averages spectrally with the acoustic hearing. In contrast, the higher first formant frequencies of /α/-/Λ/ (625-800 Hz) falls into the frequency allocation range for electrode 18 which has just as broad fusion, but little to no pitch averaging (compare Fig. 3A and Fig. 5A), which may explain the reduced binaural speech interference seen for this vowel pair. Further study is also needed to better understand the relationship between binaural speech interference and binaural pitch averaging phenomena in dichotic vowel perception.

Why does binaural pitch averaging occur? It has been proposed in other sensory systems that such averaging is necessary to reduce independent sensory noise and signal uncertainty, and improve signal detection (Hillis et al., 2002). For example, in the human visual system, two visual inputs from two eyes are fused to form a single percept, which is known as binocular vision. Unequal stimulation of the two eyes can produce a binocular brightness, which is a weighted average of two monocular impressions (Levelt, 1965). Similar averaging has also been observed for visual movement direction or color (e.g. Anstis and Rogers, 2012), and also for multisensory integration (e.g. Binda et al., 2007). Hence, perceptual averaging appears to be a common phenomenon for sensory coding with a benefit of averaging out sensory receptor noise, at least when similar stimuli are fused together. However, as noted above, this may lead to interference rather than benefit when more dissimilar stimuli are fused in individuals with broad fusion.

Recent findings show that temporally coherent modulation can increase binaural fusion in both NH listeners and HA users (Oh and Reiss, 2018). Conversely, large interaural intensity differences greater than 10-20 dB outside of the predominant physiological range may also reduce fusion, at least for speech sounds (Cutting, 1976), suggesting that spatial and pitch fusion are governed by a common mechanism, either both occurring at the brainstem level or mediated by similar top-down cues. Certainly, it is possible that temporal variations, intensity differences, and other factors may affect binaural pitch weighting. For instance, higher intensity or greater temporal modulation depth in one ear may shift weighting toward that ear. In addition, electric stimulation was conducted only at high pulse rates probably above the temporal rate pitch limit in CI ears. Thus, fusion pitch averaging was only demonstrated for place pitch coding, and further study is needed to determine if pitch averaging also modulates rate pitch. Further study using a variety of stimuli is needed to understand the mechanisms and factors governing binaural pitch weighting in both normal-hearing and hearing-impaired listeners.