Modulation Enhancement in the Electrical Signal Improves Perception of Interaural Time Differences with Bimodal Stimulation

Abstract

Interaural timing cues are important for sound source localization and for binaural unmasking of speech that is spatially separated from interfering sounds. Users of a cochlear implant (CI) with residual hearing in the non-implanted ear (bimodal listeners) can only make very limited use of interaural timing cues with their clinical devices. Previous studies showed that bimodal listeners can be sensitive to interaural time differences (ITDs) for simple single- and three-channel stimuli. The modulation enhancement strategy (MEnS) was developed to improve the ITD perception of bimodal listeners. It enhances temporal modulations on all stimulated electrodes, synchronously with modulations in the acoustic signal presented to the non-implanted ear, based on measurement of the amplitude peaks occurring at the rate of the fundamental frequency in voiced phonemes. In the first experiment, ITD detection thresholds were measured using the method of constant stimuli for five bimodal listeners for an artificial vowel, processed with either the advanced combination encoder (ACE) strategy or with MEnS. With MEnS, detection thresholds were significantly lower, and for four subjects well within the physically relevant range. In the second experiment, the extent of lateralization was measured in three subjects with both strategies, and ITD sensitivity was determined using an adaptive procedure. All subjects could lateralize sounds based on ITD and sensitivity was significantly better with MEnS than with ACE. The current results indicate that ITD cues can be provided to bimodal listeners with modified sound processing.

INTRODUCTION

For cochlear implant (CI) users with residual hearing in the non-implanted ear, combined electric and acoustic stimulation is a common treatment called bimodal stimulation. With clinical devices, it has been demonstrated that adding a hearing aid in the non-implanted ear improves sound source localization and speech perception, as illustrated in a review by Ching et al. (2007). However, with their clinical devices, most bimodal listeners are unable to use interaural timing cues which, for normal hearing (NH) listeners, are important for sound source localization and related to binaural unmasking of speech in the presence of spatially separated interfering sounds (Akeroyd 2006; Bronkhorst 2000; Colburn et al. 2006).

In our previous studies assessing interaural time difference (ITD) perception with bimodal hearing (Francart et al. 2009, 2011) we have found ITD sensitivity in the envelope of simple synthetic single- and three-channel stimuli. For multiple-channel stimuli, ITD sensitivity was found when the electrical signal contains deep modulations that are consistent across different channels (electrodes). Contrary to NH listeners, bimodal listeners do not seem to be sensitive to ITDs when timing cues are only present in the temporal fine structure on the acoustically stimulated side (Lenssen et al. 2011). When assessing the output of the clinically used advanced combination encoder (ACE) strategy for vowels, it is seen that the presence of deep and synchronous modulations depends on the input level and the number of harmonics that fall within each filter. The latter depends on the interaction between the fundamental frequency (F₀) and the formant structure of the input signal, as well as on the filter bank used. For high stimulation levels, channels tend to saturate, effectively removing most modulations. While this could be counteracted using a fast compression system, such systems can have other undesirable effects. Another problem with clinically used bimodal systems is the synchronization between devices: the CI speech processor and hearing aid (HA) have a different processing delay and the acoustically stimulated signal is delayed while traveling through the middle and inner ear (Francart et al. 2009). Previously, we have demonstrated that bimodal listeners can be sensitive to ITDs in the envelope of simple synthetic single- and three-channel stimuli (Francart et al. 2009, 2011). Therefore, it is reasonable to hypothesize that their inability to use interaural timing cues with clinical devices is due to the way the devices work rather than a fundamental inability to perceive ITDs.

The NH auditory system can make use of both stimulus onset (transient) ITDs and ongoing ITDs in the envelope (Henning 1974). ITDs in the ongoing part are dominant, unless they are ambiguous, in which case the auditory system tends to use onset cues (Buell et al. 2008; Freyman et al. 1997).

Recent studies indicate that the main factors determining envelope ITD sensitivity are temporal properties of the envelope, such as the steepness of the onsets and the duty cycle (Bernstein and Trahiotis 2009, 2010; Ewert et al. 2009; Francart et al. 2012; Laback et al. 2011). Generally, ITD sensitivity improves with increasing onset steepness and increasing dead time, which is the time the envelope remains at the minimum level in every period.

Here, a new speech processing strategy is presented: the modulation enhancement strategy (MEnS). It was designed to improve the ITD perception of bimodal listeners by explicitly modulating the electric stimulation signal. The modulations are determined by the peaks associated with the F₀ in voiced segments of the acoustic signal. MEnS enhances two main features: the shape of the envelope of the electric signal and the temporal accuracy of the onset and the modulations. Its concept is related to that of existing strategies that are designed to improve pitch perception, such as sawtooth modulation (Green et al. 2004), F0Sync (Vandali et al. 2005), frequency modulation (F0mod) (Laneau et al. 2006; Milczynski et al. 2012; 2009), MEM (Vandali et al. 2005), and eTone (Vandali and van Hoesel 2011). The fundamental frequency modulation (F0mod) sound processing strategy was developed to improve pitch perception with cochlear implants. For voiced segments of the input signal, it modulates the amplitude of the electric signal based on the output of a fundamental frequency estimator. An important feature of MEnS is that the modulations are synchronized in time with those in the acoustic signal, which is not the case in pitch-strategies such as F0mod which introduce modulations at the correct rate without maintaining the positions of temporal maxima in the acoustic signal. While MEnS mainly enhances ongoing ITD cues, it also improves the temporal resolution of onset cues.

In a first experiment, we evaluated the overall functionality of the MEnS strategy by measuring the just noticeable difference (JND) in ITD of five bimodal listeners for a vowel processed by either ACE or by MEnS, using a constant stimuli procedure. In condition Ramp, ramping was applied to the beginning and end of the vowel to suppress onset ITD cues and retain ongoing ITD cues. In condition NoRamp, no ramping was applied, such that both onset and ongoing ITD cues were available. In a second experiment, we measured sound image lateralization based on ITD for three subjects. The lateralization task provided parameters to balance the signals in level and time in a formalized way which enabled the subsequent use of an adaptive procedure to determine the JNDs in ITD.

METHODS

Signal Processing

The speech processing strategy was either the clinical ACE strategy (McDermott et al. 1992; Vandali et al. 2000) or the experimental MEnS strategy. The implementation of the ACE strategy was provided by Cochlear in the form of the Nucleus MATLAB Toolbox version 4.31. The general block diagram of the two strategies is shown in Figure 1. In both strategies, the microphone response of the Freedom speech processor was simulated and only direct computer controlled stimulation was used, i.e., signals were never presented in free field. Then the signal was analyzed by a fast Fourier transform (FFT) filter bank, followed by quadrature envelope extraction. All pre-processing options were turned off in the processing.

FIG. 1.

Block diagram of ACE and MEnS strategies.

In the ACE strategy, envelope extraction was followed by maxima selection and channel mapping. In the MEnS strategy, envelope extraction was followed by low-pass filtering and modulation based on peaks detected by the peak detection block. Then, the channels to be stimulated were selected, followed by channel mapping. Note that for MEnS, maxima selection was replaced by channel selection. In the following sections, the different blocks are described in detail.

Microphone Response

The microphone response of the Freedom processor was measured by playing a test signal through a loudspeaker in front of the speech processor mounted on an artificial head, and recording it from the audio-output socket of the processor, which is normally used by parents or clinicians to check the function of the microphone. A corresponding 128-taps FIR filter was designed using an adaptive filter. The same filter was applied in the two signal processing conditions.

Filter Bank And Envelope Extraction

An FFT filter bank and quadrature envelope extraction was used to obtain 22 envelope channels. Swanson (2008) provides a detailed description of this process. The resulting bandpass filter edge frequencies were identical to those used in the subjects’ clinical speech processors. All subsequent processing was block-based with a block size of 128 samples at a sample rate of 16 kHz. For ACE and MEnS, the block rate at the end of the processing chain was 900 and 14,400 Hz, respectively. To this end, the signals were resampled after the FFT filter bank. We did not change the sample rate at the input, as this would have changed the cut-off frequencies of the FFT filter bank.

Maxima Selection and Channel Selection

In the maxima selection block of the ACE strategy, for each block, the eight channels with the largest magnitude were selected for stimulation. The other channel magnitudes in the block were discarded. The resulting electrical pulses were temporally ordered from base to apex. This means that for an impulsive stimulus, the position in time of the first electric pulse was quantized to a multiple of the block rate, which equalled the channel stimulation rate. For the default channel stimulation rate (900 Hz), the corresponding period was 1.1 ms. This is detrimental for onset ITD cues, which range from −700 to 700 μs for realistic input signals. Therefore, in the MEnS strategy, a novel channel selection mechanism was used. This method had higher temporal resolution and selected channels at a block rate of 14,400 Hz, yielding a period of 0.069 ms. Every period, the channel with the highest amplitude was selected for stimulation. After it had been selected, a channel could not be selected for at least 14,400/900 = 16 periods, ensuring a channel stimulation rate of 900 pps, as is used in ACE. After this time, the next pulse is again quantized to a period of 0.069 ms. Note that in Figure 2, for each period onset (vertical dashed lines), the time intervals between subsequent pulses on different channels are shorter for MEnS than for ACE.

FIG. 2.

Beginning of the acoustic stimulus used in condition Ramp (panel A) and electrodograms of the stimulus processed by the ACE and MEnS strategies (panels B and C). Peaks as determined by Praat are indicated by dotted vertical lines. In the electrodograms, the height of each pulse corresponds to its magnitude, i.e., fraction of dynamic range of that particular electrode. Electrodes are numbered from apex to base and different colors were used to facilitate visual discrimination between neighboring electrodes.

Analysis of the electrodograms produced by ACE with maxima selection or with the modified channel selection for several test signals indicated that the average channel stimulation rates were similar, as was the current distribution over the different electrodes. Example electrodograms are shown in Figure 2.

Peak Detection

The open-source Praat program was used for peak detection (Boersma and Weenink 2001). It works by determining the F₀ using an autocorrelation-based method and then finding a local magnitude extremum in each period (1/F₀). For more details, we refer to the Praat documentation. Note that for the synthetic stimulus used in the current study, this step could have been eliminated as the peaks were known a priori. The peaks determined by Praat corresponded to the expected peaks.

Low-Pass Filtering and Modulation

The bandpass-filtered signal envelope was low-pass filtered with an eighth-order Butterworth low-pass filter with a cut-off frequency of 100 Hz. This cut-off frequency was chosen to retain the low envelope modulation frequencies that carry important cues for speech understanding. A stimulation envelope was formed for each channel from the multiplication of the filtered low-frequency envelope of the signal with a predefined modulation waveform, MW(t). The same MW(t) was used for all channels, to synchronize modulations across channels. MW(t) was chosen to provide a steep onset slope at each peak in the acoustic signal and a relatively long dead time before each subsequent peak, in an attempt to maximize ITD sensitivity. In pilot experiments, different modulation shapes were evaluated, but finally, an exponential decay function was selected such that the magnitude decreased from 1 to 0.25 during the first 2 ms. The algorithm also contained a function to switch between modulating and not modulating mode based on an estimate of the probability that the signal in the current block was voiced. For the current study, this function was not used. Since only vowels were used as stimuli, the signal was always modulated.

Channel Mapping

In the final stage, a compressive logarithmic loudness growth function was applied and all signals were mapped between the subjects’ threshold and comfortable levels. All blocks up to the channel mapping stage operated in units of magnitude (M) between 0 and 1. During channel mapping, these magnitudes were converted to current levels based on the threshold and comfort levels the subjects used clinically with their own processor. For a given magnitude M, threshold T, comfort level C, and volume V, the resulting current level was CL = V × M × (C − T) + T. In a clinical processor, the volume is usually set to 100 %. Volume is expressed as a percentage of the dynamic range. In the current study, it was adjusted to obtain a comfortable stimulus level.

Apparatus

The subjects’ own HAs or speech processors were not used. All stimuli were presented under direct computer control using the APEX 3 program developed at ExpORL, KU Leuven (Francart et al. 2008). For acoustic stimulation, we used an RME Multiface II sound card, connected to a single insert phone of type Etymotic ER-3A. As we could not achieve sufficient loudness for subject S02 using the insert phone, a GNResound Viking HA with direct audio input was used. Inverse filtering was used to flatten its frequency response. The filter was determined using a 2-cm³ coupler (ISO 389-2 1994). For electric stimulation, we used the Cochlear NICv2 interface, connected to an L34 experimental processor provided by Cochlear Ltd. The L34 was set up to start stimulating when a trigger pulse was received from the sound card. In this way, synchronous stimulation was achieved with accuracy higher than 200 ns. The relative temporal onset of the acoustic and electric signal was verified using the output of an implant-in-a-box and the electric output of the sound card visualized on an oscilloscope. The insert phone and HA was calibrated using a 2-cm³ coupler conforming to the ISO389-2:1994 standard (ISO 389-2).

Subjects

Five subjects were recruited from the clinical population of the University Hospital Maastricht (azM), and the University Hospital Leuven (UZLeuven). They were selected to have residual hearing thresholds better than 100-dB sound pressure level (SPL) at 1,000 and 2,000 Hz, since we found previously that this was related to good ITD sensitivity (Francart et al. 2009). They were volunteers and signed an informed consent form. This study was approved by the local medical ethical committees.

Various subject information is listed in Table 1. All subjects wore a HA contralaterally to their CI on a daily basis. S02 had an electrode array of the Cochlear Contour type; the other subjects had an array of the Contour Advance type. The clinical processors were of the Freedom type. Their unaided pure-tone audiograms are shown in Figure 3. Subjects S02, S11, and S12 also participated in our previous bimodal ITD studies (Francart et al. 2009, 2011; Lenssen et al. 2011) and were familiar with the tests and procedures. They were among the subjects with relatively good ITD sensitivity, which we found was related with the amount of residual hearing. Subject S12 participated in pilot experiments for the current study. For subjects S21 and S23, the first two test sessions of the current study were used to familiarize them with the experiments. S23 was only available for a limited amount of time, but due to quick familiarization with the procedure and his high performance level, data could be collected in a relative short time. The total number of test sessions per subject for experiment 1, including familiarization and pilot sessions, was 5, 8, 15, 11, and 5 for S02, S11, S12, S21, and S23, respectively. Each session lasted between 90 and 120 min, including breaks. Subjects S12, S21, and S23 also participated in experiment 2. They were again familiarized with the procedures for a minimum of two sessions, leading to a total number of sessions for experiment 2 of 7, 5, and 5 for subjects S12, S21, and S23, respectively.

TABLE 1.

Subject information: Age is in years at the time of testing. CI use is the number of months of implant use at the time of testing. CI side is left (L) or right (R); the HA was on the other side

Subject	Age (y)	CI use (months)	CI side	etiology
S02	69	84	R	Noise exposure
S11	65	60	R	Ménière’s disease
S12	68	40	L	Genetic (DFNA9)
S21	72	28	L	Unknown/previously Ménière’s disease
S23	65	31	L	Slow progressive genetic

FIG. 3.

Pure-tone unaided audiograms for each subject. Note that the vertical axis starts at 50 dB HL.

Stimuli

All stimuli were based on an artificial vowel /a/, generated using a Klatt-synthesizer (Klatt 1980) with formants at 700, 1,220, and 2,600 Hz, with bandwidths of 95, 113, and 199 Hz, respectively, and an F₀ of 100 Hz. In the Ramp condition, the stimulus was sinusoidally ramped during the first and last 80 ms to suppress onset and offset ITD cues. In the NoRamp condition, the stimulus was not ramped.

On the acoustically stimulated side, the artificial vowel was presented unmodified. At the electrically stimulated side, the artificial vowel was either processed according to the ACE strategy or according to the MEnS strategy as described in “Signal Processing”. The onset part of the stimulus is shown in Figure 2.

Note that in a real-time implementation, the signal processing of the electric signal needs a time window that causes the electric signal of ACE and MEnS to lag behind the acoustic signal. This delay will, however, be small compared to the HA processing delay, and can be compensated for by adding an extra delay to the acoustic signal. However, this is not possible for CI users with sufficient residual hearing in the contralateral ear, who do not use a HA.

In experiment 1, the stimulus duration was 1,000 ms; in experiment 2, the stimulus was shortened to 600 ms to reduce the measurement time, but still retaining the ongoing ITD cue. In experiment 2, only the Ramp condition was considered.

The input level to the electric processing was chosen as the value that corresponds to an acoustic input level of 70 dB A to a Freedom speech processor with a sensitivity setting of 12. All other electrical stimulation parameters were identical to those set in the subjects’ clinical devices: a channel stimulation rate of 900 Hz, monopolar (MP1 + 2) stimulation, an inter phase gap of 8 μs and a phase width of 25 μs.

EXPERIMENT 1—LEFT–RIGHT DISCRIMINATION

Procedure

There were four conditions: ACE-Ramp, MEnS-Ramp, ACE-NoRamp, and MEnS-NoRamp. For each condition, at least two measurements were made during test sessions on different days.

The procedures were very similar to those used in our previous bimodal ITD perception studies (Francart et al. 2009, 2011; Lenssen et al. 2011). First, the stimulus was balanced in level between ears. Second, the JND in ITD was determined. These two steps are described below.

Level Balancing

For each subject level balancing was done separately for each of the four conditions: ACE-Ramp, MEnS-Ramp, ACE-NoRamp, and MEnS-NoRamp. The procedure consisted of two steps: first, the levels were set monaurally based on a loudness rating, and then the interaural delay and levels were adjusted as to obtain a centered sound percept. This last step was based on the extent of lateralization; if the stimulus could be steered equally far to either side of the head by varying only ITD, it was considered balanced in level.

First, the acoustic signal was set to a level that was perceived as “comfortable” on a 7-point scale ranging from inaudible to uncomfortably loud, which is also used in the clinic. Then, the electric signal was set separately to a level perceived as comfortable (goed in Dutch). Due to differences between the modalities, physically synchronized signals (e.g., an electrical pulse coinciding in time with an acoustic click at the output of the transducer) do not necessarily lead to synchronous stimulation of the auditory nerves. To compensate for this difference we need to introduce an extra delay in the electric signal path, which will be called D. The exact mechanisms behind this delay are currently unknown, so D was empirically determined for each subject and condition. This was done in an ITD-based lateralization task, assuming that if the levels are balanced, the value of D corresponding to ITD = 0 μs will lead to a centered percept in a NH listener. In previous experiments, D was found to be in the order of 1,500 μs (Francart et al. 2009). D can be interpreted as being due to the traveling wave delay of the acoustic signal in the middle and inner ear, which is not present at the electric side, and potential differences in synaptic delay. The interpretation is complicated by the fact that the exact places of stimulation in the cochleas that lead to lateralization are unknown. This is mainly due to spread of excitation on the electrically stimulated side, and by the unknown neural survival related to the hearing loss on the acoustically stimulated side.

After setting the monaural levels, the stimulus was presented binaurally. Then, the intensity of the acoustic or electric signal was slightly adjusted to achieve a centered sound image for the initial delay D. Then, it was attempted to shift the sound image to either side of the head by changing the interaural delay (to D + δ and D − δ). The subject was asked to describe the extent of laterality by indicating or quantifying the lateral position of the sound image inside the head between their ears. If the maximal extent of laterality that could be achieved by changing the interaural delay was similar for each side of the head, the level was considered balanced. Otherwise, the intensities and D were adjusted and the extent-of-laterality procedure was repeated. Generally, only the level of the electric stimulus was adjusted, unless the subject preferred the level of the acoustic stimulus to be adjusted. Note that the final measurement of D was not made during the balancing procedure. Its exact value was estimated during the JND in ITD measurement.

The following is an example of the steps taken in one run of the balancing procedure for one subject with a CI on the left and a HA on the right. In this example, “el” refers to the electric level in percentage dynamic range, “ac” refers to the acoustic level in dB SPL, and positive interaural delays correspond to the delay of the electric signal.

1. Set monaural levels to “good” on the loudness scale: el = 90 %, ac = 80 dB SPL

2. Present the stimulus binaurally with an interaural delay (D) of 1,500 μs and decrease the electric level for a balanced percept: D = 1,500 μs, el = 85 %, ac = 80 dB SPL

3. Present the stimulus binaurally with an interaural delay of 500 μs (1,500 − 1,000 μs) and 2,500 μs (1,500 + 1,000 μs). For the first stimulus, the sound image is perceived lateralized 100 % to the left and for the second stimulus, it is lateralized 80 % to the right. We therefore decrease the electric level and adjust D accordingly to be able to shift the stimulus equally far to both sides (by finding the interaural delay for a centered percept): D = 1,300 μs, el = 80 %, ac = 80 dB SPL.

4. Present the stimulus binaurally with an interaural delay of 300 μs (1,300 − 1,000 μs) and 2,300 μs (1,300 + 1,000 μs). This leads to laterality judgments of 70 % to the left and 70 % to the right. The stimulus is therefore considered balanced in level.

JND in ITD

To determine the JND in ITD, the psychometric function for ITD detection was estimated using a single-interval, two-alternative-forced-choice constant stimuli procedure. In a single run, a number of ITDs was selected within a certain range, and a stimulus trial containing each ITD was presented three times. The subject had to indicate whether the sound was lateralized to the left or right side. The ITDs presented in a single run were regularly spaced around an initial estimate of D and, in addition to ITDs within the physiologically relevant range, some very large ITDs (up to 1,500 μs off-center) with wider spacing were included to motivate the subject. In the proximity of D, the intervals in a single run were 500, 250, or 100 μs, based on the expected JND. For example, for a subject with D = 1,500 μs and an expected JND of 150 μs, a spacing of 100 μs would be chosen and the ITDs to be presented in a single run could be: 0; 500; 1,000; 1,100; 1,200; 1,300; 1,400; 1,500; 1,600; 1,700; 1,800; 1,900; 2,000; 2,500; and 3,000 μs. In a next run, a new selection of ITDs would be chosen based on the new estimate of D and the JND. In Figure 4, an example psychometric function is shown, which shows the relation between the responses and the ITDs. Note that the reported JNDs are based on a combination of several such functions (see below). The different conditions were tested in random order.

FIG. 4.

Example psychometric function for S23. In this case, the JND in ITD was 391 μs and D was 1,489 μs. The dots indicate measurement points and the line indicates the fitted function. The error bars were determined using the bootstrap method and indicate 68 % confidence intervals.

Psychometric functions were fitted to the results using the psignifit toolbox version 2.5.6 for MATLAB1 which implements the maximum-likelihood method described by Wichmann and Hill (2001). The 68 % confidence intervals around the fitted values were obtained by the BCA bootstrap method implemented by psignifit, based on 1999 simulations. Results of a psychometric function were only regarded as valid if a confidence interval could be calculated by the bootstrap method, and if there were ITDs for which the proportion of left responses was different from 0 or 1. Multiple runs of the same condition (stimulus type) were performed during one test session, and the results of those runs were merged into a single psychometric function.

From each psychometric function, the JND in ITD was determined as half the difference in ITD between the 15 and the 85 % point in the curve. This leads to the JND defined as the difference in ITD that results in a 70.7 % change in performance. The interaural delay that led to a centered percept (D) was determined from the interaural delay at the 50 % point. If multiple psychometric functions were determined for the same subject and condition (e.g., during different test sessions), the global JND was calculated by shifting all functions such that their estimated 50 % point was at 0 μs and then performing a new estimation of the function (and its corresponding JND) formed by the combination of all (shifted) data points. Data from different test sessions were not simply merged into a psychometric function because differences in level balance, for example caused by fluctuations in residual hearing, or an unstable internal reference for the perceptual center, may have led to potentially small differences in D. Merging data with different D values would lead to an apparent increase in JND due to a reduction in the slope of the function. Generally, the difference in estimated 50 % point for different measurements of the same condition was in the order of one JND or smaller. The reported JNDs in ITD are each based on a total number of trials ranging from 92 to 255, with a median of 144 trials.

Results

JNDs in ITD for each condition and subject are shown in Figure 5. JNDs larger than 2,000 μs are indicated by full-height bars. A Wilcoxon signed-rank test was performed to test the effect of algorithm (ACE or MEnS) and the effect of ramping (NoRamp or Ramp). The test showed overall significantly higher performance for MEnS than for ACE, (z = −2.023, p = 0.043) and no effect of ramping (z = −0.944, p = 0.345). When considering individual results, subjects S02, S12, and S23 were sensitive to ITD with thresholds well below the physical maximum (700 μs) in all conditions. Subject S21 had high thresholds for ACE and much lower thresholds for MEnS. Subject S11 was not sensitive to ITD for ACE and had relatively high thresholds for MEnS.

FIG. 5.

Measured JNDs in ITD for each subject and condition, with the vertical axes displayed on log scale. The error bars indicate 68 % confidence intervals determined using a bootstrap method. Full-height bars indicated with >> mean that JNDs were found larger than 5,000 μs or immeasurable.

The values found for D ranged from 1,034 to 3,259 μs, with a median of 1,994 μs, which is comparable to previous studies (Francart et al. 2009, 2011).

Subjectively, the subjects generally did not perceive major differences in sound quality between ACE and MEnS. Two subjects (S12 and S23) reported that binaural fusion was better for MEnS.

Experiment 2—Lateralization

The procedures used in experiment 1 could have been sensitive to biases due to the non-formalized balancing procedure and the use of a single-interval constant stimuli procedure to determine ITD sensitivity. Furthermore, we were interested in the subjects’ ability to lateralize the stimuli as opposed to just discriminate. Therefore, with a subset of three listeners, we conducted a second experiment that consisted of: (1) a formalized balancing procedure based on the measurement of extent of lateralization and (2) a two-interval 2AFC adaptive procedure to determine the JND in ITD.

Procedure

Experiment 2 consisted of three main steps: (1) brief level balancing, (2) a lateralization task, and (3) an ITD discrimination task. In experiment 1, the level balance and D were estimated in a lengthy balancing task. Here, an initial rough balancing was performed in step 1. This balance was refined by measuring sound image lateralization in step 2. Lateral position shifts due to a combination of interaural level difference (ILD) and ITD were estimated within one run. The resulting estimates of the level balance (L, expressed in percentage volume adjustment of the electric stimulus) and D were used to generate stimuli for an adaptive ITD discrimination procedure to measure the JND in ITD in step 3. The separate steps are explained in more detail below.

Level Balancing

Level balancing was performed to roughly balance the signals between the ears. The procedure was a shortened version of the procedure used in experiment 1. We obtained an initial level balance L and an estimate of the ITD range R leading to a sound image perceived on the extreme left and on the extreme right. In the lateralization experiment, the balancing was further refined. In the remainder of this experiment, the acoustic level was kept fixed and the same for both strategies. When necessary, the electric level was varied to balance the signals between the ears.

Sound Image Lateralization

Subjects were asked to indicate the lateral position of the sound they heard on a 15-cm long line depicted on a computer screen. The line represented the space between their ears. Vertical line markers indicated the center and the outer limits. The subjects used the mouse to click on the horizontal position on the line to indicate the lateral position of the sound they heard. In every trial, the same stimulus was presented twice, separated by 500 ms of silence, in order to minimize the influence of previous trials on the response. Lateral position judgments were scaled between 0 (completely left) and 1 (completely right) and plotted against ITD. Figure 6 shows an example of results obtained with this procedure for subject S12. In the figure, the ITD is shown on the horizontal axis and lateral position judgments on the vertical axis. ITDs are given as the delay of the acoustic signal with respect to the electric signal in microseconds. These lateral position functions were obtained in two steps:

1. Level balance (L): This part of the procedure aimed to find the level balance L for which (1) the lateralization function changed symmetrically around the perceptive midline with changing ITD and (2) the total extent of lateralization was maximal. This is based on the observation of Sayers (1964) and Domnitz (1973), that for normal hearing listeners, a non-zero ILD combined with an ITD caused asymmetries in the lateral position function around the midline. As an example, consider panel A and B of Figure 6. Separate lines represent functions fitted to responses at different ILDs. In panel A, for an ILD of −10 %, the function was shifted towards the left of the midline. For an ILD of −5 %, the solid line in the same figure, the lateral position function was symmetric around the midline, and the total extent the largest.

   ITDs were chosen regularly in the range D − R to D + R (obtained from the rough level balancing), at 1,000-μs intervals, to cover the range from an extreme left to an extreme right percept. ILDs were chosen as L + 5 % DR, L, and L − 5 % DR, with DR as the electric dynamic range of the subject. Within one run, each stimulus was presented three times, in random order. One run with n ITDs thus contained a total of n ITDs × 3 ILDs × 2 strategies × 3 repetitions = n × 18 trials. A run was repeated at least two times. If no symmetric lateralization function was found, another ILD was tested. In the end, results obtained for the same ILD but in separate runs were combined. Responses were normalized for each subject. Responses were normalized by subtracting 0.5 and then dividing by the 95th percentile of the absolute response values for the two strategies combined. Finally, 0.5 was added to shift the functions back to values between 0 and 1. Logistic functions were fitted to the lateralization responses for each ILD as a function of ITD, with a non-linear least squares method provided by MATLAB. The best ILD was selected based on the following criteria for the lateralization function:

   • Symmetry: the difference between maximal right (R_max) and left (L_max) extent of lateralization was less than 10 % of the total range: |R_max − 0.5| − |L_max − 0.5| < 0.1
   • Maximal extent of lateralization: arg max In panels A and B of Figure 6 for subject S12, ILD = −5 % was selected. The values for symmetry were 0.06 and 0.1, and for the maximal extent, 0.67 and 0.58 for ACE and MEnS, respectively.

2. D: In this part of the procedure, the lateral position task was repeated at ILD = L with a smaller step size in ITD in order to obtain an estimate of D. In a first run, the step size was 500 μs and in a second run, 250 μs, with ITDs evenly distributed around the current estimate of D. Again, both strategies were tested within a single run with randomly interleaved trials. Multiple runs were performed and combined into one graph. Then, D was determined at the ITD at the intercept of the fitted line with a response of 0.5, corresponding to a perceptually centered image position.

FIG. 6.

Logistic functions fitted to lateral position judgements for ACE and MEnS for S12 (panels A, B). Different lines represent different ILDs, the solid line represents the ILD (L) that was selected for further testing. Lateral position judgements for ACE (C) and MEnS (D) measured with the levels fixed at L for S12. D was estimated at a response of 0.5, corresponding to a centered lateral position. Error bars represent 95 % confidence intervals on the fit.

JND in ITD

In this step, a 1-up-2-down adaptive staircase procedure was used to determine the JND in ITD at 70.7 % correct. A trial consisted of two intervals with opposite ITDs. One interval contained an ITD of D + δ and the other contained an ITD of D − δ. Subjects were asked to respond to whether the second stimulus was perceived to the left or to the right of the first, irrespective of the absolute location of both sounds. The start value was δ = 2,000 μs. The initial step size was a factor of 1.22, and decreased to a factor of 1.11 after two reversals. The procedure was stopped after 12 reversals. The geometric mean of the last 10 reversals was used to determine the JND in ITD. This value was multiplied by two to obtain the final JND in ITD, since in each trial the ITD difference between the intervals was 2δ. Each subject performed three runs per condition.

Results

All three subjects were able to indicate the lateral position perceived due to ITDs. In Figure 6, an example result of the lateral position judgment task is shown for subject S12. In panels A and B, ILD = −5 % yielded the most symmetric lateral position judgments around the midline. In the second step of the lateral position task, D was determined from the logistic fit to the lateral position curves, shown in panels B and C of Figure 6 for subject S12, and in Figure 7 for subjects S21 and S23. For both strategies, a similar extent of lateralization was obtained. As shown in this figure, the subjects were able to distinguish differences in lateral position between stimuli with different ITDs. Note that when considering lateralization performance, ITDs outside the physiologically relevant range should be disregarded, since they cannot be related to a physically realistic cue. Pearson correlation coefficients were calculated between mean responses and the ITD for ITDs within a range of D ± 800 μs (see Table 2). Correlations were significant at the 0.05 level for all subjects with MEnS, and for one subject (S12) with ACE. The values of D estimated for the three subjects were in the range of 1,900 to 3,225 μs with a median of 2,800 for ACE and within a range of 2,000 to 3,200 μs with a median of 2,588 for MEnS, which is comparable to the values of D found in experiment 1. The values of D estimated for the three subjects were in the range of 1,900 to 3,225 μs with a median of 2,800 for ACE and within a range of 2,000 to 3,200 μs with a median of 2,588 for MEnS, which is comparable to the values of D found in experiment 1.

FIG. 7.

Results of the lateralization task for subjects S21 (A, B) and S23 (panels C, D). The curves show a logistic function fit to the responses (indicated by the dots). D was estimated at a response of 0.5, corresponding to a centered lateral position. Error bars represent 95 % confidence intervals on the fit. Note that when considering lateralization performance, ITDs outside the physiologically relevant range should be disregarded.

TABLE 2.

Correlation coefficients ρ and p values for correlations calculated between the mean responses and the ITD for ITDs within the range D ± 800 μs

Subject	ρ ACE	p ACE	ρ MEnS	p MEnS
S12	−0.91	0.012	−0.998	<<0.01
S21	−0.67	0.14	−0.91	0.012
S23	−0.92	0.25	−0.97	0.032

In Figure 8 JNDs in ITD obtained with the adaptive procedure are shown. With MEnS, thresholds were a factor of 2.3, 2.6, and 2.7 smaller than with ACE for S12, S21, and S23, respectively. For subjects S12 and S23, thresholds were below 200 μs. Before statistical analysis, data were logarithmically transformed to correct for a positive skew. A repeated-measures ANOVA with factors repetition and algorithm showed that thresholds were significantly lower with MEnS than with ACE (F (1, 1) = 218, p < 0.01), with no effect of repetition (F (1, 2) = 2.18, p = 0.23).

FIG. 8.

RIGHT JNDs in ITD obtained with the staircase procedure for three subjects (ACE-/MEnS) with error bars representing the standard deviation of the mean of separate measurements. The vertical axis is displayed on log scale.

DISCUSSION

In experiment 1, JNDs in ITD were determined for five bimodal listeners using an artificial vowel processed by either ACE or MEnS, in a condition with and without onset ITD cues. MEnS improved ITD sensitivity significantly. Two subjects were not sensitive to ITD in condition ACE-Ramp and had high thresholds in condition ACE-NoRamp.

Three out of five subjects were able to use ongoing ITD cues with ACE. They probably used the envelope modulations present in some of the channels. Unfortunately, these cues would not be consistent or usable in a real-life scenario. At the end of the processing chain, ACE runs at a block rate equal to the channel stimulation rate, which is usually around 900 Hz. This means that the position of each block is quantized to the corresponding period, i.e., 1.1 ms, so if the peaks in the acoustic signal are not provided at a regular rate which is a submultiple of the channel stimulation rate (such as the 100 Hz F₀ of the artificial vowel used in the current study), ongoing cues would be severely distorted. For this reason, the novel channel selection mechanism as described in “Maxima Selection and Channel Selection” was employed for MEnS. As an illustration, in Figure 9, electrodograms are shown resulting after processing of a recorded vowel /a/, cut from the word “paat,” with different strategies. The envelope peaks in ACE now do not coincide with the envelope peaks in the acoustic signal as they did for the synthetic vowel with a constant 100 Hz F₀. For CIS, in the non-saturated channels, the envelope shows modulations, but they are not consistent with the peaks in the acoustic signal. Similarly, for F0mod, the peaks in each cycle do not correspond to the peaks in the acoustic signal. With MEnS, modulations are enhanced at the peaks detected at the F₀ of the acoustic signal. As can be seen in panel E of Figure 9, this leads to synchronous modulations between the acoustic and electric signal.

FIG. 9.

Vowel /a/ cut from the word “paat” processed by CIS with eight channels (B), ACE (C) and F0mod (D), and MEnS (E). The vertical dashed lines indicate peaks determined by Praat.

The enhancement of the envelope performed in MEnS is a combination of the more precise timing of the modulation onsets and the enhanced modulation shape which ensures dead time and a steep onset slope. The improvement seen in the current result might be due to a combination of these changes, the current data do not provide information on the effect of either factor separately. Considering the use of onset versus ongoing cues, as can be seen in Figure 2, MEnS mainly enhances ongoing cues. Every modulation cycle, there was a steep slope in level and dead time between these peaks. MEnS also enhances the stimulus onset cue. While for ACE there was a gradual increase in current at the onset of the stimulus, for MEnS there was a steep increase. There was a tendency for better performance in condition MEnS-NoRamp compared to MEnS-Ramp for four subjects, indicating a benefit provided by the enhanced onset cue, but this was not significant. Therefore, the benefit of the enhanced onset cue could not be confirmed.

The subjects showed good sensitivity in condition MEnS-Ramp, where only very limited onset cues were available, so they were able to use ongoing cues. Buell et al. (2008) found that sensitivity to ongoing cues was higher than sensitivity to onset cues and that ongoing cues dominate onset cues. From the electrodogram for MEnS in Figure 2, it might be inferred that there could have been a residual onset cue in the Ramp condition. However, on the one hand, the long Ramp in the acoustic signal would have prevented its use (Kunov and Abel 1981) and on the other hand, if, due to its low stimulation level, the first period would have been perceived at the acoustic side but not at the electric side, this would have led to a delay shift of a whole period (10 ms), which would have been immediately obvious from the measured D value.

As usual, for CI listeners, there were large inter-subject differences. S11 had higher ITD thresholds than the other subjects in conditions MEnS-Ramp and MEnS-NoRamp. This subject was also tested in our previous bimodal ITD perception studies (Francart et al. 2009, 2011) and also showed relatively poor performance there. The reasons for these inter-subject differences are unknown. We could not find a correlation with electrode array type, threshold and comfort levels, residual hearing, or etiology. S11’s lower performance might be related to his Ménière’s disease, the effect of which was reflected in fluctuations in residual hearing thresholds between test sessions.

In experiment 2, all three tested subjects were able to lateralize the stimuli based on ITD with MEnS. Although the lateral extent was limited and never covered the full scale, significant correlations were found between the responses and the ITD for all subjects with MEnS, and for one subject with ACE. The lateral image position task was used to estimate L and D. This enabled the use of a 2I2AFC adaptive staircase procedure to determine the JND in ITD. While the estimate of D had confidence intervals in the order of maximally 500 μs to either side, this probably did not affect the sensitivity estimate from the adaptive procedure: Domnitz (1973) showed that with an ITD offset up to 400 μs to either side, sensitivity was the same as when no ITD offset was present. Performance was significantly higher with MEnS than with ACE. Thresholds were between 2.3 and 2.7 times lower with MEnS than with ACE and below 200 μs for S12 and S23. Despite procedural differences, experiments 1 and 2 had very similar outcomes. It is hard to compare the obtained thresholds directly with those from other studies because of the differences in stimuli and procedures. Qualitatively, our results correspond to the results found in the literature for perception of ITDs in the envelope. The ITD thresholds for MEnS in experiment 1 ranged from 166 to 374 μs with a median of 244 (except for S11), and were in the same order as the thresholds found in experiment 2, where they ranged from 145 to 380 with a median of 182 μs. This is comparable to the best thresholds found in our previous studies with single and 3-channel stimuli (Francart et al. 2009, 2011). The thresholds are also comparable to thresholds found for the best bilateral CI listeners with single-channel stimuli (Laback et al. 2007, 2004; Lawson et al. 1998; Litovsky et al. 2010, 2012; Long et al. 2003; Majdak et al. 2006; Senn et al. 2005; van Hoesel 2004, 2007; van Hoesel and Tyler 2003). That sensitivity with multiple-channel stimuli is comparable to performance with single-channel stimuli is unlike performance often reported for NH listeners, for whom sensitivity increases with increasing bandwidth. This could be due to a detrimental effect caused by the large spread of excitation in electric stimulation or the widened auditory filters in impaired acoustic hearing. We are not aware of any studies that investigated bilateral CI ITD thresholds with multi-channel stimuli under direct computer control. With clinical processors, however, performance is generally poorer than in the single-channel studies (Grantham et al. 2008; Laback et al. 2004; Senn et al. 2005). For NH listeners, Akeroyd (2003) measured JNDs in ITD for full-bandwidth vowel sounds (thus containing temporal fine structure ITD cues), and obtained values in the order of 40 μs. However, temporal fine structure cues are not available to bimodal listeners (Lenssen et al. 2011). NH listeners show lower sensitivity when no temporal fine structure cues are available. For example, Bernstein and Trahiotis (2002) measured JNDs in ITD for transposed tones (high-frequency tones, modulated with a half-wave rectified low-frequency sinusoid) and found thresholds in the order of 100 μs for a modulation frequency of 100 Hz, which corresponds to the best JNDs found with our bimodal listeners.

Issues still to be investigated in the development of this strategy for bimodal stimulation include (1) designing a method to balance loudness in real time in order to preserve the perceptual equivalent of ILDs (Francart and McDermott 2012a, b), (2) investigating the synchronization between the acoustic and electric stimuli for unvoiced sounds, (3) determining peaks in real time, (4) determining whether it is better to enhance the modulations of all channels synchronously or only a selection, (5) estimating the appropriate D for each subject, and (6) optimizing the different parameters described in section “Signal Processing.” Furthermore, the effect of enhancing modulations on speech perception and sound quality needs to be investigated. In studies with pitch-enhancement algorithms that manipulate the electrical signal in a similar way as MEnS, no adverse effects on speech perception were found (Laneau et al. 2006; Milczynski et al. 2009, 2012; Vandali et al. 2005).

To our knowledge, this is the first time that ITD sensitivity has been shown for bimodal listeners with a speech token (vowel sound) processed through a speech processing strategy in a controlled setup, as opposed to simple single-channel stimuli. Furthermore, in experiment 2, lateralization based on ITD was measured for the first time with bimodal stimulation. Although conclusions need to be carefully drawn given the limited sample size and the use of carefully laboratory-controlled stimuli, the perceptual results are promising, as JNDs were often within the physiologically relevant range (approximately −700 to 700 μs) in both experiments. For two subjects, the change from ACE to MEnS meant that their JNDs in ITD dropped into this range. Potentially, the same effect may be achieved with bilateral CIs, with MEnS applied at two sides. The current positive results mean that the concept of enhancing modulations at the CI side is promising for application in a binaural strategy for bimodal stimulation.

CONCLUSIONS

The MEnS speech processing strategy was developed to improve ITD perception for bimodal listeners by emphasizing modulations in the electrically stimulated signal. In two experiments, JNDs in ITD were determined for five bimodal listeners using an artificial vowel processed by either ACE or MEnS, with or without onset ramping. All subjects were sensitive to ITD with MEnS, and only three out of five were sensitive to ITD with ACE. With MEnS, performance was significantly better for all subjects, with and without onset ramping. Additionally, with MEnS three out of the three subjects that participated in experiment 2 showed the ability to lateralize the stimuli based on ITD.

List of acronyms

CI

Cochlear implant

F0

Fundamental frequency

HA

Hearing aid

ILD

Interaural level difference

ITD

Interaural time difference

JND

Just noticeable difference

MEnS

Modulation enhancement strategy

NH

Normal hearing