Pitch contour identification with combined place and temporal cues using cochlear implants

Abstract

This study investigated the integration of place- and temporal-pitch cues in pitch contour identification (PCI), in which cochlear implant (CI) users were asked to judge the overall pitch-change direction of stimuli. Falling and rising pitch contours were created either by continuously steering current between adjacent electrodes (place pitch), by continuously changing amplitude modulation (AM) frequency (temporal pitch), or both. The percentage of rising responses was recorded as a function of current steering or AM frequency change, with single or combined pitch cues. A significant correlation was found between subjects’ sensitivity to current steering and AM frequency change. The integration of place- and temporal-pitch cues was most effective when the two cues were similarly discriminable in isolation. Adding the other (place or temporal) pitch cues shifted the temporal- or place-pitch psychometric functions horizontally without changing the slopes. PCI was significantly better with consistent place- and temporal-pitch cues than with inconsistent cues. PCI with single cues and integration of pitch cues were similar on different electrodes. The results suggest that CI users effectively integrate place- and temporal-pitch cues in relative pitch perception tasks. Current steering and AM frequency change should be coordinated to better transmit dynamic pitch information to CI users.

INTRODUCTION

Although pitch perception has been studied extensively in normal-hearing (NH) listeners, its mechanism remains a topic of debate. Place and temporal coding are the two most widely accepted mechanisms, both supported by physiological and psychophysical evidence (e.g., Oxenham, 2008). When a pure tone enters the normal auditory system, its pitch is represented by both the place of maximal excitation along the basilar membrane (i.e., the tonotopic frequency-to-place map) and by the timing of auditory nerve firing (i.e., the phase locking phenomenon; e.g., Johnson, 1980). Voiced speech segments and music sounds mostly consist of harmonic complex tones with pitch corresponding to the fundamental frequency (F0). In the excitation pattern of a harmonic complex tone along the basilar membrane, resolved harmonics produce a series of peaks, and the harmonic relationship between these peaks provides place cues for F0 calculation (e.g., Goldstein, 1973). On the other hand, temporal cues such as the fine structure of resolved harmonics and the envelope of unresolved harmonics can also be used to determine the periodicity and F0 of a harmonic complex tone (e.g., Meddis and O’Mard, 1997). Each model (place or temporal) can only explain some but not all aspects of pitch perception, suggesting that in the normal auditory system, pitch may be perceived with combined place and temporal cues. For example, Oxenham et al. (2004) found that in order to elicit robust pitch perception, temporal information should be presented at the correct tonotopic location along the basilar membrane.

Compared to NH listeners, cochlear implant (CI) users have much poorer pitch perception, which adversely affects speech recognition in a competing talker background, speech prosody identification, and music listening (e.g., Stickney et al., 2004; Luo et al., 2007, 2008; McDermott, 2004). While place and temporal cues usually covary with each other in acoustic hearing, they can be manipulated independently to encode pitch for CI users. The perception elicited by stimulation of different electrodes is often described as place pitch. In addition to physical electrodes, locations between adjacent electrodes [i.e., virtual channels (VCs)] can be stimulated using current steering, to increase the number of place-pitch percepts (e.g., Donaldson et al., 2005; Landsberger and Srinivasan, 2009). CI users also perceive temporal-pitch changes from different stimulation rates or amplitude modulation (AM) frequencies. However, temporal-pitch perception typically saturates around 300 Hz for CI users (e.g., Zeng, 2002), much lower than the upper limit of phase locking in NH listeners (up to 2000 Hz; e.g., Johnson, 1980). This is likely due to the fact that auditory nerve responses to electric stimulation with CI have different properties (e.g., lack of stochastic activity or traveling-wave delay across nerve fibers) than those in acoustic hearing (Carlyon et al., 2010). Moreover, in current CI processing, no effort is made to present temporal information at the correct tonotopic location or to deliver consistent place- and temporal-pitch cues.

The conclusion that place and temporal cues are perceived independently with CI has been favored in the literature. Tong et al. (1983) asked a multichannel CI user to judge the (dis)similarity between stimuli with different combinations of stimulation rate and electrode position. Multidimensional scaling analysis on the subject’s (dis)similarity ratings revealed that these stimuli were perceived along two dimensions, one related to stimulation rate and the other related to electrode position. McKay et al. (2000) measured CI users’ discrimination of stimuli with combined place and rate changes. The selected place and rate changes were discriminated at a performance level of d’ = 1 when presented alone. Sequential dual-electrode stimuli were used to create small changes in place-of-excitation between adjacent electrodes. It was found that although stimuli with combined cues were better discriminated than those with single cues, discrimination was similar with consistent or inconsistent cues (in terms of the direction of pitch change), suggesting that place and rate changes were used as independent cues in such a same-different task. Although place and rate are separate dimensions in CI perception, both are described as pitch. When CI users are asked to rank or scale the pitches associated with multiple stimulation rates on various electrodes, there is a strong overlap across electrodes and rates in the resultant pitch structure (Zeng, 2002; Stohl et al., 2008). For example, a lower stimulation rate on a more basal electrode may sound similar in pitch to a higher stimulation rate on a more apical electrode, which indicates that place and rate changes are integrated to generate an overall pitch percept. Thus, the interaction between place- and temporal-pitch cues may be more easily observed in a pitch-specific task than in a same-different task.

Most studies on CI pitch perception utilized steady-state stimuli to measure pitch ranking and discrimination. However, speech and music signals are made up of dynamic changes of frequency components over time (e.g., fundamental and formant frequencies). As such, a more relevant task to speech and musical perception is to identify the direction of dynamic pitch change over the duration of a syllable or a sentence, or from one music note to the next [i.e., pitch contour identification (PCI)]. In a recent study, Luo et al. (2010b) investigated the feasibility of pitch contour coding using current steering in CI. Various pitch contours were created by continuously shifting the excitation peak from one electrode to the next over a fixed time period. CI users were able to correctly identify basic pitch contours (e.g., rising, falling, and flat) using time-varying place-pitch cues as long as the contour was 300 ms or longer. These encouraging PCI results suggest that current steering can be used to efficiently encode frequency changes in speech and music (e.g., speech intonations, formant transitions, and music melodies) for CI users. However, speech and tone recognition does not improve (as expected) with the HiRes-120 strategy, which uses current steering to encode spectral peaks within individual frequency bands (Firszt et al., 2009; Han et al., 2009). Only place-pitch contours were tested in Luo et al. (2010b), while both the excitation peak and the temporal pattern of electric stimulation may vary simultaneously in multiple channels of the HiRes-120 strategy. To create the optimal pitch contour coding strategy, it is important to understand CI users’ ability to integrate place- and temporal-pitch cues in the PCI task.

This issue has been briefly investigated by Tong et al. (1983). In their experiment, 300-ms stimuli with continuously increasing or decreasing pulse rates were delivered to a single electrode (apical or basal), or to multiple electrodes in succession (moving from the apical to the basal electrode or vice versa). The task was to identify the stimuli as “questions” or “statements,” presumably based on whether the perceived pitch was rising or falling. Although the subject’s intonation identification was significantly affected by the direction and degree of pulse rate change, the effect of electrode condition (i.e., single or multiple electrodes and the stimulation order of multiple electrodes) and the interaction between electrode condition and pulse rate change were not significant. However, one should be cautious when interpreting these preliminary data. First, the single-subject design in Tong et al. (1983) limits the generalization of their findings. Second, they only tested a few stimuli with large (thus well perceived) changes in pulse rate (e.g., from 80 to 160 Hz) and electrode position (e.g., from electrode 4 to 8 on a 10-electrode array—a distance of around 6 mm). A larger variety of place and rate changes, which result in chance to perfect discrimination when presented alone, may be necessary to fully explore the interaction between place and temporal cues. Third, in Tong et al. (1983), pulse rate varied continuously while electrode position varied discretely from one electrode to the next and so on. The discrete place-pitch changes may not be optimally combined with the continuous temporal-pitch changes for the intonation identification task. Finally, Tong et al. (1983) analyzed their results based on the stimulation order of multiple electrodes (i.e., the direction of place-pitch change). However, for each electrode condition (e.g., moving from electrode 4 to 8), half of the stimuli had consistent temporal-pitch changes (e.g., with increasing pulse rates), while the other half had inconsistent temporal-pitch changes (e.g., with decreasing pulse rates). If consistent place- and temporal-pitch changes would result in better intonation identification than inconsistent changes, an analysis that considers the consistency between the two cues may be more appropriate.

We revisited the issue of integration of place- and temporal-pitch cues with two primary motivations. The first motivation was to investigate CI users’ sensitivity to dynamic place- or temporal-pitch cues. Falling and rising pitch contours were created either by steering current between adjacent electrodes or by changing AM frequencies, which yielded continuous place- and temporal-pitch changes, respectively. Psychometric functions relating the percentage of rising responses to current steering or AM frequency change were recorded. Various amounts of current steering and AM frequency change were tested, resulting in chance to perfect PCI performance. The second motivation stemmed from our speculation that the orthogonality of place- and temporal-pitch cues reported by Tong et al. (1983) was due to the high salience of both cues. We hypothesize that when the place- and temporal-pitch cues are not highly salient, they may interact with each other (i.e., not be orthogonal) in the pitch-specific PCI task, similar to the observations in pitch-ranking studies (Zeng, 2002; Stohl et al., 2008). Further, the interaction may be particularly strong when the two pitch cues have similar perceptual salience in isolation. If one cue is much weaker than the other, then it may not affect the perception of the stronger cue.

In the present study, we sought to determine if PCI performance would be improved through the use of consistent place and temporal cues rather than with single cues alone and, in turn, rather than with inconsistent cues. To fully explore the integration of place- and temporal-pitch cues, the amount of current steering identified with d’ = 1 in isolation was added to the temporal-pitch contour stimuli with various amounts of AM frequency change. Similarly, the amount of AM frequency change identified with d’ = 1 in isolation was added to the place-pitch contour stimuli with various amounts of current steering. The experiments were repeated on both apical and medial electrode pairs. Although static pitch discrimination and pitch contour identification with current steering appears to be similar across different electrode pairs (Donaldson et al., 2005; Luo et al., 2010b), the electrode location may have an effect on AM frequency discrimination (Chatterjee and Peng, 2008). Besides, stimuli on the medial electrode pair had higher place pitch (or sharper sound quality) than those on the apical electrode pair, while stimuli in both locations had similar low-frequency temporal pitch because of the same AM frequency range. PCI scores on the two electrode pairs were compared to test the hypothesis that the integration of place- and temporal-pitch cues is less effective on the medial electrode pair, where the place- and temporal-pitch percepts are less similar to each other.

METHODS

Subjects

Five postlingually deafened adult CI users participated in this study. They all used an Advanced Bionics implant system (either Clarion CII or HiRes 90 K), which supports current steering with simultaneous dual-electrode stimuli. The demographic details of subjects are shown in Table TABLE I.. Subject P1 was tested at Purdue University, while the other subjects were tested at the House Research Institute. This study was reviewed and approved by the local IRB committees at both institutions. All subjects provided informed consent and received compensation for their participation in this study.

TABLE I.

Subject demographic details.

Subject	Age	Gender	Etiology	Prosthesis	Strategy	Years with prosthesis
C1	77	M	Unknown	CII	HiRes-P w/Fidelity 120	7
C4	63	F	Cochlear otosclerosis	HiRes90K	HiRes-S	4
C7	61	F	Fever and streptomycin	HiRes90K	HiRes-P w/Fidelity 120	3
C19	61	M	Auto-immune	CII	HiRes-S w/Fidelity 120	11
P1	82	F	Unknown	HiRes90K	HiRes-P w/Fidelity 120	2

Stimuli

All stimuli were 300-ms, 1000-pulses/s (pps), 42-μs/phase biphasic pulse trains, which were delivered using the Bionic Ear Data Collection System (Advanced Bionics, 2005) in monopolar mode to the apical electrode pair 2-3 or the medial electrode pair 7-8. Due to time limitation, subject C4 was only tested on electrode pair 2-3. Pilot studies for a previous experiment (Landsberger and Srinivasan, 2009) had shown that in tripolar mode, thresholds were significantly elevated and loudness growth was poor for electrode 7 for subject C1. This suggests that electrode 7 primarily stimulates a region with a poor electrode-neuron interface (e.g., a cochlear dead region or an ossified section of the cochlea) as argued by both Bierer and Faulkner (2010) and Goldwyn et al. (2010). Therefore, for subject C1, electrode pair 8-9 was used instead of electrode pair 7-8.

Current levels on individual electrodes were adjusted so that single-electrode stimuli were equally loud at a comfortable level. Three types of stimuli were tested: place-pitch contour stimuli, temporal-pitch contour stimuli, and place- and temporal-pitch contour stimuli. The stimuli with only place or temporal cues were described as follows, while those with combined place and temporal cues were described in the Sec. 2C, because the combined stimulation parameters were derived from perceptual results with single cues.

Place-pitch contours

Using the same method as in Luo et al. (2010b), rising and falling pitch contours were created with current steering. In this place-coding strategy, two electrodes in each pair were stimulated at the same time and the proportion of current delivered to the basal electrode (α) varied linearly from pulse to pulse over the 300-ms duration of the stimulus. The pulse at the midpoint in time was always presented with α = 0.5 (i.e., to the middle VC between the two electrodes). The starting and ending α for each place-pitch contour are shown in Table TABLE II.. These α ranges were selected because they resulted in chance to perfect PCI performance (see Sec. 3). For each α range (e.g., 0.2–0.8), there was a rising (with gradually increasing α) and a falling pitch contour (with gradually decreasing α), giving a total of 12 different place-pitch contours. The current levels of simultaneous dual-electrode stimuli were linearly interpolated between those of single-electrode stimuli, to minimize loudness variations within each pitch contour.

TABLE II.

Starting and ending α for each place-pitch contour.

Rising pitch contours			Falling pitch contours
Starting α	Ending α	Δα	Starting α	Ending α	Δα
0.20	0.80	+0.6	0.80	0.20	−0.6
0.25	0.75	+0.5	0.75	0.25	−0.5
0.30	0.70	+0.4	0.70	0.30	−0.4
0.35	0.65	+0.3	0.65	0.35	−0.3
0.40	0.60	+0.2	0.60	0.40	−0.2
0.45	0.55	+0.1	0.55	0.45	−0.1

Temporal-pitch contours

In the temporal-coding strategy, rising and falling pitch contours were created with time-varying AM frequencies. The sinusoidal AM (SAM) super-imposed on the stimuli was in sine phase and had an AM depth of 30%, which was adequate to allow for perception of AM frequency sweeps in the 150-Hz AM range (see Sec. 3). Over the 300-ms duration of the stimulus, the AM frequency varied linearly and was always 150 Hz at the midpoint in time. The starting and ending AM frequencies for each temporal-pitch contour are shown in Table TABLE III.. The maximum AM frequency used (176 Hz) was well under the typical upper limit of temporal-pitch perception with CI (∼300 Hz). The AM frequency ranges used in this experiment resulted in chance to perfect PCI performance (see Sec. 3). Also note that, the 1000-pps carrier pulse rate was more than 4 times the highest tested AM frequency (176 Hz), thus was high enough to avoid perceptual effects of sub-sampling (McKay et al., 1994). For each AM frequency range (e.g., 123–176 Hz), there was a rising (with gradually increasing AM frequencies) and a falling pitch contour (with gradually decreasing AM frequencies), giving a total of 12 different temporal-pitch contours. According to the AM loudness model of McKay and Henshall (2010), the AM frequency changes should have had little effect on the loudness of AM stimuli within each pitch contour.

TABLE III.

Starting and ending AM frequencies (in Hz) for each temporal-pitch contour.

Rising pitch contours			Falling pitch contours
Starting F	Ending F	ΔF	Starting F	Ending F	ΔF
123	176	+53	176	123	−53
129	171	+42	171	129	−42
134	167	+33	167	134	−33
138	162	+24	162	138	−24
142	158	+16	158	142	−16
146	154	+8	154	146	−8

Procedures

PCI was tested using a two-alternative, forced-choice (2AFC) task. After listening to each stimulus, each subject was asked to identify the pitch contour by clicking on one of the two response choices shown on a screen. To indicate the corresponding pitch contour, a response choice was labeled with a rising line, while the other was labeled with a falling line. Subjects were instructed to ignore any remaining loudness variations and base their judgments on whether pitch was perceived to rise or fall. In each single- or combined-cue condition on each test electrode pair, stimuli were presented in four blocks; different pitch contours were played in a random order for ten times in each block. No training or feedback was provided during the tests. The percentage of rising responses was recorded for each pitch contour in each condition.

The apical and the medial electrode pairs were tested in a counterbalanced order across subjects. On each test electrode pair, PCI with place or temporal cues alone was tested first, again in a counterbalanced order across subjects. In the place-only condition, the stimuli had different amounts of current steering (see Sec. 2B) as well as a 30%, 150-Hz SAM. Although the fixed-rate SAM may affect subjects’ sensitivity to place-pitch changes, it was added so that the stimuli in all testing conditions had the same AM depth, and thus had similar loudness regardless of whether or not AM frequency changed. The Δα value that resulted in ∼76% correct PCI (i.e., d′ = 1 in a 2AFC task) was considered the Δα threshold and was calculated by interpolating the place-pitch psychometric function. In the temporal-only condition, the stimuli had different amounts of AM frequency change (see Sec. 2B) and were presented to the middle VC of the electrode pair (i.e., α was fixed at 0.5). Similarly, the ΔF value that resulted in ∼76% correct PCI was considered the ΔF threshold and was calculated by interpolating the temporal-pitch psychometric function.

Two conditions with combined place and temporal cues were then tested, again in a counterbalanced order across subjects. In one condition, the previously measured Δα value at threshold was added (with α increasing or decreasing) to the temporal-pitch contour stimuli with different amounts of AM frequency change (see Table TABLE III. for the specific ΔF values). In the other condition, the previously measured ΔF value at threshold was added (with AM frequency increasing or decreasing) to the place-pitch contour stimuli with different amounts of current steering (see Table TABLE II. for the specific Δα values). In both conditions, half of the stimuli had consistent place- and temporal-pitch changes (i.e., α and AM frequency both increased or both decreased), while the other half had inconsistent place- and temporal-pitch changes (i.e., α increased while AM frequency decreased or vice-versa).

RESULTS

PCI with place cues alone

Individual subjects’ PCI performance with place cues alone is shown in different panels of Fig. 1a (for the apical electrode pair) and Fig. 1b (for the medial electrode pair) using open circles. For each subject, the percentage of rising responses as a function of Δα was modeled with a two-parameter sigmoid function, following the equation

$$y=\frac{100}{1+e^{-(x-x_0)/b}},$$ (1)

where y is the percentage of rising responses and x is Δα from the beginning to the end of each pitch contour. The parameter x₀ is the Δα value that corresponds to 50% rising responses, and the parameter b is in inverse proportion to the slope of psychometric function. The best-fit sigmoid functions are shown by solid curves in Fig. 1 and their corresponding parameters are listed in Table TABLE IV..

Figure 1.

Percentage of rising responses on the apical and the medial electrode pairs (left and right panels, respectively) as a function of Δα. The results from individual subjects are shown in different panels. The open circles, black squares, and gray diamonds show the data with place cues alone, falling temporal cues, and rising temporal cues, respectively. Their best-fit sigmoid functions are shown by the solid, dashed, and dashed-dotted curves, respectively. The Δα values indicated by the arrows corresponded to 76% correct PCI with place cues alone and were added to the temporal-pitch contour stimuli with different amounts of AM frequency change.

TABLE IV.

Parameters of the best-fit sigmoid functions shown in Fig. 1.

		Apical electrode pair			Medial electrode pair
Subject	Condition	b	x0	r2	b	x0	r2
C1	Place cues alone	0.31	0.05	0.95	0.26	−0.17	0.94
	Falling temporal cues	0.53	0.24	0.90	0.16	0.18	0.99
	Rising temporal cues	0.46	−0.24	0.88	0.14	−0.20	0.99
C7	Place cues alone	0.04	0.10	1.00	0.11	−0.01	1.00
	Falling temporal cues	0.06	0.12	1.00	0.10	0.10	0.99
	Rising temporal cues	0.04	−0.01	1.00	0.14	−0.04	0.98
P1	Place cues alone	0.60	−0.21	0.56	0.41	−0.06	0.85
	Falling temporal cues	0.30	0.31	0.69	0.23	0.24	0.94
	Rising temporal cues	0.68	−0.32	0.63	0.34	−0.08	0.82
C19	Place cues alone	0.12	−0.06	1.00	0.10	−0.16	0.99
	Falling temporal cues	0.15	0.19	0.99	0.08	0.05	1.00
	Rising temporal cues	0.25	−0.34	0.93	0.09	−0.23	0.98
C4	Place cues alone	0.13	0.01	0.99
	Falling temporal cues	0.11	0.21	0.98
	Rising temporal cues	0.13	−0.09	0.99

As expected, subjects performed near chance (∼50% rising responses) when Δα was close to 0, responded more often with falling contours for negative Δα, and more often with rising contours for positive Δα. PCI data with place cues alone were analyzed using a two-way repeated-measures (RM) analysis of variance (ANOVA), with Δα and electrode pair as the two factors. The analysis showed that the percentage of rising responses significantly increased with Δα [F(11,33) = 20.50, p < 0.001]. The effect of electrode pair was not significant [F(1,33) = 1.48, p = 0.31], while there was a significant interaction between electrode pair and Δα [F(11,33) = 5.10, p < 0.001]. Post hoc t-tests with the Bonferroni correction showed that there were significantly more rising responses (p < 0.05) on the medial than on the apical electrode pair when Δα was 0.1, −0.1, and −0.2 (all close to 0). It is unclear why subject responses shifted towards rising contours and why it occurred only on the medial electrode pair for small ranges of current steering that were perceptually ambiguous.

PCI with temporal cues alone

Individual subjects’ PCI performance with temporal cues alone is shown in different panels of Fig. 2a (for the apical electrode pair) and Fig. 2b (for the medial electrode pair) using open circles. Again, solid curves show the best-fit sigmoid functions. The function parameters are listed in Table TABLE V.. Similar to PCI with place cues alone, the percentage of rising responses greatly increased from 0 to 100 % as ΔF varied from strongly negative to strongly positive. A two-way RM ANOVA with ΔF and electrode pair as the two factors showed that PCI with temporal cues alone was significantly affected by ΔF [F(11,33) = 127.06, p < 0.001]. The effect of electrode pair was not significant [F(1,33) = 3.07, p = 0.18], neither was the interaction between electrode pair and ΔF [F(11,33) = 1.65, p = 0.13].

Figure 2.

Percentage of rising responses on the apical and the medial electrode pairs (left and right panels, respectively) as a function of ΔF. The results from individual subjects are shown in different panels. The open circles, black squares, and gray diamonds show the data with temporal cues alone, falling place cues, and rising place cues, respectively. Their best-fit sigmoid functions are shown by the solid, dashed, and dashed-dotted curves, respectively. The ΔF values indicated by the arrows corresponded to 76% correct PCI with temporal cues alone and were added to the place-pitch contour stimuli with different amounts of current steering.

TABLE V.

Parameters of the best-fit sigmoid functions shown in Fig. 2.

		Apical electrode pair			Medial electrode pair
Subject	Condition	b	x0	r2	b	x0	r2
C1	Temporal cues alone	14.06	5.73	0.99	13.42	−8.86	0.99
	Falling place cues	14.41	12.71	0.98	14.72	36.36	0.96
	Rising place cues	11.95	−16.27	0.99	23.99	−50.56	0.94
C7	Temporal cues alone	8.37	−2.25	1.00	10.07	−3.41	1.00
	Falling place cues	9.53	20.87	0.99	7.42	12.62	0.99
	Rising place cues	10.87	−27.03	0.99	8.25	−16.46	0.99
P1	Temporal cues alone	17.10	0.01	0.95	9.57	−0.05	0.99
	Falling place cues	17.25	3.75	0.92	10.06	9.86	0.98
	Rising place cues	16.85	−11.66	0.98	9.91	−2.57	0.97
C19	Temporal cues alone	5.00	0.00	1.00	3.36	−3.87	1.00
	Falling place cues	5.58	4.67	1.00	7.41	5.06	0.99
	Rising place cues	5.94	−12.54	0.99	6.98	−18.85	0.99
C4	Temporal cues alone	9.59	8.07	0.99
	Falling place cues	12.47	14.96	0.96
	Rising place cues	8.50	−11.26	0.93

Subjects’ sensitivity to place- or temporal-pitch changes can be quantified by the slopes of the corresponding psychometric functions. The steeper the function slope [or the smaller the parameter b, see Eq. 1], the greater the pitch direction sensitivity. In Figs. 12, there was a large intersubject variability in the sensitivity to both place- and temporal-pitch changes. However, for each subject, pitch sensitivity was similar between the two electrode pairs. Interestingly, across subjects and electrode pairs, there was a significant correlation between the sensitivity to place- and temporal-pitch changes or between the parameter b of the corresponding psychometric functions (r²= 0.56, p = 0.02; see Fig. 3).

Figure 3.

Parameter b of place-pitch psychometric function as a function of that of temporal-pitch psychometric function. The filled circles show the data on the apical electrode pair, while the open circles show the data on the medial electrode pair. Individual subjects are identified by the codes to the right of the circles. The solid line shows the linear regression between parameter b of the two psychometric functions.

PCI with combined place and temporal cues

ΔF values at threshold combined with various Δα

Effects of rising or falling temporal-pitch cues.

Figure 1 also shows PCI scores as a function of Δα with increasing or decreasing AM frequencies (identified with d′ = 1 in isolation). The ΔF values at threshold (from ±8 to ±24 Hz for different subjects) were added to the place-pitch contour stimuli and are indicated by the arrows in Fig. 2. Data with falling temporal-pitch cues (or negative ΔF values at threshold) and their best-fit sigmoid functions are shown by black squares and dashed curves, while those with rising temporal-pitch cues are shown by gray diamonds and dashed-dotted curves. The parameters of the best-fit sigmoid functions can be found in Table TABLE IV..

PCI scores on the two electrode pairs shown in Fig. 1 were analyzed separately using two-way RM ANOVAs, with Δα and the level of added ΔF (e.g., positive, negative, or none) as the two factors. Similar statistical results were found on both electrode pairs. PCI was significantly affected by both Δα [apical: F(11,88) = 26.21, p < 0.001; medial: F(11,88) = 12.60, p < 0.001] and the added ΔF [apical: F(2,88) = 18.57, p < 0.001; medial: F(2,88) = 11.25, p = 0.005]. There was also a significant interaction between Δα and the added ΔF [apical: F(22,88) = 2.43, p = 0.002; medial: F(22,88) = 5.42, p < 0.001]. According to post hoc t-tests with the Bonferroni correction on both electrode pairs, when Δα was greater than 0.3 or less than −0.3 (i.e., beyond Δα thresholds), subjects’ PCI was dominated by the relatively large place-pitch changes and was not significantly affected by the added ΔF (p > 0.05). For Δα from 0.3 to −0.3, the percentage of rising responses significantly decreased with the added negative ΔF, but increased with the added positive ΔF (p < 0.05).

The effects of adding ΔF values at threshold to various Δα on PCI performance can also be seen from the horizontal shifts of place-pitch psychometric functions. Two-way RM ANOVAs with electrode pair and the level of added ΔF as the two factors were used to analyze the function parameters x₀ and b separately. The intercepts (x₀) of place-pitch psychometric functions or the Δα with 50% rising responses were similar across the two electrode pairs [F(1,6) = 0.17, p = 0.71], but were significantly higher (or shifted to the right) with the added negative ΔF than with the added positive ΔF or no ΔF [F(2,6) = 17.48, p = 0.003]. However, the slopes (b) of place-pitch psychometric functions were similar across the two electrode pairs [F(1,6) = 2.77, p = 0.20] and with the added positive or negative ΔF [F(2,6) = 0.83, p = 0.48].

Effects of consistent or inconsistent temporal-pitch cues.

The data in Fig. 1 were re-plotted in Fig. 4 based on the consistency between place- and temporal-pitch cues. In the condition with consistent pitch cues (black upward triangles), the data points in the left half of each figure (with negative Δα values) were combined with the negative ΔF value at threshold, while those in the right half (with positive Δα values) were combined with the positive ΔF value at threshold. In contrast, the positive ΔF value at threshold was added to the left data points in each figure, while the negative ΔF value at threshold was added to the right data points in the condition with inconsistent pitch cues (gray downward triangles). Since in each condition with consistent or inconsistent pitch cues, opposite ΔF values were used for the left and right data points, it may not be valid to fit them using a single sigmoid function.

Figure 4.

Percentage of rising responses on the apical and the medial electrode pairs (left and right panels, respectively) as a function of Δα. The results from individual subjects are shown in different panels. The open circles, black upward triangles, and gray downward triangles show the data with place cues alone, consistent temporal cues, and inconsistent temporal cues, respectively. The solid curves show the best-fit sigmoid functions for the data with place cues alone.

Only when Δα was near its threshold values, the perception of intended pitch contours was better with consistent than with inconsistent temporal cues, suggesting that these two cues are not orthogonal. With consistent temporal cues, the percentage of rising responses significantly increased when Δα was 0.1, but decreased for Δα from −0.3 to −0.1. On the other hand, with inconsistent temporal cues, the percentage of rising responses significantly decreased for Δα from 0.1 to 0.3, but increased for Δα from −0.3 to −0.1; sometimes the responses even shifted from rising to falling or vice versa. When Δα was greater than 0.3 or less than −0.3, PCI did not significantly change with either consistent or inconsistent temporal cues. The statistical significances of these findings were confirmed by two-way RM ANOVAs with Δα and cue consistency as the two factors. On the apical electrode pair, PCI was significantly affected by Δα [F(11,88) = 26.21, p < 0.001], but not by cue consistency [F(2,88) = 0.36, p = 0.71]. On the medial electrode pair, PCI was significantly affected by both Δα [F(11,88) = 12.60, p < 0.001] and cue consistency [F(2,88) = 5.41, p = 0.03]. Notably, Δα and cue consistency significantly interacted with each other on both electrode pairs [apical: F(22,88) = 8.77, p < 0.001; medial: F(22,88) = 11.86, p < 0.001].

Δα values at threshold combined with various ΔF

Effects of rising or falling place-pitch cues.

Figure 2 also shows PCI scores as a function of ΔF with increasing or decreasing α values (identified with d′ = 1 in isolation). The Δα values at threshold (from ± 0.2 to ± 0.4 for different subjects) were added to the temporal-pitch contour stimuli and are indicated by the arrows in Fig. 1. As in Fig. 1, black squares and dashed curves represent data with falling place-pitch cues (or negative Δα values at threshold) and their best-fit sigmoid functions, while gray diamonds and dashed-dotted curves represent those with rising place-pitch cues. The parameters of the best-fit sigmoid functions can be found in Table TABLE V..

For the PCI scores on the two electrode pairs shown in Fig. 2, two-way RM ANOVAs with ΔF and the level of added Δα as the two factors showed similar statistical results: the main effects of ΔF [apical: F(11,88) = 149.71, p < 0.001; medial: F(11,88) = 13.33, p < 0.001] and the added Δα [apical: F(2,88) = 16.67, p = 0.001; medial: F(2,88) = 5.83, p = 0.03], as well as the interaction between ΔF and the added Δα [apical: F(22,88) = 6.86, p < 0.001; medial: F(22,88) = 7.56, p < 0.001] were all significant. Post hoc Bonferroni t-tests showed that on both electrode pairs, when ΔF was greater than 24 Hz or less than −24 Hz (i.e., beyond ΔF thresholds), PCI was not significantly affected by the added Δα (p > 0.05). For ΔF from 24 to −24 Hz, the percentage of rising responses significantly decreased with the added negative Δα, but increased with the added positive Δα (p < 0.05).

The effects of adding Δα values at threshold to various ΔF on PCI performance can also be seen from the horizontal shifts of temporal-pitch psychometric functions. Again, two-way RM ANOVAs with electrode pair and the level of added Δα as the two factors were used to analyze the function parameters x₀ and b separately. The intercepts (x₀) of temporal-pitch psychometric functions or the ΔF with 50% rising responses were similar across the two electrode pairs [F(1,6) = 0.30, p = 0.62], but were significantly higher (or shifted to the right) with the added negative Δα than with the added positive Δα [F(2,6) = 10.87, p = 0.01]. However, the slopes (b) of temporal-pitch psychometric functions were similar across the two electrode pairs [F(1,6) = 0.18, p = 0.70] and with the added positive or negative Δα [F(2,6) = 2.21, p = 0.19].

Effects of consistent or inconsistent place-pitch cues.

The data in Fig. 2 were replotted in Fig. 5 based on the consistency between place- and temporal-pitch cues. Data points with consistent or inconsistent place- and temporal-pitch cues are indicated by the same symbols as in Fig. 4. Again, in each condition, the left and right data points in each figure contained opposite Δα values, thus it may not be valid to fit them using a single sigmoid function.

Figure 5.

Percentage of rising responses on the apical and the medial electrode pairs (left and right panels, respectively) as a function of ΔF. The results from individual subjects are shown in different panels. The open circles, black upward triangles, and gray downward triangles show the data with temporal cues alone, consistent place cues, and inconsistent place cues, respectively. The solid curves show the best-fit sigmoid functions for the data with temporal cues alone.

Similar to the patterns in Fig. 4, the perception of intended pitch contours was better with consistent than with inconsistent place cues, only when ΔF was near its threshold values. With consistent place cues, the percentage of rising responses significantly increased when ΔF was 8 Hz, but decreased when ΔF was −8 Hz. On the other hand, with inconsistent place cues, the percentage of rising responses significantly decreased for ΔF from 8 to 24 Hz, but increased for ΔF from −24 to −8 Hz; the response reversals from rising to falling or vice versa were apparent in subjects like C1 and C7. Again, when ΔF was greater than 24 Hz or less than −24 Hz, consistent or inconsistent place cues did not significantly change PCI. The statistical significances of these findings were confirmed by two-way RM ANOVAs with ΔF and cue consistency as the two factors. On the apical electrode pair, PCI was significantly affected by both ΔF [F(11,88) = 149.70, p < 0.001] and cue consistency [F(2,88) = 4.57, p < 0.05]. On the medial electrode pair, PCI was significantly affected by ΔF [F(11,88) = 13.33, p < 0.001], but not by cue consistency [F(2,88) = 1.95, p = 0.20]. The significant interaction between ΔF and cue consistency was noted on both electrode pairs [apical: F(22,88) = 10.79, p < 0.001; medial: F(22,88) = 8.92, p < 0.001].

DISCUSSION

The present PCI data with place cues alone extend our previous observations (Luo et al., 2010b) of CI users’ accurate perception of relatively wide current steering from one electrode all the way to the next (i.e., Δα = ± 1). Various frequency changes in speech and music signals may be encoded with more or less current steering like those tested in this study. With a smaller current-steering range (Δα around ± 0.6), our subjects still achieved similar identification of rising and falling pitch contours as those in Luo et al. (2010b) Not surprisingly, PCI performance approached chance level (∼50% correct) when Δα was close to 0. Overall, the resultant place-pitch psychometric functions had a typical S-shape with different slopes for different subjects. The Δα thresholds for PCI were calculated by interpolating the psychometric functions. Subjects C1, C4, and C7’s VC discrimination thresholds were also available from Landsberger and Srinivasan (2009). For each of the three subjects, the present Δα thresholds for dynamic place-pitch perception were actually close to those for static place-pitch perception, suggesting similar levels of difficulty for the two tasks. For unknown reasons, subjects’ responses shifted towards rising contours with relatively small Δα of ± 0.1 on the medial electrode pair. In spite of such response shift, no significant difference was found between identification of rising and falling place-pitch contours with the same current-steering range, consistent with the results of Luo et al. (2010b) Another similarity between the two studies is that PCI with place cues alone was not significantly affected by the position of electrode pair.

The present study also verified the feasibility of encoding pitch contours using time-varying AM frequencies. CI users perceived continuously increasing or decreasing AM frequencies on a fixed VC as rising or falling pitch contours respectively, similarly to time-varying pulse rates (Tong et al., 1983). Compared to the temporal rate pitch, the temporal envelope pitch studied in this study may be more relevant to current implant processing strategies such as CIS, ACE, HiRes, and HiRes-120 (see a review in Wilson, 2004), in which temporal envelopes of incoming sounds are super-imposed on fixed-rate pulse trains. Note that for CI users to perceive changes in temporal envelope pitch, AM depth should be at least above the AM detection threshold, and AM frequency may not exceed the upper limit of temporal-pitch perception (∼300 Hz), as in the present study. The psychometric functions with temporal cues alone resemble those with place cues alone in shape. PCI was near perfect with ΔF around ±53 Hz, and was less accurate with smaller AM frequency changes. The present ΔF thresholds for PCI (or dynamic temporal-pitch perception) were in the lower end of previously reported static AM frequency discrimination thresholds (Luo et al., 2010a; Chatterjee and Peng, 2008), which were obtained with a higher d′ value of 1.63 (79.4% correct in a 3AFC task). Future studies need to test if these two ΔF thresholds match with each other in individual CI users. Madden and Fire (1997) showed that NH listeners can detect 400-ms, 12-Hz frequency glides around 500 Hz. Although obtained with different experimental setups, these NH data were slightly lower than our CI users’ ΔF thresholds. Like the case with only place cues, PCI with temporal cues alone was also not significantly different between falling and rising contours (with the same AM frequency range), or between the two electrode pairs. Taken together, both AM frequency change and current steering are suitable for encoding pitch contours such as speech intonations with CI.

There was a significant correlation between subjects’ sensitivity to temporal- and place-pitch changes (as quantified by the slopes of psychometric functions) in this study, even though the number of subjects was relatively small. The variability in the present PCI data was mostly across subjects rather than between the electrode pairs. The positive correlation implies that if a subject has poor sensitivity to current steering, his/her access to AM frequency change would also be poor. A similar correlation was found between measures of temporal (AM detection) and spectral resolutions (electrode discrimination) across CI users (Chatterjee and Yu, 2010). In that study, significant correlations were found between measures in bipolar mode but not in monopolar mode, at soft levels (20–30 % of the dynamic range; DR) but not at medium levels (40% of the DR), which suggests that narrower excitation patterns may be necessary for both measures to reflect common underlying factors such as local neural survival. Nevertheless, our measures of CI users’ sensitivity to place- and temporal-pitch changes in monopolar mode at the most comfortable level (∼70% of the DR) were significantly correlated with each other. Possibly, stronger correlations may have been observed if pitch sensitivity was measured in a narrower stimulation mode. In line with the arguments of Chatterjee and Yu (2010), while current steering and AM frequency change are processed by different neural coding mechanisms (i.e., place and temporal coding), their perception seems to be affected by some common factor (e.g., CI user’s neural survival), and thus is significantly correlated.

The main finding of the present study is that CI users can effectively integrate place and temporal cues when asked to judge the overall pitch patterns of stimuli. With the tested pitch changes, the two pitch cues did interact with each other in the PCI task. It is not necessary that place and temporal cues were fused together to form a single pitch percept. Instead, subjects’ attention and/or perceptual weighting may have been switched between the two pitch dimensions. The significant interaction between Δα (or ΔF) and the added ΔF (or Δα) suggests that the integration of place- and temporal-pitch cues was most effective when the cues were similarly discriminable in isolation. When one of the cues was much more salient than the other, PCI judgments largely followed the more salient cue. This can be seen in most subjects when the ΔF (or Δα) value at threshold was combined with relatively large Δα (or ΔF). On each psychometric function, temporal- and place-pitch cues were comparable (in terms of perceptual salience) around the points indicated by the arrows. In such case, adding the other pitch cues significantly affected PCI judgments. The dependence of effective integration of pitch cues on their relative salience is consistent with our hypothesis and may explain why the discrete changes of electrode position did not affect the perception of perceptually salient pulse rate changes for the single subject in Tong et al. (1983). In the present study, the interaction between pitch cues was also not apparent in some subjects. For example, subject C7 was highly sensitive to current steering (even with the smallest tested Δα value), and the added AM frequency changes made negligible differences to her PCI judgments. However, she may be able to effectively integrate smaller current-steering ranges with greater AM frequency changes than those tested.

Adding the other pitch cues shifted the psychometric functions horizontally, but did not change their slopes (see Figs. 12). Thus, the function slopes still reflect subjects’ sensitivity to the original pitch changes. The values of x₀ with 50% rising responses were more positive when falling pitch cues from the other dimension were added. This shift to the right is consistent with the overall reduction of rising responses, a bias caused by the added falling pitch cues. Also, the increased x₀ values in the original pitch dimension should be perceptually as salient as the added falling pitch cues from the other dimension, leading to 50% rising responses. Similarly, the psychometric functions shifted to the left when rising pitch cues from the other dimension were added.

In line with our hypothesis, the analyses based on cue consistency showed that PCI was better with consistent place- and temporal-pitch cues than with inconsistent cues, when the two cues were similarly discriminable in isolation. It is not surprising that subjects were less confused with the overall pitch patterns of stimuli when the two pitch cues changed in the same direction than in the opposite directions. Inconsistent cues had stronger effects on PCI performance than consistent cues (see Sec. 3). For some subjects (such as C19 listening to the temporal-pitch contour stimuli), high-level PCI performance was achieved with single cues alone, which may have limited the room for improvement with consistent cues.

The present study revealed several other properties of pitch-cue integration in CI users. PCI results changed in the same way when the ΔF value at threshold was added to the place-pitch contour stimuli or when the Δα value at threshold was added to the temporal-pitch contour stimuli, showing a bidirectional and symmetric interaction between place- and temporal-pitch cues. Both PCI with single cues and integration of pitch cues were generally similar on the two electrode pairs, with no clear support for our hypothesis that lower place-pitch percepts on the apical electrode pair would be more effectively integrated with the low-frequency temporal-pitch percepts in this study. The overlap or separation between absolute place- and temporal-pitch ranges may not be that important for pitch-cue integration in relative pitch tasks such as PCI.

The pitch-specific PCI task was used to observe the integration of place and temporal cues. Although both have been referred to in the literature as pitch, the temporal and place pitches may form different perceptual dimensions. In our study design, subjects were asked to judge if the stimuli were rising in pitch. Subject responses in such a task would depend on how they interpreted the experiment instructions and internally labeled the two pitch dimensions. For subjects showing a lack of integration of pitch cues, their data may have been less variable and easier to interpret if they had simply been asked to judge if the stimuli were rising (rather than specifically in pitch).

The present data also have important implications for CI signal processing. For strategies intended to transmit independent speech information using current steering and temporal AM on the same frequency channel (e.g., HiRes-120 for the Advanced Bionics devices; Firszt et al., 2009), our results suggest that both methods can be used to encode dynamic frequency changes and these two cues may interact with each other (depending on their relative salience and directions). For example, the perception of one weaker cue may be limited by the simultaneous presentation of the other stronger cue. This may be one of the reasons why speech performance only slightly improved with HiRes-120 than with traditional strategies (e.g., Firszt et al., 2009). For CI strategies intended to better encode particular information such as speech intonations and music melodies (e.g., Luo et al., 2010b), current steering and temporal AM are both useful. To enhance the transmission of target pitch contours, one can coordinate both coding methods with consistent pitch cues on the same low-frequency channel. In this study, both place- and temporal-pitch cues were presented to the same frequency channel. Future studies need to investigate the interaction between place- and temporal-pitch cues on different frequency channels, as happens in multi-channel CI speech processing strategies.