Computer-Assisted Speech Training for Cochlear Implant Patients: Feasibility, Outcomes, and Future Directions

Abstract

Learning electrically stimulated speech patterns can be a new and difficult experience for cochlear implant patients. Cochlear implantation alone may not fully meet the needs of many patients, and additional auditory rehabilitation may be necessary to maximize the benefits of the implant device. A recently developed computer-assisted speech-training program provides cochlear implant patients with the means to conduct auditory rehabilitation at home. The training software targets important acoustic contrasts between speech stimuli and provides auditory and visual feedback as well as progressive training, thereby maintaining patients’ interest in the auditory training exercises. Recent scientific studies have demonstrated the effectiveness of such specialized auditory training programs in improving cochlear implant patients’ speech recognition performance. Provided with an inexpensive and accessible auditory training program, cochlear implant patients may find the motivation and momentum to get the most from the implant device.

I. INTRODUCTION

The cochlear implant (CI) is an electronic device that provides hearing sensation to patients with profound hearing loss. As the science and technology of the cochlear implant has developed over the past 50 years, the overall speech recognition of CI patients has steadily improved. With the most advanced implant technology and speech processing strategies, many patients receive great benefit, and are capable of conversing with friends and family over the telephone. However, considerable variability remains in individual patient outcomes. Some patients receive little benefit from the latest implant technology, even after many years of daily use of the device. This variability in patient outcomes is reflected not only in differences in speech recognition performance, but also in the time course of adaptation to novel speech patterns via electric hearing. While some patients may easily and quickly adapt to their implant, others may require an extensive learning period.

Much research has been devoted to exploring the sources of variability in CI patient outcomes. Some studies have shown that patient-related factors, such as duration of deafness, are correlated with speech performance [9, 22]. Several psychophysical measures, including electrode discrimination [7], temporal modulation detection [5, 12], and gap detection [2, 4, 27], have also been correlated with speech performance. Furthermore, Kelly et al. [22] found that earlier P2 latencies of auditory evoked potentials (AEPs) were associated with shorter durations of deafness and higher speech scores. Also, mismatch negativity (MMN) was absent or degraded in CI subjects with poor speech scores. The correlation between duration of deafness and speech performance suggests that early implantation may benefit profoundly deaf patients. The correlation between psychophysical measures and speech performance suggests that both spectral resolution and temporal resolution are important for speech recognition in electric hearing. The correlation between AEPs and speech performance provides objective evidence of central auditory processing differences among experienced CI users.

Besides the high variability in CI patient outcomes, individual patients also differ in terms of the time course of adaptation to electric hearing. During the initial period of use, post-lingually deafened CI patients must adapt to differences between their previous experience with normal acoustic hearing and the pattern of activation produced by electrical stimulation. Many studies have tracked changes in performance over time in “naïve” or newly implanted CI users. These longitudinal studies showed that most gains in performance occur in the first 3 months of use [18, 19, 25, 33, 39]. However, continued improvement has been observed over longer periods for some CI patients [36]. Experienced CI users must also adapt to new electrical stimulation patterns provided by updated speech processors, speech processing strategies and/or changes to speech processor parameters. For these patients, the greatest gains in performance also occurred during the first 3 - 6 months, with little or no improvement beyond 6 months [8, 21].

Overall, these results suggest that considerable auditory plasticity exists in CI patients, even after years of experience with their device. Because of the spectrally degraded speech patterns provided by the implant, “passive” learning via long-term use of the device may not fully engage patients’ capacity to learn novel stimulation patterns. “Active” auditory training may better exploit CI patients’ auditory plasticity and facilitate learning of electrically stimulated speech patterns. Auditory training, an important facet of aural rehabilitation, has been shown to improve hearing-impaired patients’ speech comprehension and communication [34]. Currently, fewer than 10 % of practicing audiologists offer hearing-impaired patients comprehensive auditory training, compared with 16 % in 1990 and 31 % in 1980 [1]. Hearing healthcare professionals acknowledge that cochlear implantation alone may not fully meet the needs of many patients, and that additional auditory therapy may enhance the benefits of the implant device. However, for a variety of reasons, few hearing healthcare professionals routinely include auditory training in the services they provide to CI patients. Time and cost considerations often preclude providing extensive and intensive auditory therapy. Given the limited number of speech pathologists working with CI patients and the high costs associated with auditory rehabilitation, it has long been urgent to develop an inexpensive and effective auditory training system for CI patients, especially for those patients who have the greatest difficulty with speech understanding.

II. METHODS

The Computer-Assisted Speech Training (CAST) program, developed at the House Ear Institute, provides CI patients with self-directed auditory rehabilitation via home computer. As an auditory training tool, CAST provides some appealing advantages over conventional, site-specific approaches for auditory rehabilitation. First, the cost of individualized auditory training is significantly lower than that with traditional auditory rehabilitation provided by hearing clinics and hospitals. Second, CAST is easily accessible by CI patients, who can practice at home at any time, provided they have access to a computer. Third, patients’ progress can be easily monitored and shared with their clinical practitioners. Finally, CAST provides auditory training that is specific to the needs of CI users, and more importantly to the needs of individual CI patients. CAST targets acoustic contrasts that are especially problematic for CI users, as opposed to targeting cognitive/developmental learning. Besides being affordable, accessible and appropriate for CI patients, CAST has many other unique features.

1. Comprehensive training materials

CAST provides a comprehensive set of training materials with which to train recognition of many kinds of sounds, not just speech. Training materials include simple pure tones, environmental sounds, monosyllabic words, consonant training stimuli (in v/C, v/C/v and C/v contexts), familiar words, familiar sentences, simple melodic sequences and familiar melodies. For targeted phonetic contrast training, CAST uses more than 1000 novel monosyllabic words (for initial, medial and final vowel and consonant training) and nonsense words (for initial, medial and final consonant training), spoken by 4 different talkers. Additional training materials and/or training modules can be easily incorporated for individual patients to maximize training outcomes. For advanced users, CAST can also mimic difficult listening environments by adding background noise or competing speech in real time.

2. Individualized training protocols

CAST also provides individualized training protocols for CI patients. The level of difficulty is automatically adjusted according to individual patient performance by increasing the number of response choices and/or reducing the acoustic differences between response choices. During training, both auditory and visual feedback is provided, allowing users to repeatedly compare incorrect responses to correct responses. As performance improves, the level of difficulty is automatically increased. If performance does not improve, the level of difficulty is decreased. At the end of each testing and training session, the program offers training guidance. For example, based on testing or training results, the program suggests the appropriate training level, whether users should go to an easier or more difficult training level, etc. Feedback and encouragement are offered throughout, as the positive effects of “good will” can help patients build momentum during the rehabilitation process.

3. Integrated training structures

CAST provides easy integration for auditory rehabilitation between CI patients and their audiologists/speech therapists. All training and test results are saved in a standardized database. Users can easily view results and ongoing progress and data can be easily exported and shared with the patient’s hearing health practitioner. Results are automatically saved and include: user’s name, test and training results, test and training date, training time for each exercise and total training time. Audiologists can use these results to track a patient’s progress over time, check for the potential problems with the CI device and optimize the speech processor. For example, during a clinical visit, the audiologist can review the test results and adjust processor parameters (e.g., frequency allocation, channel gains, etc.) to improve performance in problem areas (e.g., reception of fricatives, etc.). Over a longer term, different MAPs (with different parameter settings) can be loaded onto the processor and performance can be tracked over time to determine the optimal MAP; the combination of take-home experience and lab testing (with easy access to results) will allow for better long term optimization of the processor. CAST provides a visually clean and user-friendly graphic interface, allowing patients to concentrate on listening rather than navigating an unfamiliar computer program.

III. TRAINING OUTCOMES

While CAST provides inexpensive and accessible auditory rehabilitation for CI patients, the most important question is whether computer-assisted auditory training can effectively improve CI patients’ speech recognition performance, especially for patients who have great difficulty with speech understanding. Auditory training has been shown effective in the rehabilitation of children with central auditory processing disorders [20, 28], children with language-learning impairment [26, 35], and hearing aid (HA) users [34]. Some earlier studies have assessed the effects of auditory training on speech recognition by poor-performing CI patients [3, 6]. Busby et al. [3] examined the effects of ten one-hour speech perception training sessions (1 - 2 sessions per week); three prelingually deafened CI users (two adolescents and one adult) participated in the experiment. After the training period, there were only minimal changes in speech performance; the subject with the greatest improvement was implanted at an earlier age, and therefore had a shorter period of deafness. Dawson and Clark [6] reported more encouraging results for vowel recognition training in five CI users. Each subject had been deaf for at least four years prior to implantation, and none had achieved open-set speech recognition. Training consisted of one 50-minute training session per week for ten weeks. Following training, four of the five subjects showed improvement in some measures; this improvement was retained on subsequent testing three weeks after training was completed. The mixed and generally poor outcomes from earlier CI speech training studies may be due to the amount and type of training employed in these studies. Several recent studies have evaluated the effects of computer-assisted auditory training (using the CAST program) on CI patient performance, and are summarized as follows:

1. Phoneme recognition training in CI patients

Fu et al. [15] investigated the effects of computer-assisted speech training on the recognition performance of CI users. Ten adult CI patients with limited speech recognition capabilities participated in the study. Baseline (pre-training) multi-talker phoneme recognition performance was measured for at least two weeks, or until performance asymptoted. After baseline measures were complete, subjects trained at home using the CAST program (loaded onto their personal computers or loaner laptops). Subjects were instructed to train at home 1 hour per day, 5 days per week for a period of 1 month or longer. Subjects returned to the lab every two weeks for re-testing (same tests as baseline measures). Results showed that both vowel and consonant recognition significantly improved for all subjects after training. Mean vowel recognition improved from 23.7 to 39.5 % correct (paired t-test: p<0.0001). Mean consonant recognition improved from 25.1 to 38.6 % correct (p<0.005). For a subset of CI subjects, mean sentence recognition also improved from 27.9 to 55.8 % correct (p<0.01) after training. While performance significantly improved for all subjects after four weeks or more of moderate training, there was significant inter-subject variability in terms of the amount and time course of improvement. For some subjects, performance significantly improved after only a few hours of training, while others required a much longer time course.

2. Melodic sequence identification training in CI patients

Music perception and appreciation remains difficult for most CI patients. Galvin et al. [17] developed an objective test of CI users’ music perception: melodic sequence identification (MSI). During the MSI test, CI subjects were asked to identify simple 5-note melodic sequences, in which the pitch contour was systematically varied. The interval between successive notes was systematically varied between 1 and 5 semitones to test the intonation provided by the CI device. Results showed large inter-subject variability in MSI performance. The best performers correctly identified more than 90 % of the melodic sequences when there were 2 semitones between notes in a sequence, while poor performers correctly identified less than 40 % of the sequences with 5 semitones between notes. Six CI patients with poor MSI performance were trained using the CAST software to see whether MSI performance could be improved with moderate auditory training. Subjects were instructed to train for 1 hour per day, 5 days per week, for a period of 1 month or longer. For all subjects, performance significantly improved with training (p=0.004); the amount of improvement ranged from 15.5 to 45.4 percentage points. Pre- and post-training familiar melody identification (FMI) was also measured in 4 of the 6 CI subjects; 12 melodies were tested without rhythm cues, similar to Kong et al. [23]. Results showed that after training, mean FMI performance significantly improved by 20.8 percentage points (p=0.020). Anecdotal reports suggested that CI subjects’ music perception and appreciation generally improved after MSI training. For example, some subjects reported that they were better able to separate the singer’s voice from the background music while listening to music in the car.

3. Chinese tone recognition training in Mandarin-speaking CI patients

In tonal languages such as Mandarin Chinese, the tonality of a syllable is lexically important [24, 40] and Chinese sentence recognition is highly correlated with tone recognition [10, 14]. While the fundamental frequency (F0) contour contributes most strongly to tone recognition [24], other temporal cues that co-vary with tonal patterns (e.g., amplitude contour and periodicity fluctuations) also contribute to tone recognition, especially when F0 cues are reduced or unavailable [10, 11, 41]. Wu et al. [43] recently investigated whether moderate auditory training can improve the recognition of Chinese vowels, consonants and tones by hearing impaired children. Ten Mandarin-speaking children (7 CI users and 3 HA users) participated in the study. Training was conducted at home using the CAST program; subjects trained for one half-hour per day, five days per week, for a period of 10 weeks. Results showed that mean vowel recognition scores significantly improved from 63.1 to 84.8 % correct (paired t-test: p=0.006) after the 10-week training period. Similarly, mean consonant recognition scores significantly improved from 39.7 to 58.2 % correct (p<0.001) and Chinese tone recognition scores significantly improved from 56.0 to 71.1% correct (p=0.007). Vowel, consonant, and tone recognition performance was re-measured after training was completed; follow-up performance was measured at 1, 2, 4 and 8 weeks after training was completed. Follow-up measures remained significantly higher than pre-training baseline measured for vowel (p=0.018), consonant (p=0.002), and tone recognition (p=0.009), suggesting that the improved performance with training was retained well after training had stopped.

IV. DISCUSSION

The results from these studies demonstrate that moderate amounts of auditory training with the CAST program improved CI and HA users’ recognition of phonemes, melodic sequences, and Chinese tones. While auditory training generally improved performance in the targeted listening task, the improvement sometimes generalized to auditory tasks that were not explicitly trained (i.e., improved sentence recognition after training with phonetic contrasts, or improved familiar melody identification after training with simple melodic sequences). More importantly, the improved performance was not transitory, as post-training performance levels remained higher than pre-training baseline levels 1 – 2 months after training was completed. Because most CI subjects had a minimum of one year of experience with their device prior to training (i.e., at least one year of “passive” learning), the results suggest that “active” learning was necessary to improve speech performance. The improvements in speech recognition with auditory training were comparable to (and often much greater than) gains reported with recent advances in CI technology and speech processing. Taken together, the results suggest that computer-assisted auditory training, as implemented in the CAST program, provides an effective, inexpensive and accessible auditory rehabilitation for CI patients.

While the results from these recent training studies with the CAST program have shown successful training outcomes with CI patients, earlier studies have shown somewhat mixed results [3, 6]. The better training outcomes with the CAST program may have been due to differences in terms of training protocols. For example, one-to-one personal training that targeted a variety of phonetic contrasts was used in Dawson and Clark [6], while computer-assisted progressive training that targeted minimal phonetic contrasts was used the CAST studies. Fu et al. [16] systematically evaluated the effects of training protocols on NH listeners’ adaptation to 8-channel spectrally shifted speech. Four different training protocols have been evaluated, including: test-only (repeated baseline measures), stimulus preview, phonetic contrast training and sentence training. Results showed a significant effect of the training protocol on the training outcomes. The targeted, progressive phonetic contrast training (which is used in the CAST program) produced the greatest improvement in recognition of spectrally shifted vowels. The improved vowel recognition also generalized to improved consonant and sentence recognition with spectrally shifted speech.

The better training outcomes with the CAST program may have also been due to differences in terms of the frequency of training, i.e., how often the training was conducted during the training period. In previous CI training studies [3, 6], subjects trained 50 minutes once per week, for 10 weeks, while in the CAST studies, subjects trained 30 – 60 minutes per day, 5 days per week, for a period of one month or longer. Nogaki et al. [29] systematically evaluated the effect of training frequency on NH listeners’ adaptation to 8-channel spectrally shifted speech. The subjects completed 5 training sessions using the phonetic contrast training protocol [15, 16]. Eighteen subjects were divided into 3 groups, in which 1-hour training sessions were provided either 1, 3 or 5 times per week. The results suggest that, while more frequent training seemed to provide better adaptation over the 5 training sessions, the frequency of training did not significantly affect training outcomes (at least for the training frequencies that were studied). Thus, it may be more important to complete an adequate number of training sessions over a reasonable time period. These simulation results suggest that CI patients may significantly benefit from auditory training, even when infrequently performed. The results further suggest that the difference in CI training outcomes between these recent studies and other earlier studies may not be simply due to time-intensive training, but rather due to differences in training protocols and (perhaps) training stimuli. CAST makes use of a far greater number of multi-talker speech stimuli (more than 4000 stimuli) than used in previous CI speech training studies. The results suggest that it may be more important to develop effective training protocols and stimuli than to simply increase patients’ time commitment.

It is important to note that in the CAST training studies [15, 16, 17, 29, 43], the stimuli used for baseline measures were not used for training. Baseline measures were conducted using standard speech test databases (e.g., the vowel stimuli recorded by Hillendbrand et al. [21], or the consonant stimuli recorded by Shannon et al. [31]), while training was conducted using monosyllable and/or nonsense words produced by completely different talkers. Because improved performance for a trained auditory task often generalized to improved performance in other auditory tasks (e.g., phonetic contrast training improved both phoneme and sentence recognition, melodic sequence training improved both sequence and familiar melody identification), the large multi-talker database used in the CAST may provide some advantage over smaller, more limited databases. The generalized improvements in performance agree with previous neuro-physiological studies [37], which showed that both behavioral performance and neuro-physiological changes observed after auditory training generalized to stimuli not used in the training, thus demonstrating behavioral “transfer of learning” and plasticity in underlying physiologic processes. Tremblay et al. [38] reported results further indicating that training-associated changes in neural activity may precede behavioral learning. The results in Fu et al. [15] also indicated that the time course of improvement varied significantly across CI subjects. Depending on patient-related factors (e.g., the number of implanted electrodes, the insertion depth of the electrode array, duration of deafness, etc.), some CI patients may require much more auditory training to noticeably improve their performance. Objective neuro-physiological measures may provide useful information about the progression of training, i.e., whether a particular training protocol should be continued or whether the training protocol and/or training materials should be adjusted. These objective measures may allow for the development of more efficient training protocols for CI patients; fortunately, the flexibility in computer-assisted auditory training (as in CAST) allows such changes to be easily implemented.

A future direction that may further improve CI patient training outcomes is to fully integrate the standardized testing and training features of the CAST program into the clinical fitting system. Many previous CI studies [8, 13, 42] have shown that acutely measured performance may significantly under-estimate the effects of changes in speech processing. Even a short period of adaptation may result in significantly better speech performance [16, 31]. Longer adaptation periods may be required to fully evaluate the effects of changes in speech processing. By integrating the test and training platforms into the clinical fitting procedures, CAST may help accelerate the adaptation process over the short and long term. Audiologists have little time available to test and re-test performance with different MAPs; CAST allows training and testing with little or no supervision on the part of the audiologist. Thus, during clinical visits, MAPs may be more accurately evaluated after some amount of training and testing. Also, during a clinical visit, CI patients largely judge the quality of a speech processor according to the audiologist’s voice; by providing multiple talkers, CAST allows patients to better judge the quality of the MAP. Over the long term, subjects can compare multiple MAPs loaded onto their speech processors in everyday listening conditions outside of the clinic (CAST training can also help in this period of adaptation), and then return to the clinic for testing. Testing results can be used to determine optimal MAPs or further adjustments to speech processors.

How a parameter change is introduced may also influence the time course and degree of adaptation. Fu et al. [13] found that CI subjects were able to only partially adapt to an abrupt change in the acoustic frequency allocation, even after 3 months of continuous use. In a follow-up study with one of the CI subjects, gradually introducing a severe shift in frequency allocation provided for a more complete adaptation, albeit over a much longer adaptation period (18 months). Thus, while “passive learning” (i.e., incidental learning via everyday experience with the CI device) may allow for some degree of adaptation to changes in speech processing, “active” auditory training may help to accelerate the adaptation process and provide more complete adaptation. Because it is important to evaluate speech processor settings in light of adaptation, and because auditory training may accelerate the adaptation process, test and training results from the CAST program may be used to guide gradual adjustments to speech processor settings, thereby making the adaptation more complete and less stressful in the long run.

Advances in computer technology have contributed greatly to the latest-generation CI technology. These advances have also allowed for the development of important rehabilitation tools, which may ultimately provide the greatest benefit for those CI patients who have difficulty with their device. Motivation is an important factor in successful patient outcomes. With the right tools and resources for auditory rehabilitation, CI patients may find the necessary motivation and momentum to get the most from their implant.

V. SUMMARY AND CONCLUSIONS

Computer-assisted auditory training, as implemented in the CAST program, provides an economical and effective alternative or complement to site-specific auditory rehabilitation. Moderate amounts of auditory training performed at home with the CAST software resulted in significant improvements in CI patients’ speech, music and Chinese tone recognition. These improvements were retained months after training was completed. Targeted auditory training tasks with novel training stimuli also generalized to improved performance in other auditory training tasks.

Learning objectives.

1. To understand the importance of auditory training for cochlear implant patients.

2. To learn the recent development of computer-assisted speech training specially designed for cochlear implant patients.

List of abbreviations

CAST

Computer-Assisted Speech Training

NH

Normal-hearing

CI

Cochlear implant

HA

Hearing aid