Development of Audiovisual Comprehension Skills in Prelingually Deaf Children With Cochlear Implants

Abstract

Objective

The present study investigated the development of audiovisual comprehension skills in prelingually deaf children who received cochlear implants.

Design

We analyzed results obtained with the Common Phrases (Robbins et al., 1995) test of sentence comprehension from 80 prelingually deaf children with cochlear implants who were enrolled in a longitudinal study, from pre-implantation to 5 years after implantation.

Results

The results revealed that prelingually deaf children with cochlear implants performed better under audiovisual (AV) presentation compared with auditory-alone (A-alone) or visual-alone (V-alone) conditions. AV sentence comprehension skills were found to be strongly correlated with several clinical outcome measures of speech perception, speech intelligibility, and language. Finally, pre-implantation V-alone performance on the Common Phrases test was strongly correlated with 3-year postimplantation performance on clinical outcome measures of speech perception, speech intelligibility, and language skills.

Conclusions

The results suggest that lipreading skills and AV speech perception reflect a common source of variance associated with the development of phonological processing skills that is shared among a wide range of speech and language outcome measures.

Speech perception in normal-hearing listeners is a multisensory process that typically involves attending to and encoding not only the auditory properties of the speech signal but also the visual articulatory attributes of the talker. Multisensory integration of speech occurs naturally and automatically in normal-hearing listeners of all ages (Arnold & Köpsel, 1996; Desjardins, Rogers, & Werker, 1997; Dodd, 1979; Erber, 1969; McGurk & Mac-Donald, 1976; Sumby & Pollack, 1954). Several studies have shown that even young infants are capable of audiovisual (AV) speech perception (Kuhl & Meltzoff, 1982; Patterson & Werker, 1999, 2002, 2003). Interestingly, children with hearing loss also show evidence of AV enhancement when perceiving speech (Erber, 1972, 1979). The purpose of the present longitudinal study was to investigate the development of AV comprehension skills in children with a profound prelingual hearing loss who subsequently acquire hearing via cochlear implants and assess the effects of early experience and age at implantation on AV comprehension skills in this clinical population.

Many studies over the years have found that normal-hearing children and even young infants are capable of multisensory perception (Desjardins et al., 1997; Kuhl & Meltzoff, 1982; Lewkowicz, 1992, 2000; Lewkowicz & Kraebel, 2004; MacKain, Studdert-Kennedy, Spieker, & Stern, 1983; Spelke, 1979, 1981). Normal-hearing preschool and school-aged children are susceptible to the McGurk illusion, although children tend to show less visual influence than adults (Desjardins et al., 1997; McGurk & MacDonald, 1976). In a study of lipreading in infants, Dodd (1979) found that 10- to 16-week-olds attended more to synchronous than asynchronous AV presentations of nursery rhymes.

In another study of infant AV speech perception, Kuhl & Meltzoff (1982) presented 18- to 20-week-old infants with two faces articulating vowel sounds (/a/and/i/) and one sound track synchronized with one of the faces. They found that infants preferred to look longer at the matching face than the nonmatching face. Similarly, MacKain and colleagues presented infants with consonant-vowel/consonant-vowel pairs in which the auditory speech corresponded with one of two visually presented video clips (MacKain et al., 1983). Infants preferred to look longer at the videos that not only were synchronized in time with the auditory speech but also exactly matched the consonant-vowel/consonant-vowel pairs. These studies suggest that children and very young infants perceive the world in a multimodal rather than unimodal fashion.

Studies of AV speech perception in hearing-impaired children have also found similar patterns of lipreading and AV benefit (Arnold & Köpsel, 1996; Erber, 1971, 1972, 1975, 1979). For example, Erber (1972) reported that normal-hearing and severely hearing-impaired children performed best in the AV condition of a consonant recognition test compared with the A-alone and V-alone conditions. However, profoundly deaf children did not perform any better in the AV condition compared with the V-alone condition of the consonant recognition test (Erber, 1972), but they did show some evidence of AV enhancement when tested on word recognition (Erber, 1979). Interestingly, when presented with AV word stimuli in which the auditory and visual words were in conflict, children with poorer hearing levels relied more on visual cues to identify the target word, whereas children with better hearing levels relied more on auditory cues to identify the target word (Seewald, Ross, Giolas, & Yonovitz, 1985). This differential pattern of results suggests that the primary sensory modality for speech perception in normal-hearing children is audition, whereas the primary sensory modality for speech perception in hearing-impaired children is vision, using lipreading cues.

If deaf children depend primarily on the visual channel for speech perception, what happens to their lipreading abilities once their auditory channel is restored via a cochlear implant? Does reliance on the primary sensory channel change and become reorganized after cochlear implantation? It is possible that children who receive cochlear implants still depend primarily on visual speech cues and merely supplement lipreading information with the additional auditory input from their implant. Alternatively, it is possible that children who receive cochlear implants may gradually learn to rely primarily on auditory speech cues as they gain more implant experience over time and reorganize the processing of sensory inputs.

Several earlier studies on prelingually deaf children with cochlear implants have found that combined AV information in tests of spoken language perception improved performance over A-alone and V-alone conditions (Blamey et al., 2001; Geers, 1994; Geers & Brenner, 1994; Geers, Brenner, & Davidson, 2003; Lachs, Pisoni, & Kirk, 2001; Staller, Dowell, Beiter, & Brimacombe, 1991; Surowiecki, Grayden, Dowell, Clark, & Maruff, Reference Note 1; Tyler et al., 1997; Tyler, Opie, Fryauf-Bertschy, & Gantz, 1992). Tyler et al. (1997) compared consonant feature recognition performance in a small group of children with 2 years of implant use and a second small group of children with 4 years of implant use. They reported an improvement in lipreading performance over time after implantation. As the children accumulated experience with the implant, their lipreading skills in perceiving speech via vision alone also improved.

In another study, Geers and Brenner (1994) investigated AV speech perception in 13 children from before implantation to 3 years after implantation. They found similar performance in V-alone and AV conditions before implantation, but better performance in the AV condition than the V-alone condition by 3 years after implantation. However, the results they reported were based on a combination of spoken word and sentence comprehension tests and they did not include an A-alone presentation condition in their study.

A recent cross-sectional study by Lachs et al. (2001) examined AV speech perception in a small group of prelingually deaf children who had used their cochlear implants for a period of 2 years. The authors administered an open-set sentence comprehension test under three presentation conditions: A-alone, V-alone, and AV. Lachs et al. (2001) found that prelingually deaf children with cochlear implants performed better on AV than on either the A-alone or V-alone conditions. This pattern of results was similar to previous findings observed with postlingually deaf adults with cochlear implants (Kaiser, Kirk, Lachs, & Pisoni, 2003) and prelingually deaf children with cochlear implants (Geers et al., 2003; Tyler et al., 1997). However, Lachs et al. found no difference in performance between the A-alone and V-alone conditions. This pattern was different from the results typically observed with normal-hearing adults and postlingually deaf adult cochlear implant users, who routinely perform better in A-alone than V-alone conditions (Bergeson, Pisoni, Lachs, & Reese, 2003; Kaiser et al., 2003). Overall, the findings reported by Lachs et al. (2001) demonstrated that prelingually deaf children who use cochlear implants show evidence of multisensory enhancement and benefit when speech is presented in an AV format compared to A-alone and V-alone formats.

In addition to reporting new findings on AV speech perception in this clinical population, Lachs et al. (2001) also found that the benefits observed under AV presentation conditions after two years of implant use were strongly correlated with A-alone spoken word recognition and speech production skills, both of which make use of a common set of underlying phonological processing abilities. The children who displayed more AV gain from the combined auditory and visual sensory inputs performed better on standardized clinical tests of spoken word recognition and speech intelligibility. These findings suggest that AV processes are not isolated or independent from other speech and language skills but may reflect common properties of the language processing system itself.

Finally, in a recent study, Bergeson, Pisoni, & Davis (2003) assessed A-alone, V-alone, and AV spoken word and sentence recognition performance in a large group of prelingually deaf children with cochlear implants from before implantation to 3 years after implantation using the Pediatric Sentence Intelligibility (PSI) test (Jerger, Lewis, Hawkins, & Jerger, 1980). The PSI is a forced-choice, closed-set test used clinically to measure word and sentence recognition skills in young children. The results revealed that performance in all three presentation conditions improved over time after implantation. Similar to the results of Lachs et al. (2001), they also found that performance on the PSI word and sentence recognition subtests was strongly correlated with performance on other outcome measures of speech and language skills. Moreover, they found that before implantation, V-alone word and sentence recognition performance was correlated with these speech and language outcome measures 3 years after implantation, suggesting that pre-implantation lipreading skills may be a good predictor of later success and benefit with a cochlear implant.

However, there were several limitations in the Bergeson et al. (2003) study. First, they used a forced-choice, closed-set test, designed to measure word and sentence recognition performance in young children. In the PSI test, children hear and/or see a word or sentence articulated by an experimenter and are then asked to point to one of six pictures that corresponds to the word or sentence. Thus, the PSI could be done by simply matching patterns rather than comprehending the words or sentences. Second, the same words and sentences were used in all three presentation conditions, as well as in all test sessions across the longitudinal study. These two factors combined to produce near ceiling effects in performance by 2 years after implantation, making it difficult to observe any AV benefit.

Several factors have been found to contribute to success and benefit with a cochlear implant (Bergeson & Pisoni, 2004; Bergeson et al., 2003; Fryauf-Bertschy, Tyler, Kelsay, Gantz, & Woodworth, 1997; Kirk, Miyamoto, Ying, Perdew, & Zuganelis, 2002; Lachs et al., 2001; Waltzman et al., 1997). One source of variability is the nature of the early sensory and linguistic environment that deaf children experience after receiving their implant. Clinicians typically partition deaf children who use spoken language into one of two communication modes: oral communication (OC), in which children use auditory-oral skills and are educated using an oral approach, and total communication (TC), in which children use simultaneous signed and spoken English. Oral communication methods can range from auditory-verbal therapy, in which auditory information is heavily emphasized and lipreading is discouraged (Ling, 1993; Rhoades, 1982), to cued speech, in which specific hand cues are used to supplement visual lipreading information (Cornett & Daisey, 2000). Similarly, total or simultaneous communication can range from an emphasis on spoken English, an equal emphasis on signed and spoken English (e.g., Signing Exact English (Gustason & Zawolkow, 1993)), and finally to an emphasis on manual signs (Geers et al., Reference Note 2). Note that even the latter extreme, which places emphasis on manual signs, is still not a completely unimodal sign language, such as American Sign Language.

Previous research has shown that OC children have significantly better spoken word recognition, expressive language, and speech intelligibility skills than TC children (e.g., Cullington, Hodges, Butts, Dolan-Ash, & Balkany, 2000; Hodges, Dolan-Ash, Balkany, Schloffman, & Butts, 1999; Kirk, Pisoni, & Miyamoto, 2000; Miyamoto, Kirk, Svirsky, & Sehgal, 1999; Svirsky, Robbins, Kirk, Pisoni, & Miyamoto, 2000). Both Lachs et al. (2001) and Bergeson et al. (2003) found that the children’s early sensory and linguistic experience, in the form of OC or TC education, was related to speech intelligibility and to their ability to use auditory and visual sources of information. Correlations between AV speech perception skills and measures of spoken word recognition and speech intelligibility were stronger in OC children than in TC children, demonstrating effects of early sensory experience and language processing activities on the ability to use multisensory sources of information about speech (Lachs et al., 2001). Bergeson et al. (2003) also found that OC children performed better overall on tests of AV word and sentence recognition than TC children.

Another major source of variability that has been found to contribute to success and benefit with a cochlear implant is age at implantation. Children who are implanted at younger ages and have shorter periods of auditory deprivation display better speech and language outcomes than children who are implanted at older ages (Bergeson et al., 2003; Fryauf-Bertschy et al., 1997; Kirk et al., 2002; Staller et al., 1991; Waltzman et al., 1997). Although this is the typical pattern of results observed for auditory speech perception, Bergeson et al. (2003) found that children who were implanted later and thus had a longer duration of deafness before cochlear implantation were actually better lipreaders than early-implanted children from the pre-implantation interval to 3 years after implant, based on measures from the PSI test.

The recent findings reported by Lachs et al. (2001) and Bergeson et al. (2003) uncovered strong relations between AV speech perception skills and speech and language outcomes. Both studies found effects of communication mode and age at implantation on AV speech perception performance. These findings are theoretically important because they suggest that AV speech perception is not based on a set of isolated or autonomous perceptual skills but is strongly correlated with other measures of speech and language processing that may reflect important milestones in speech and language development.

The cross-sectional results reported by Lachs et al. (2001) represent the first step toward understanding how deaf children with cochlear implants perceive multisensory speech events. However, it is important to investigate change over time to understand the development of AV speech perception skills in this clinical population. It is in the process of development that we can begin to see the effects of early sensory and linguistic experience and study the time-course of perceptual learning and development both before and after cochlear implantation. The recent longitudinal study by Bergeson et al. (2003) investigated change over time in AV speech perception skills using measures obtained from the PSI test, but there were several limitations in their study that resulted in ceiling effects in some children by 2 years after implantation and in most children by 3 years after implantation.

Do children who receive cochlear implants rely primarily on auditory, visual, or a combination of audiovisual speech cues as they gain implant experience over time? To investigate the development of AV comprehension skills in this clinical population, we measured open-set A-alone, V-alone, and AV sentence comprehension using the Common Phrases (CP) test. We studied the effects of early experience (i.e., communication mode) and age at implantation on AV comprehension in a large group of prelingually deaf children with cochlear implants longitudinally, from before implantation to 5 years after implantation.

Method

Participants

Participants in this study consisted of 80 children who experienced a profound hearing loss before the age of 36 months, received a cochlear implant before 9 years of age, and used either OC or TC methods. Classification of communication method is based primarily on parental report and was confirmed by the child’s educational setting. Based on a median split, children were divided equally into an early-implanted group (implanted before or at the age of 53 months) and a late-implanted group (implanted after the age of 53 months). Children were tested once every 6 months to a year for 5 years. Because many of the participants lived long distances from the Indianapolis area, not all children could be tested at each interval. All children completed the CP test in at least two intervals; 15 children completed the CP test in five or six intervals (M = 3.3 intervals).

Table 1 provides a summary of the demographics of these children. Most of the children had a Nucleus 22 or 24 implant model; only one child had a Clarion implant model. The majority of the children used the Spectra or MSP processors with MPEAK or SPEAK strategies at the time of testing. Other processors used were WSP, Esprit, and Sprint, and other strategies used were F0F1F2, F0F1F2F5, CIS, and ACE. Of the children who changed processors and/or strategies over the period of testing, 17 were late-implanted and 13 were early-implanted. Most of these children changed to the Spectra processor and the SPEAK strategy.

TABLE 1.

Characteristics of participants

Communication mode	Age group		Age at implantation (mo)	Unaided PTA (dB HL)	Aided PTA (CI) (dB HL)	Number of electrodes
Oral communication (n = 39)	Early (n = 23)	Mean	36	113	34	19.4
		SD	9.6	4.9	3.9	3.5
		Range	17–53	103–121	25– 40	8 –22
	Late (n = 16)	Mean	73	111	32	19.7
		SD	12.9	6.7	5.4	4.3
		Range	57–106	97–119	22–39	9 –22
Total communication (n = 41)	Early (n = 17)	Mean	38	115	37	20.6
		SD	8.6	4.0	8.3	3.5
		Range	22–53	108 –120	25–58	8 –22
	Late (n = 24)	Mean	74	115	34	20.4
		SD	17.5	4.5	9.0	3.0
		Range	55–106	106 –122	21–58	11–22

Procedure

All tests were administered by a licensed speech pathologist or audiologist at the Indiana University Medical Center. The CP test (Robbins, Renshaw, & Osberger, 1995) was administered by live voice, using three presentation conditions in the following order: A-alone, V-alone, and AV. To eliminate visual cues in the A-alone condition, the clinician covered her face with a black mesh-cloth screen that did not mask the auditory signal.

The CP test is an open-set test of speech perception that measures a child’s ability to understand and comprehend everyday sentences. The CP test consists of 6 sets of 10 sentences each. A different set of 10 sentences is presented in each presentation condition, randomized across children, condition, and testing sessions. The child receives a score of 1 on a sentence if he/she correctly repeats the sentence, correctly answers the question (e.g., responds “July” when asked to repeat the sentence “When is your birthday?”) or responds appropriately (e.g., turns around when asked to repeat the sentence “Turn around.”). A percent correct score is calculated based on the total number of correct trials out of a possible 10 trials in each condition. Note that this test is not administered to young children under the age of 3 years, so very early-implanted children were not evaluated with the CP test until they were old enough to understand the task (i.e., postimplantation interval 3 to 4).

In addition to the CP test, we also carried out analyses of scores obtained from five other behavioral tests of speech and language development that are routinely used to measure outcome and assess benefit of cochlear implantation (Kirk, 2000). All of these measures were administered at the same intervals as the CP test; only the scores available from the 3-year postimplantation interval were used in the present study. The Phonetically Balanced-Kindergarten (PBK) test (Haskins, Reference Note 3) is a live-voice, open-set test used to assess A-alone speech perception. The child hears a spoken word and is then asked to repeat the word aloud. Children’s responses are scored as the percentage of words and/or phonemes correctly repeated. The items on the PBK test are phonetically balanced, monosyllabic words.

The Peabody Picture Vocabulary Test (PPVT) (Dunn & Dunn, 1997) is a closed-set test used to assess receptive vocabulary knowledge. The clinician presents a spoken word to the child, and the child is asked to point to the target word depicted by one of four pictures. This test is administered using the child’s preferred mode of communication, either spoken English or Signing Exact English, which is simultaneously signed and spoken English. That is, target words are presented using only the auditory modality for OC children, but they are simultaneously spoken and signed for the TC children. An age equivalence score was calculated by comparing the raw score to normative data obtained from normal-hearing children and determining the age of most children who receive a similar score.

The Reynell Developmental Language Scales 3rd Edition (RDLS-III) (Edwards et al., 1997; Reynell & Huntley, 1985) is a test used to assess children’s language skills. The receptive language scale consists of 10 subtests that assess skills ranging from word recognition and sentence comprehension to verbal comprehension of ideational content. The expressive language scale consists of three subtests that assess such skills as children’s spontaneous expression of speech and their ability to describe a novel picture. In the present study, the receptive and expressive language scores were obtained using the child’s preferred mode of communication. The children received credit for signed and/or spoken correct responses. Raw scores on the RDLS-III scales were converted into age equivalence scores based on normative data obtained from normal-hearing children, which reflect the age that most typical-developing children obtain similar scores.

Finally, the Beginner’s Intelligibility Test (BIT) (Osberger, Robbins, Todd, Riley, & Miyamoto, 1994) was administered to obtain a measure of the child’s speech intelligibility. The child is asked to repeat 10 sentences presented by the clinician. Audio recordings of children’s speech productions are then presented to three naïve adult listeners who are asked to transcribe what the child said. A speech intelligibility score is computed based on the average number of words transcribed correctly by the three listeners.

Procedure

Data used in this study were obtained from a clinical population enrolled in a larger longitudinal research project. As a consequence, not all children could be tested at each interval. To deal with the problem of missing data, we used the SAS Mixed Procedure (Wolfinger & Chang, Reference Note 4) to analyze the fixed effects in this study. The traditional repeated-measures analysis of variance test, commonly used to analyze variance in longitudinal designs, eliminates participants with missing data. However, systematically eliminating participants with missing data in clinical populations can lead to skewed or biased results as well as an underestimation of variability (Schafer & Graham, 2002). Because the data set used for the present study consisted of repeated measures from the same participants, a maximum-likelihood estimation method, such as the Mixed Procedure, can use all data available to create a model without eliminating any participants (Schafer & Graham, 2002).

Results

The effects of duration of implant use, communication mode (OC versus TC), age at implantation (early versus late), and presentation format (A-alone, V-alone, AV) were the main effects included in the initial analyses.

CP Accuracy Scores

Figures 1 and 2 show the longitudinal results obtained as a function of communication mode (OC, TC) for the three presentation formats (A-alone, V-alone, AV) over 5 years of cochlear implant use (Figure 1A, top panel) and for early-implanted and late-implanted children (Figure 2A, top panel). The abscissa shows the years of implant use (0 = before implantation) and the ordinate shows the percentage of sentences correctly repeated on the CP test. The error bars represent standard error. Table 2 shows the number of participants tested at each interval.

Fig. 1.

Raw scores from the Common Phrases Test as a factor of communication method under auditory-alone, visual-alone, and audiovisual conditions for oral communication (OC) and total communication (TC) children. A, Mean percent correct sentence comprehension over time under auditory-alone, visual-alone, and audiovisual conditions; B, auditory and visual gain scores. Error bars represent standard error.

Fig. 2.

Raw scores from the Common Phrases Test as a factor of age at implantation under auditory-alone, visual-alone, and audiovisual conditions for early-implanted (early) and late-implanted (late) children. A, Mean percent correct sentence comprehension over time under auditory-alone, visual-alone, and audiovisual conditions; B, auditory and visual gain scores. Error bars represent standard error.

TABLE 2.

Number of participants at each testing interval

			Implant use (yr)
			0	1	2	3	4	5
A	OC	Early	1	3	6	12	12	5
		Late	8	13	13	12	7	6
	TC	Early	1	2	6	12	17	9
		Late	13	15	18	15	8	5
V	OC	Early	0	3	5	12	13	6
		Late	6	11	9	9	7	6
	TC	Early	1	1	4	12	17	9
		Late	7	13	17	15	9	6
AV	OC	Early	0	3	5	12	12	5
		Late	8	14	11	10	5	5
	TC	Early	1	3	7	12	17	8
		Late	13	16	18	14	7	3

A = Auditory-alone; V = visual-alone; AV = audiovisual; OC = oral communication; TC = total communication; Early = early-implanted; Late = late-implanted.

We found statistically significant main effects of duration of implant use [F(5, 565) = 46.45, p < 0.0001], communication mode [F(1, 158) = 12.61, p = 0.0005], age at implantation [F(1, 90.9) = 6.15, p = 0.015], and presentation format [F(2, 516) = 117.81, p < 0.0001]. Performance of all children, regardless of communication mode, age at implantation, and presentation format, improved over time from before implantation to 5 years after implantation. Also, OC children consistently performed better overall than TC children, and late-implanted children performed better overall than early-implanted children. Finally, performance was better in the combined AV presentation condition compared with either the A-alone and V-alone presentation conditions.

We also found statistically significant two-way interactions between communication mode and duration of implant use [F(5, 565) = 3.87, p = 0.0018], age at implantation and duration of implant use [F(5, 565) = 3.96, p = 0.0015], and age at implantation and presentation format [F(2, 516) = 41.22, p < 0.0001]. Although OC children performed better than TC children across presentation conditions in the early intervals, TC children’s performance was more similar to OC children’s performance after 5 years of cochlear implant use. Similarly, whereas late-implanted children started out performing better than early-implanted children, the early-implanted children performed at a similar level overall to late-implanted children after 4 years of implant use. Early-implanted children performed better than late-implanted children in the A-alone condition, late-implanted children performed better than early-implanted children in the V-alone condition, and both groups of children performed at similar levels in the AV condition (see Figure 2A).

The three-way interaction between communication method, duration of implant use, and age at implantation was also statistically significant [F(6, 467) = 2.24, p = 0.0382]. Early-implanted children’s performance improved over time with cochlear implant use regardless of communication method. However, late-implanted children’s performance improved over time only if they used the OC method. That is, late-implanted TC children’s performance remained essentially flat over time.

To ensure that children’s performance in the pre-implantation interval did not unduly influence the results, we also analyzed the data excluding the pre-implantation interval. Once again, we found statistically significant main effects of duration of implant use [F(4, 502) = 55.50, p < 0.0001], communication mode [F(1, 93.2) = 18.98, p < 0.0001], and presentation format [F(2, 470) = 114.07, p < 0.0001], as well as interactions between duration of implant use and age at implantation [F(4, 502) = 6.19, p < 0.0001] and between presentation format and age at implantation [F(2, 470) = 36.92, p < 0.0001]. However, the interaction between communication mode and duration of implant use was not significant, nor was the three-way interaction between communication mode, duration of implant use, and age at implantation. Thus, when the pre-implantation scores were excluded from the analysis, OC and TC children’s performance across the three presentation conditions improved at similar rates over time after receiving a cochlear implant, regardless of age at implantation.

Finally, a linear trend analysis showed that children’s performance increased at a greater rate over time after cochlear implantation in A-alone and AV presentation conditions compared with the V-alone presentation condition [F(2, 560) = 13.65, p < 0.0001]. The change in performance over time was significantly steeper in the A-alone and AV conditions than in the V-alone condition [A-alone: t(560) = 4.71, p < 0.0001, AV: t(561) = 4.43, p < 0.0001]. There was no significant difference between the slopes in the A-alone and AV conditions.

CP Audiovisual Gain

Although the accuracy scores in the three presentation formats are informative and reveal change over time, previous research has shown that audiovisual speech perception is more complex than just the simple addition of auditory and visual cues to speech (Bernstein, Demorest, & Tucker, 2000; Massaro & Cohen, 1999; McGurk & MacDonald, 1976; Sumby & Pollack, 1954). In normal-hearing adult listeners, the gain in performance from combined AV information is superadditive in nature (Sumby & Pollack, 1954). That is, the observed performance in the AV presentation condition is greater than the simple sum of the scores in the unimodal A-alone and V-alone conditions. Thus, it is important to examine the relative gains in AV speech perception that result from the additional visual information compared with the A-alone condition and the additional auditory information compared with the V-alone condition. Two scores are routinely computed: visual gain and auditory gain.

Visual gain is the relative increase in AV speech perception performance due to the addition of visual information to the auditory signal (Sumby & Pollack, 1954). We computed visual gain, or R_a, by combining accuracy scores in the A and AV presentation conditions, using the formula R_a = (AV − A)/(100 − A).

Thus, R_a measures the gain in performance in the AV condition relative to performance in the A-alone condition, normalized relative to the amount by which speech intelligibility could have improved above A-alone scores.

Auditory gain, on the other hand, is the relative increase in AV speech perception performance due to the addition of auditory information to the visual signal. We computed auditory gain, or R_v, by combining accuracy scores in the V-alone and AV pre-sentation conditions using the formula R_v = (AV − V)/(100 − V).

Although most studies of AV speech perception with normal-hearing listeners report only visual gain scores, the use of auditory gain measures is appropriate for deaf children because they typically rely more on visual than auditory input for speech perception (see Grant & Seitz, 1998; Seewald et al., 1985). Moreover, as these deaf children gain experience with sound via their cochlear implants, their auditory gain scores may also change over time after implantation.

Figures 1 and 2 show the auditory and visual gain scores obtained over 5 years of cochlear implant use for OC and TC children (Figure 1B, bottom panel) and for early-implanted and late-implanted children (Figure 2B, bottom panel). Table 3 shows the number of participants with gain scores at each interval. It is important to note that children who have 100% accuracy scores in either the A-alone or V-alone condition have no room to improve their scores in the AV condition. These children (0 to 5 in each interval, the majority of whom were OC) were eliminated from the present analyses. Using the SAS Mixed Model, we assessed the effects of duration of implant use, communication mode (OC versus TC), age at implantation (early versus late), and type of gain (R_v versus R_a).

TABLE 3.

Number of participants at each testing interval

			Implant use (yr)
			0	1	2	3	4	5
Ra	OC	Early	0	3	4	10	8	3
		Late	8	11	11	9	5	4
	TC	Early	1	2	6	12	15	8
		Late	13	13	16	14	7	3
Rv	OC	Early	0	3	5	11	12	5
		Late	5	10	8	8	5	4
	TC	Early	1	1	4	12	17	8
		Late	7	13	15	13	7	3

Ra = Visual gain; Rv = auditory gain; OC = oral communication; TC = total communication; Early = early-implanted; Late = late-implanted.

We found a significant main effect of duration of implant use [F(5, 318) = 10.23, p < 0.0001]. Both auditory and visual gain increased from before implantation to 5 years after implantation. We also found a marginally significant effect of communication mode [F(1, 75) = 3.54, p = 0.06], showing a trend for OC children to display greater overall auditory and visual gain compared with TC children. Type of gain and age at implantation effects were not significant. Finally, we found a significant interaction between type of gain and age at implantation [F(1, 282) = 28.73, p < 0.0001]. Early-implanted children had greater auditory gain than late-implanted children, but late-implanted children had greater visual gain than early-implanted children. We also carried out these analyses excluding the pre-implantation interval. We found that the main effect of duration of implant use [F(4, 289) = 8.24, p < 0.0001] as well as the interaction between type of gain and age at implantation [F(1, 252) = 25.59, p < 0.0001] remained statistically significant, but we did not find a significant main effect of communication mode.

Correlations Between CP and Outcome Measures of Speech and Language

To determine the relation between scores on the CP test and the five clinical outcome measures of speech and language skills, we performed correlation analyses on these scores for all children at the 3-year postimplantation interval. This interval was chosen because it contained scores from the largest number of children. In the earlier cross-sectional study from our laboratory with a smaller sample of children, Lachs et al. (2001) assessed the relationship between the CP test and outcome measures at the 2-year postimplantation interval. To obtain an adequate sample size for the correlation analyses, we combined the scores for early-implanted and late-implanted children.

As shown in Table 4, OC children’s performance in the A-alone condition was significantly correlated with outcome measures of open-set word and phoneme recognition (PBK phonemes and words), vocabulary (PPVT), language (RDLS-III expressive/receptive), and speech intelligibility (BIT). This pattern of results indicates that OC children who had higher scores in the CP A-alone condition also tended to have higher scores on all of the clinical outcome measures. In contrast, TC children’s performance in the A-alone condition was only significantly correlated with measures of open-set word and phoneme recognition (PBK) and speech intelligibility (BIT). Although results for the A-alone condition were correlated with other speech and language skills for both communication methods, the correlations were consistently much stronger for the OC compared with TC children. These differences reached statistical significance for measures of vocabulary (PPVT, p = 0.06) and language skills (RDLS-III, expressive [p = 0.05] and receptive [p < 0.0001]). Also, more of the outcome measures were correlated with the CP test for the OC children than TC children.

TABLE 4.

Correlations for common phrases and outcome measures 3 years after implant

		OC			TC
		A	V	AV	A	V	AV
PBK-words	r	0.635†	0.313	0.559†	0.561†	−0.234	0.388
	N	24	21	22	25	25	24
PBK-phonemes	r	0.657†	−0.062	0.471*	0.705†	−0.250	0.197
	N	24	21	22	25	25	24
PPVT	r	0.422*	0.628†	0.577†	−0.009	0.484*	0.310
	N	23	20	21	25	25	24
RDLS Expr	r	0.704†	0.572*	0.789†	0.214	0.454*	0.500*
	N	13	14	13	20	20	20
RDLS Rec	r	0.688†	0.385	0.815†	0.018	0.332	0.406
	N	14	14	12	20	21	20
BIT	r	0.734†	0.557*	0.734†	0.611†	0.038	0.526*
	N	16	15	16	17	18	17

^*p < 0.05,

^†p < 0.01.

Both OC and TC children’s performance in the V-alone condition was significantly correlated with vocabulary scores (PPVT) and expressive language skills (RDLS-III). In addition, OC children’s V-alone performance was also significantly correlated with speech intelligibility (BIT). Although the correlations were consistently stronger for the OC children compared with TC children, the only statistically significant difference was in the measure of speech intelligibility (BIT, p = 0.06).

Finally, OC children’s performance in the AV condition was significantly correlated with measures of open-set word and phoneme recognition (PBK), vocabulary (PPVT), expressive and receptive language (RDLS-III), and speech intelligibility (BIT). On the other hand, TC children’s performance in the AV condition was correlated only with expressive language (RDLS-III) and speech intelligibility (BIT). Similar to the pattern of performance in the A-alone and V-alone conditions, open-set sentence comprehension skills were strongly correlated with other speech and language skills. A trend of stronger correlations for OC children compared with TC children was observed, although this difference reached statistical significance only in the case of receptive language scores (RDLS-III, p = 0.04).

Correlations Between Pre-implant and Postimplant Outcome Measures

Further correlation analyses were conducted to determine whether children’s pre-implantation scores on the CP test could predict their skills on speech and language outcome measures after 3 years of implant use. To obtain an adequate sample size for this analysis, the correlations were carried out by combining the scores across the communication mode and age at implantation variables. Because of floor effects at pre-implantation, we also excluded the A-alone condition in these analyses. As shown in Table 5, all correlations were strong, positive, and significant. Interestingly, in all cases the V-alone CP pre-implant measure was the strongest predictor of later speech and language outcomes after 3 years of cochlear implant use.

TABLE 5.

Correlations for preimplant common phrases and 3 years after implant outcome measures

		V	AV
PBK-words	r	0.906†	0.724†
	N	10	17
PBK-phonemes	r	0.814†	0.760†
	N	10	17
PPVT	r	0.704*	0.679†
	N	9	16
BIT	r	0.883*	0.839†
	N	6	11

^*p < 0.05,

^†p < 0.01.

To test whether other pre-implantation outcome measures from these children would predict 3-year postimplantation performance on these outcome measures, several additional correlation analyses were conducted using each of these outcome measures (PBK-words, PBK-phonemes, PPVT, and BIT) as a pre-implantation predictor. A few sporadic correlations (positive and negative) emerged, but none were stronger than the correlations between the V-alone condition of the CP test and various outcome measures.

Finally, to determine whether measures of general intelligence could predict postimplantation speech and language skills, we carried out a correlation analysis between pre-implantation scores on the performance subscale of the Wechsler Intelligence Scale for Children, 3rd Edition (WISC-III) (Wechsler, 1991) and 3-year postimplantation performance on several speech and language outcome measures. The results of this analysis also failed to reveal any statistically significant correlations. In short, pre-implantation lipreading performance on the CP test may serve as a reliable behavioral marker that can be used to predict subsequent speech and language performance and measure benefit after implantation.

Discussion

The present analysis of the longitudinal data obtained from a large group of prelingually deaf children over a period of 5 years after cochlear implantation revealed several important patterns of performance. Children consistently performed better in the AV presentation conditions than in the A-alone and V-alone conditions. These findings were expected, based on the results from the recent studies on deaf children with cochlear implants carried out by Lachs et al. (2001) and Bergeson et al. (2003), as well as results from earlier studies of AV speech perception in prelingually deaf children and postlingually deaf adults who received cochlear implants (e.g., Geers et al., 2003; e.g., Kaiser et al., 2003; Tyler, Parkinson, Woodworth, Lowder, & Gantz, 1997).

The longitudinal data in the present study, however, provide new findings on the development of AV speech perception skills in a large group of hearing-impaired children with cochlear implants and how they change over time after implantation. The results of the present study revealed that AV sentence comprehension skills consistently improved over the 5-year period after cochlear implantation. We also found that the A-alone and AV scores improved at a greater rate than the V-alone scores over the 5-year period after cochlear implantation. This finding was not surprising because cochlear implants primarily improve audition rather than vision.

The present study also revealed that prelingually deaf children with cochlear implants displayed reliable increases in their auditory and visual gain scores over the 5 years after cochlear implantation. The children also showed multimodal gain and enhancement when speech was presented in an AV format compared with A-alone and V-alone formats. Surprisingly, these children displayed similar visual and auditory gain scores. They did not derive more benefit from the addition of the auditory signal compared with the benefit received from the addition of lipreading cues. However, children who reached ceiling in either the A-alone or V-alone condition were eliminated from the analyses, which could have influenced the results.

What do the AV scores on the CP test tell us about the development of speech and language skills in this clinical population? How are the measures of AV speech perception related to other outcome measures of speech perception, speech intelligibility, and language processing? To answer these questions, we carried out two different sets of correlations. First, to assess the validity of the measures of AV speech perception, we looked at the intercorrelations between scores on the CP test and a small, representative set of clinical outcome measures of speech and language obtained 3 years after cochlear implantation. We found strong positive correlations of the CP test scores with several independent measures of spoken word recognition, receptive vocabulary development, expressive and receptive language, and speech intelligibility. Moreover, the pattern of these correlations was much stronger for OC than TC children, although both groups showed similar overall patterns of change over time after implantation.

The strong intercorrelations between AV speech perception performance on the CP test and other behavioral tests used to measure speech and language benefit after 3 years of implant use suggest that these outcome measures share a common underlying source of variance. This pattern of correlations is important theoretically because it suggests that the same sensory, cognitive, and linguistic processes used to carry out AV speech perception are also used in other language processing tasks (see also Blamey et al., 2001). Thus, scores on the CP test generalize beyond the specific experimental paradigm and are not merely task-specific measures of isolated and independent perceptual skills. All of the tasks included in the present study to measure speech and language outcomes involve rapid encoding of temporal sequences followed by immediate reproduction of a phonological pattern (Gupta & MacWhinney, 1997). Higher-level cognitive processes such as perception, attention, learning, memory, as well as phonological and lexical coding are assumed to play a contributing role in all of these traditional clinical outcome measures (see Pisoni, 2000; Pisoni & Cleary, 2003; Pisoni, Cleary, Geers, & Tobey, 2000).

To identify possible early predictors of performance and benefit with use of a cochlear implant, we also carried out a set of correlations between the pre-implantation CP V-alone and AV scores and these same speech and language outcome measures after 3 years of implant use. We found that pre-implantation CP scores were strongly correlated with open-set word recognition, vocabulary knowledge, and speech intelligibility scores obtained after 3 years of implant use. The strongest correlations between pre-implantation performance on the CP test and 3-year postimplant outcome measures were obtained with the V-alone measures. It is possible that deaf children who are most efficient at making use of any source of sensory information about speech are able to use the only available cues before cochlear implantation, that is, lipreading cues, and then once their hearing is restored via a cochlear implant, they are able to make use of both auditory and visual cues to perceive linguistically significant differences.

The present findings suggest that pre-implantation measures of AV speech perception may provide behavioral markers that can be used to predict children’s speech and language benefit from their cochlear implants after several years of implant use (see also Knutson et al., 1991; Pressman, Pipp-Siegel, Yoshinaga-Itano, & Deas, 1999; Tait, Lutman, & Robinson, 2000; Yoshinaga-Itano, 2000). Although the CP pre-implantation correlations reported here are based on small sample sizes, the results suggest that simple lipreading measures obtained before implantation may reveal the operation of fundamental perceptual processes that are used to recover linguistically significant information about speech articulation.

Two previous studies have attempted to identify pre-implantation predictors of success and benefit with cochlear implants in both adults and children. Knutson et al. (1991) found that postlingually deaf adults’ pre-implantation performance on a visual monitoring task predicted audiological outcome after 18 months of implant use. The authors suggest that the cognitive processing operations and skills needed to rapidly extract information from sequentially arrayed visual patterns may also be used in processing complex auditory signals, and may underlie the successful use of a cochlear implant.

In a study of prelingually deaf children with cochlear implants, Tait et al. (2000) found that preverbal communicative and autonomy behaviors that are present before implantation are associated with outcome measures of speech perception and language comprehension 3 years after implantation. To obtain preverbal measures of communication, they recorded turn-taking behavior of deaf children before cochlear implantation and a known adult. Deaf children who initiated turns more often within a conversation received higher pre-implantation autonomy scores, which were subsequently found to be positively correlated with scores on closed-set sentence perception and sentence repetition tasks after 3 years of cochlear implant experience.

The present findings on the development of AV sentence comprehension skills are of clinical interest because they suggest new behavioral measures that could be used to assess and predict performance in this clinical population. AV speech perception skills may function as behavioral markers of the development of speech and language.

We also found substantial effects of early sensory experience associated with the language-learning environment on performance in the CP test. Children who used OC methods displayed consistently higher scores on the CP test than children who used TC methods, similar to results of previous studies. This difference was observed in all three presentation conditions. We were surprised to find that OC children obtained higher CP scores even before implantation compared with the TC children. It is unlikely that this result is due to differences in residual hearing between the two groups of children: A t-test revealed no statistically significant difference in pre-implant unaided pure-tone threshold averages (see Table 1). Some other factor or set of factors may be responsible for these differences.

One factor is that classification of communication method is based primarily on parental report and confirmed by the child’s educational setting. Parents are encouraged to enroll their child in one of these educational settings when they first discover their child’s hearing loss. We have no way of knowing whether children are enrolled in OC or TC methods, based on any systematic characteristics, such as hearing thresholds or cognitive maturity.

It is likely that the pre-implantation differences between the two groups of children observed in all conditions of the CP test may reflect the strong emphasis on both the visual and auditory properties of spoken language that is the hallmark of the aural-oral approach to deaf education (e.g., Carney & Moeller, 1998). Hearing-impaired children who are placed in aural-oral educational environments at an early age after being diagnosed with a hearing loss are fully engaged in speaking and listening activities everyday by their teachers and caregivers (e.g., Yoshinaga-Itano, 2000). Thus, the surrounding language-learning environment, like the sensory environment of a normal-hearing, typically developing child, is fundamentally multisensory in nature and is specifically focused on and oriented to spoken language processing activities such as speech perception, speech production, and spoken language comprehension. Because of the emphasis on gestures and manual signs, TC children may have significant cognitive and linguistic disadvantages in deriving optimal benefits from the limited auditory information provided by their cochlear implants and combining this source of information with the visual speech information provided by talkers in their language-learning environments.

Why do these differences in communication mode and early experience occur and what factors are responsible for the better performance of OC children under these conditions? Several factors related to early experience and activities may be responsible for the large and consistent differences observed in performance between the two groups of children in the present study and in other studies reported in the literature (e.g., Archbold et al., 2000; Bergeson & Pisoni, 2004). One consequence of using simultaneous communication methods with TC children is the presence of competition between speech and manual communication for limited attention and processing resources in working memory, both of which are assumed to play major roles in all language comprehension and word recognition tasks (Baddeley, Gathercole, & Papagno, 1998; Doherty-Sneddon, Bonner, & Bruce, 2001).

Another factor is that the language samples that deaf children in TC environments are exposed to are more likely to be impoverished models of the target language (Moeller & Luetke-Stahlman, 1990; Spencer, 1993; Swisher & Thompson, 1985). For example, almost all hearing-impaired children who receive cochlear implants have hearing parents who do not have a good working knowledge of sign language or manual communication methods. Finally, the combination of competition and impoverished linguistic models may lead to more limited exposure to speech in TC children than OC children. As a result, TC children may have more difficulty encoding, maintaining, and retrieving phonological representations of spoken words and sentences (e.g., Burkholder & Pisoni, 2003; e.g., Pisoni & Cleary, 2003).

The present analysis of performance on the CP test also revealed an effect of age at implantation. Surprisingly, we found that children who were implanted at an older age, that is, had a longer period of deafness before implantation, initially performed much better than children implanted earlier in life, that is, had a shorter period of deafness (see Figure 1B). This main effect was qualified by interactions with the other factors and disappeared when the pre-implantation interval was excluded from the analyses. It is important to note here that at each testing interval, the late-implanted children were also on average 3 years older than the early-implanted children. Moreover, the CP test cannot be administered to very young children, as demonstrated by the small number of participants in the pre-implantation interval. These sampling effects could be responsible for the finding that late-implanted children performed better on the CP test than early-implanted children.

An important result, however, was the finding that early-implanted children performed better than late-implanted children in the A-alone and AV presentation conditions, whereas late-implanted children performed better than early-implanted children in the V-alone presentation condition. Moreover, early-implanted children had higher auditory gain scores than late-implanted children, but late-implanted children had higher visual gain scores than early-implanted children in these tests (see Figure 2B). Thus, children who experienced a longer period of profound deafness before implantation were better lipreaders than children who were profoundly deaf for shorter periods of time. Tillberg, Rönnberg, Svärd, & Ahlner (1996) found that adults with early-onset deafness performed more accurately on V-alone word and sentence recognition tests than adults with late-onset deafness. The present finding with prelingually deaf children replicates the earlier report that adults who were deaf for longer periods of time were better lip-readers than adults who were deaf for shorter periods of time.

Age at implantation has been found to be an extremely important demographic factor in studies on cochlear implantation in hearing-impaired children because a period of prolonged auditory deprivation at an early point in neural and cognitive development may result in significant reorganization in the central auditory system. Early cochlear implantation may lead to restoration of auditory abilities due to neural plasticity. However, such plasticity and the potential for neural reorganization after implantation has been found to occur only during critical or sensitive periods of development (e.g., up to 6 years of age) (Beggs & Foreman, 1980; Bruer, 2001; Moore, 2002; Neville & Bruer, 2001; Robinson, 1998; Shepherd & Hardie, 2001; Shepherd, Hartmann, Heid, Hardie, & Klinke, 1997; Wolff & Thatcher, 1990). A recent study of cortical response latencies to speech in congenitally deaf children and adults with cochlear implants reported maximal plasticity in a sensitive period up to 3.5 years of age, with plasticity remaining in some children even up to 7 years of age (Sharma, Dorman, & Spahr, 2002).

The pattern of results that we found in this study suggests that even before a hearing-impaired child receives a cochlear implant, it may be possible to obtain reliable behavioral measures based on V-alone speech perception using lipreading measures that will successfully predict outcome and benefit after implantation. Pre-implantation measures of AV perception and lipreading skills in hearing-impaired infants and young children may therefore provide some initial measures of the coupling between auditory and visual sensory systems and the potential for unity and convergence of neural processing of common phonetic events between these two different sensory systems, even in the absence of sound during early development that is a result of a period of profound deafness. Investigations of AV speech perception may therefore offer new insights into the underlying neural basis of central auditory and linguistic processes used in speech and language perception. Studies of multisensory speech perception can provide both clinicians and researchers with new behavioral measures of how hearing-impaired children perceive speech and understand spoken language after receiving a cochlear implant.