Assessing Speech Intelligibility in Children with Hearing Loss: Toward Revitalizing a Valuable Clinical Tool

Abstract

Background

Newborn hearing screening, early intervention programs, and advancements in cochlear implant and hearing aid technology have greatly increased opportunities for children with hearing loss to become intelligible talkers. Optimizing speech intelligibility requires that progress be monitored closely. Although direct assessment of intelligibility has been a cumbersome undertaking in the past, advancements in digital recording technology and expanded strategies for recruiting listener-judges can make this tool much more practical in contemporary school and clinical settings.

Purpose

The main purposes of this tutorial are to present a rationale for assessing children’s connected speech intelligibility, to review important uses for intelligibility scores, and to describe time-efficient ways to estimate how well children’s connected speech can be understood. This information is offered to encourage routine assessment of connected speech intelligibility in pre-school and school-age children with hearing loss.

Readily intelligible connected speech is the ultimate goal of speech intervention for children with hearing loss. It seems reasonable, then, that children’s progress toward this important goal would be monitored closely. However, as Monsen noted nearly 30 years ago, “A strange fact about the contemporary education of the hearing-impaired is that the intelligibility of their speech is seldom measured.” (1981, p. 845). Based on the current author’s discussions with educational administrators, speech-language pathologists, and teachers of children with hearing loss, Monsen’s observation seems to apply to today’s schools as well. This tutorial provides a rationale for assessing children’s connected speech intelligibility, a review of important uses for intelligibility scores, and some practical and time-efficient ways to estimate how well children’s speech can be understood.

A Rationale for Direct Assessment of Children’s Connected Speech Intelligibility

Speech intelligibility is “that aspect of speech-language output that allows a listener to understand what a speaker is saying” (Nicolosi, Harryman, & Kresheck, 1996). Highly intelligible speech allows “naïve listeners to understand most of the child’s speech at first introduction” (Monsen, 1981; p. 849 – 850). As intelligibility decreases, listeners experience greater difficulty in understanding what they hear, until just a few--or even no--words are recognized. Research completed during the 1960s – 1980s revealed that the speech of children with severe to profound hearing losses was approximately 20% intelligible on average (see Osberger, 1992 for review). Clearly, low levels of intelligibility can lead to substantial communication difficulties at home, in school, and in other everyday situations. The widespread adoption of newborn hearing screening, increased availability of parent-infant intervention programs, and advancements in sensory aid technologies have increased optimism that today’s children with hearing loss can become readily intelligible talkers.

This positive outlook has been bolstered by sizable gains in intelligibility after children receive sensory aids at relatively young ages. For example, Chin, Tsai, and Gao (2003) found that children with a mean age of 3;2 (years; months) at the time of cochlear implantation achieved an average intelligibility score of 34.5% after 28 months of cochlear implant (CI) experience. This score is an increase of nearly 15 percentage points over the 20% level for hearing aid users noted earlier, and the authors noted that further gains were anticipated with continued CI use. The benefits of such long-term CI use have been observed in the speech of children studied by Peng, Spencer, and Tomblin (2004). The participants in this study averaged 72% intelligible speech after 7 years of CI use, despite implantation at relatively older ages (M = 5;1 years). Moderately high scores have also been observed in younger CI users (Mean age-at-implant = 3;5) whose speech was 63.5% intelligible after 5.5 years of CI experience (Tobey, Geers, Brenner, Altuna, & Gabbert, 2003).

Children who are implanted at even younger ages appear to make more rapid progress than those implanted at older ages. Ertmer (2008) found that, on average, intelligibility scores increased to 62% after just 36 months of CI experience, in children who were implanted by their third birthdays. Although large intelligibility gains have been documented mainly in young CI recipients, the advantages of early identification and intervention have also been noted for young hearing aid (HA) users. Markides (1986) found that children who began using hearing aids during the first 6 months of life achieved higher intelligibility ratings than those who received HAs later in childhood. In addition to verifying much higher intelligibility levels than seen in the latter part of the 20^th century, studies of CI users showed that children are likely to develop intelligible speech at different rates and that progress remains slower than seen in children who are typically developing (Chin, Tsai, & Gao, 2003; Ertmer, 2008). The latter findings support the need to monitor intelligibility on a regular basis.

Speech assessments that rely mainly on clinician impressions and word-based articulation tests appear to be inadequate for monitoring the development of intelligible connected speech. There are several problems with this approach. First, clinicians and teachers of children with hearing loss are quite familiar with the speech patterns of the children they serve. As a result, their estimates of connected speech intelligibility are likely to be higher than those of listeners who are unfamiliar with children who have hearing loss (see Osberger, 1992). Over-estimation might lead to reduced intervention emphasis on the “clarity” of phrases and sentences, and limit improvements in connected speech intelligibility.

Further, recent research has shown that word-based articulation tests are not dependable estimators of connected speech intelligibility. Ertmer (in press) administered the popular “Sounds in Words” subtest of the Goldman-Fristoe Test of Articulation- 2 (GFTA-2; Goldman & Fristoe, 2000) to 44 children with hearing losses. Percent-correct scores were calculated for seven word-based variables (i.e., initial, medial, and final consonants, consonant clusters, initial consonants without clusters, vowels, and total GFTA-2 targets). The children’s productions of short sentences were also recorded and played for unfamiliar, adult listeners so that the percent of words identified by the listeners could be calculated for each child. Results showed that, on average, percent-correct scores for the word-based variables were considerably greater than the percent of words identified in short sentences by naïve listeners (86.7% Vs 54.5%, respectively). Multiple-regression analysis also revealed that word-based scores accounted for ≤25% of the variability in children’s intelligibility scores--even when three sets of word-scores were combined as predictors. It was concluded that word-based articulation scores were poor predictors of connected speech intelligibility. That is, children may correctly articulate a variety of consonants and vowels in single words but still not have readily intelligible connected speech. Although word articulation tests are easy to use and yield useful information for selecting intervention priorities, connected speech samples appear to provide much closer estimates of children’s intelligibility.

In summary, newborn hearing screening, early intervention programs, and advancements in sensory aid technology have greatly increased the potential of children with hearing loss to become intelligible talkers. Adjustments to clinical practices are needed to ensure that children take advantage of these advancements. In particular, the limitations of clinician impressions and word-based articulation test scores highlight the need to assess connected speech intelligibility directly. The routine use of such assessments can provide crucial information for intervention and educational planning.

Uses for Speech Intelligibility Scores

There are at least three compelling uses for speech intelligibility scores. The first has to do with sensory aid functioning and children’s speech perception abilities. Research has shown that speech intelligibility and auditory speech perception scores are strongly correlated in children with cochlear implants and those with hearing aids (Blamey, Sarant, Paatsch, Barry, Bow, Wales, R. J., et al, 2001; Svirsky, Robbins, Kirk, Pisoni, & Miyamoto, 2000). The extent to which speech production improves is, therefore, likely to be an indicator of the auditory perceptual benefits received from sensory aids in young sensory aid recipients who do not have secondary learning or speech motor disabilities. Thus, improvements in intelligibility after CI or HA fitting can provide indirect evidence of sensory aid benefit. In contrast, a lack of improvement in speech intelligibility, in combination with low or unimproved speech perception scores, might indicate that children are not receiving full benefit from their sensory aids. In short, intelligibility scores can provide important supplemental information for decisions regarding the adjustment or replacement of sensory aids.

Intelligibility scores can also guide intervention planning and improve clinical accountability. By assessing connected speech intelligibility at regular intervals (e.g., every 6 or 12 months), clinicians can measure children’s progress and determine whether a greater emphasis on connected speech is needed. As McReynolds (1981) noted, the transfer of phonological learning from words to connected speech requires both higher-level understanding of spoken communication and the ability to generalize learning to more complex speech tasks. Specialized strategies may be needed to facilitate the transfer of articulation training targets to connected speech, and to develop the self-regulation skills needed to become a readily intelligible talker (see Ertmer & Ertmer, 1998 for suggestions). Regarding accountability, intelligibility scores can be used to develop short-term goals in Individual Educational Plans (IEPs), document progress toward intelligible connected speech, and determine whether children are ready for dismissal from speech-language intervention programs.

Finally, intelligibility scores provide crucial information for determining whether children are likely to be successful oral communicators in mainstream educational placements. Clinicians who have completed a direct assessment of connected speech intelligibility can use ratings or percent-intelligible scores to better predict how well children’s speech will be understood by adults and classmates in integrated school settings.

Clinicians who routinely assess connected speech intelligibility gain useful information for monitoring sensory aid benefit, individualizing speech instruction, and making mainstreaming decisions. Ideas for practical and time-efficient assessments will be discussed next.

Speech Intelligibility Assessments

Scaling Procedures

Two main kinds of speech intelligibility assessments have been used in research and clinical settings: scaling and item identification (see Kent, Weismer, Kent, & Rosenbek, 1989; Osberger, 1992 for reviews). Scaling consists of asking listeners to rate speech samples (e.g., sentences or spoken narratives) along a continuum of intelligibility. For example, a 10 point-scale can represent a continuum between the lowest and highest levels of intelligibility. Descriptors such as “not at all,” “seldom,” “sometimes,” “most of the time,” and “always” can also be used to estimate how often speech is understood. Scaling is implemented by audio-recording a speech sample, playing it for listeners with normal hearing, and asking them to select a number or a descriptor to indicate how well they understood the sample.

Scaling is quick and relatively easy to complete, but it has several drawbacks. First, listeners may have different internal criteria when rating speech samples. For example, a “6” may mean “pretty good” to one listener, but “not very good” to another. It is difficult to characterize intelligibility when numeric choices have unclear meanings. In addition, scaling is not sensitive to differences among speech samples that fall in the middle range of intelligibility (Samar & Metz, 1988). For example, listener ratings may not distinguish between a child whose speech is 30% intelligible and one whose speech is 60% intelligible. This limitation makes it difficult to verify improvement until a high level of intelligibility is achieved. In summary, scaling can provide a quick estimate of children’s intelligibility, but differences in listener’s internal criteria and limited sensitivity for changes within the mid-range of intelligibility make ratings difficult to interpret.

Several adaptations can make scaling more useful in clinical settings. Concerns about differences in listeners’ internal criteria can be lessened by asking the same listeners to be raters for several children. In this way, the same internal standards are used across a group. It is essential, however, to record different speech materials (i.e., sentence lists) for each child, so the listener remains unfamiliar with the content of each sample. In addition, using the same listener(s) to rate subsequent samples produced by the same child (e.g., at 6 month intervals) is likely to be more reliable than using different listeners each time. Concerns about interpreting results can also be lessened by using clear descriptors such as “no words were understood,” “a few words were understood,” “approximately half of the words were understood,” “most of the words were understood,” and “almost all of the words were understood,” rather than a numeric scale with unspecified values (Schiavetti, Metz, & Sitler, 1981). Although not sensitive to small improvements and mid-range progress, playing samples for the same listeners and using rating scales with clear descriptors can give an indication of children’s progress over time.

Item Identification Procedures

Open-set item identification assessments require listeners to write down the words that they understand from speech samples. To assess connected speech, listeners are presented with audio-recordings of unfamiliar sentences and asked to write down the words they understand in each sentence. To keep attention high, they are also asked to make an X for words they do not understand. The listener’s written responses are scored for the number of times they match the words produced from a set of sentences so that a percent-intelligible score can be calculated for each sample. Listener scores are averaged together whenever the same sample is played for more than one person. Presenting a set of 10 sentences requires slightly more time than scaling, but can be completed in less than10 minutes under normal conditions. Finally, open-set item-identification tasks have an advantage over rating scales in that, rather than relying on subjective impressions, they measure the actual number of words that are understood by listeners. Although item-identification procedures are slightly more complex than scaling, they provide a quantifiable measure of how well the listener actually understands what the speaker is saying (Nicolosi, Harryman, & Kresheck, 1996).

Two sets of speech intelligibility sentences are presented in appendices A and B. The Beginner’s Intelligibility Test (BIT; Osberger, Robbins, Todd, & Riley, 1994) was developed for preschool and early elementary children with hearing loss. The 10 sentences in each of the four BIT lists are short, use basic vocabulary, and have simple syntax. During administration, clinicians say each BIT sentence while using small objects to act it out. Children watch the demonstration and then imitate each sentence. The second set, the Monsen-Indiana University sentences (M-IU sentences; Osberger, Maso & Sam, 1993), was developed for students who can read. These sentences are slightly longer than the BIT sentences and contain more consonant clusters. Each Monsen-IU sentence is presented on an index card as the clinician says it. The card is then turned over before the child is asked to say the sentence so that the influence of reading on speech can be minimized. Following the procedures used by the author in recent studies (Ertmer, 2008; in press), each BIT and Monsen-IU sentence is presented twice for listeners. Additional lists of sentences include the Central Institute for the Deaf (CID) Everyday Sentences, which was developed for older children, adolescents, and adults (see Alpiner & McCarthy, 2000) and the sentences developed by McGarr (1983).

Several guidelines are presented to clarify some of the scoring issues that clinicians are likely to encounter. First, only the words that the child actually says--as determined by the clinician-- are used to calculate percent-intelligible scores. For example, if the target sentence is The boy is walking to the table, but the child says “Boy table,” only two words (rather than seven) are used as the denominator to calculate the percentage of words identified by listeners. Then, if the listener recognizes only “boy,” the score would be 50% intelligible (i.e., one word recognized divided by two words produced). A percent-intelligible score is calculated for each listener by dividing the total number of words identified by the total number of words actually spoken by the child. When multiple judges are used, the child’s overall intelligibility score is the average of the judges’ scores. Children are credited with saying an identifiable word if the root of the word is understood by listeners. For example, a listener response of “swim” would be counted as correct even though the target word was swims. Finally, children are not penalized for incorrect morphology or syntax; if a child says “see” instead of the target saw, the former word would be accepted if identified by listeners. Additional guidelines can be found in Chin, Tsai, and Gao (2003) and Flipsen and Colvard (2006).

Both scaling and item identification procedures can be influenced by factors such as whether the listeners have familiarity with talkers who have hearing loss, the presence/absence of speechreading cues, knowledge of the context of the speech sample, and the number of times speech samples are presented. Listeners who have little or no exposure to the speech of deaf children are most often recruited for intelligibility assessments. Each sentence is typically presented twice in an auditory-only format via headphones as listeners write their responses. For more information on factors affecting scores and procedures for assessing intelligibility see Osberger (1992).

Overcoming Barriers to Direct Assessment of Speech Intelligibility

Several problems appear to have contributed to the limited use of intelligibility assessments previously noted by Monsen (1981) and contemporary school professionals. First, until recently, the clinician’s model had to be edited out of recordings and the stimulus items had to be ordered into play lists before presentation to listeners. Recruiting listeners to judge the samples has also been challenging; potential volunteers had to be found, appointments made, and a convenient location reserved. Taken together, these procedures were often impractical for clinicians—especially those with large caseloads. Fortunately, technology is now available to simplify the recording and presentation process, and a few simple strategies can make listener recruitment easier.

Digital Recording Technology

Hand-held digital recorder-players (Figure 1) have made collecting and presenting speech samples much more practical. Now, instead of the complicated and time-consuming procedures used to digitize analog recordings and make play lists using personal computers, digital recordings and play lists are made simultaneously. Speech samples are captured by simply pressing the “Record” button at the start of the child’s utterance and turning it off at the end. Each utterance is automatically saved as an individual digital file. In this way, the clinician’s models are not recorded and do not have to be edited out of the recording. It is recommended, however, that several practice sentences be reviewed to ensure that only the child’s voice is recorded and to determine how closely the digital recorder should be placed to the child.

Figure 1.

Sony hand-held digital recorder-player.

After recordings are made, numbered files are presented to listeners by simply pressing “Play” to hear the first sentence and pressing “Pause” between presentations to allow time for written responses. Clinicians press “Play” to repeat the same sentence, or “Next” to present the following sentence. Headphones should be used by all listeners so that background noise is minimized. Hand-held digital recorders are relatively inexpensive, easy to use, and can store large numbers of samples.

Recruiting Listener-judges

In addition to concerns about recording and presenting speech samples, interviews with school professionals revealed that listener recruitment is often problematic. Not only is it difficult to locate volunteers, but scheduling appointments during the workday can be particularly challenging. These problems are increased when multiple volunteers are needed for each speech sample. The following suggestions are offered to overcome these barriers. Readers are encouraged to evaluate the feasibility of these strategies in their work places, and if necessary, to develop alternative ideas for their particular situations.

Adult volunteers with normal hearing and little exposure to the speech of children with hearing loss are ideal for both scaling and item identification assessments. These individuals can be found in schools and in the broader community. At school, volunteers can be recruited during parent-teacher organization meetings, from volunteer lists compiled by school administrators and parent-teacher organizations at the beginning of each year, and among employees who do not routinely interact with children who have hearing loss. It might also be possible to recruit student teachers and university students who are observing classrooms in the school.

Adult listener-judges might also be found by contacting community volunteer bureaus and service organizations such as the Lions, Kiwanis International, or Rotary International clubs that have a mission to serve children with disabilities. Distributing a printed handout to potential volunteers will help them to understand the purpose of the assessment and what their participation would entail. It is important to stress that volunteers should have normal hearing and limited exposure to individuals with deafness. Persons who report hearing problems or are familiar with deaf talkers should not be included as listeners. Normal hearing should be verified through a hearing screening prior to presenting speech samples. This can be accomplished by conducting a pure-tone hearing screening at 25 dB HL for the frequencies 1000, 2,000, and 4,000 Hz bilaterally. Screenings are passed when listeners respond to two of three pure-tones presented at each frequency in each ear (ASHA, 1997).

Typically developing children who are at least 9 years-old might also be recruited as listener-judges. The main advantage of recruiting children as listener-judges is that they are readily available in schools, making recruitment and presentation sessions quite convenient. There are at least two reasons to expect that older children, adolescents, and teens can be reliable listener-judges. First, by 9 years of age, children are mature enough to understand rating scales that have unambiguous descriptors such as those mentioned above. Second, most have acquired the literacy skills needed to write-down the words contained in the relatively simple BIT, Monsen-IU, and CID Everyday sentences. Thus, it seems reasonable that--given grade-appropriate reading levels and a conscientious attitude-- children 9 years and older can be successful listener-judges. Caution must be taken, however, to ensure that they freely volunteer to participate so that motivation is high, that they maintain attention throughout the entire task, and that they know to report any problems in listening to the recordings.

Having students act as listener-judges is similar to the peer grading procedures accepted under the Family Educational Rights and Privacy Act (1974). Further, confidentiality can be maintained by using a code to identify each child (e.g., Hector LaCerte would be HELA) or by presenting sentence lists without identifying the talker. Clinicians who work in regular schools can recruit students within their buildings. Those who work in self-contained special education centers may need cooperation from a neighboring school to locate volunteers and conduct presentation sessions in that building. As with adult listeners, students who act as listener-judges must have hearing within normal limits and little or no exposure to the speech of children with hearing loss. An additional convenience for this population is that pure-tone screening results may be available from school records. Although studies are needed to compare the performance of adult and child listeners, recruiting older children appears to have high face validity and can make intelligibility assessments practical and time-efficient.

Presentations for Groups of Listeners

Valid estimates of children’s intelligibility are more likely to be obtained with two or three listeners, rather than a single listener. However, presenting samples individually can be very time-consuming. A multi-channel amplifier can streamline presentations by playing samples to more than one listener at the same time. Figure 2 shows listener-judges using an ART Headamp4 multichannel amplifier with four headphone ports and individual volume controls. This headphone amplifier (and similar models with individual volume controls) enables listeners to set a comfortable listening level as they listen to speech samples. Multichannel amplifiers are relatively inexpensive and can be purchased at many electronics stores. During presentation, the volume level on the hand-held digital recorder-player should be set at mid-level so that individual listeners can have access to a wider range of adjustments through the amplifier. It is also recommended that clinicians listen along with the listener-judges to be sure that all equipment is working properly.

Figure 2.

Listener-judges using a multi-channel headphone amplifier as they write the words they understand from an audio-recorded speech sample.

Summary

The widespread adoption of newborn hearing screening, advances in sensory aid technology, and the extensive availability of early intervention programs have increased expectations for intelligible speech in today’s children who have hearing loss. Direct assessment of connected speech intelligibility provides a way to monitor children’s progress toward the ultimate goal of speech training: readily intelligible connected speech. Although cumbersome in the past, this valuable clinical tool is now more practical than ever. Clinicians who apply it on a regular basis gain vital information for intervention planning and educational decision-making.

Appendix A

The Beginners Intelligibility Test (Osberger, Robbins, Todd, & Riley, 1994)

List 1 The baby falls. Mommy walks. The duck swims. The boy sits. Grandma sleeps. That is a little bed. The boy walked to the table. My car is blue. He is brushing his teeth. She is taking a bath.	List 2 Daddy runs. The baby cries. The dog eats. The girl drinks. The clown falls. That is a big bed. The boy walked to the chair. My van is green. They are playing the drums. She is talking on the phone.
List 3 Daddy walks. The bunny drinks. The dog sleeps. The girl jumps. Mommy reads. That is a brown chair. The boy is on the table. My airplane is big. He is tying his shoe. She is brushing her hair.	List 4 The bear sleeps. Mommy sits. The rabbit hops. The cowboy jumps. Grandma falls. That is a black hat. The boy is under the table. My airplane is small. He is painting the chair. She is cooking dinner.

Toys and small objects used to administer BIT sentences: People: Baby, Mommy, Boy, Grandma, Daddy, Girl, Clown, and Cowboy Objects: Duck, Bed, Table, Blue Car, Toothbrush, Bathtub, Dog, Drink, Green Van, Drum, Telephone, Bear, Rabbit, Black Hat, Airplane, Paint, Pot/Pan, Book, Brown Chair, and Hairbrush

Appendix B

Monsen – Indiana University Sentences (Osberger, Robbins, Todd, & Riley, 1993)

List 1	List 2	List 3
This house is white. My dog is mean. Can he make any? Did you find some? You got a nice haircut. We made a nice birdhouse. Can he stop them? Did she bring it? My grandmother is beautiful. That elephant was dangerous.	Our car is safe. That lake is deep. Can you tell us? Do you want any? They saw a long sunset. She saw the poor cowboy. Can you start it? Have they reached it? My television is broken. That newspaper was interesting.	His boat is white. My bike is new. Did we call them? Did you buy it? She ate a good hot dog. We bought a new baseball. Did you steal it? Did you try it? Her sweater is purple. The butterfly is sleeping.