Teaching Children with Hearing Loss to Recognize Speech: Gains Made with Computer-Based Auditory and/or Speechreading Training

Abstract

Objective:

Transfer appropriate processing (TAP) refers to a general finding that training gains are maximized when training and testing are conducted under the same conditions. The current study tested the extent to which TAP applies to speech perception training in children with hearing loss. Specifically, we assessed the benefits of computer-based speech perception training games for enhancing children’s speech recognition by comparing three training groups: auditory training (AT), audiovisual training (AVT), and a combination of these two (AT/AVT). We also determined whether talker-specific training, as might occur when children train with the speech of a next year’s classroom teacher, leads to better recognition of that talker’s speech and if so, the extent to which training benefits generalize to untrained talkers. Consistent with TAP theory, we predicted that children would improve their ability to recognize the speech of the trained talker more than that of three untrained talkers and, depending upon their training group, would improve more on an auditory-only (listening) or audiovisual (speechreading) speech perception assessment, that matched the type of training they received. We also hypothesized that benefit would generalize to untrained talkers and to test modalities in which they did not train, albeit to a lesser extent.

Design:

Ninety-nine elementary-school aged children with hearing loss were enrolled into a randomized control trial with a repeated measures A-A-B experimental mixed design in which children served as their own control for the assessment of overall benefit of a particular training type and three different groups of children yielded data for comparing the three types of training. We also assessed talker-specific learning and transfer of learning by including speech perception tests with stimuli spoken by the talker with whom a child trained and stimuli spoken by three talkers with whom the child did not train and by including speech perception tests that presented both auditory (listening) and audiovisual (speechreading) stimuli. Children received 16 hours of gamified training. The games provided word identification and connected-speech comprehension training activities

Results:

Overall, children showed significant improvement in both their listening and speechreading performance. Consistent with TAP theory, children improved more on their trained talker than on the untrained talkers. Also consistent with TAP theory, the children who received AT improved more on the listening than the speechreading. However, children who received AVT improved on both types of assessment equally, which is not consistent with our predictions derived from a TAP perspective. Age, language level, and phonological awareness were either not predictive of training benefits or only negligibly so.

Conclusions:

The findings provide support for the practice of providing children who have hearing loss with structured speech perception training and suggest that future aural rehabilitation programs might include teacher-specific speech perception training to prepare children for an upcoming school year, especially since training will generalize to other talkers. The results also suggest that benefits of speech perception training, were not significantly related to age, language level, or degree of phonological awareness. The findings are largely consistent with TAP theory, suggesting that the more aligned a training task is with the desired outcome, the more likely benefit will accrue.

INTRODUCTION

Formal speech perception training, especially auditory training but sometimes speechreading training, is considered a key component of many aural rehabilitation plans designed for children who have hearing loss (e.g., Tye-Murray, 2020; Schow & Nerbonne, 2017; Hull, 2021). The goal of auditory training is to help children maximally use their residual hearing to recognize speech and, in some instantiations, use their hearing to recognize environmental sounds and appreciate music. Auditory training is assumed to help the brain distinguish between sounds, recognize words, tune out background noise, and comprehend meaning. Ultimately, children who receive auditory training may be able to learn new information under more natural listening conditions, as happens during face-to-face conversations, when they are in a classroom looking down, listening, and taking notes or when they overhear conversation and experience incidental learning.

One key goal of speechreading training is to focus children’s attention on the visual speech signal to supplement the degraded auditory signal they receive as a consequence of their hearing loss. The visual signal can greatly enhance the ability of people with hearing loss to recognize speech (e.g., Tye-Murray, Sommers, & Spehar, 2007) and comprehend discourse (Tye-Murray, Sommers, Spehar, Myerson, Hale, & Rose, 2008). For example, in a study on older adults with hearing loss, Tye-Murray et al. (2007) found that the addition of the visual speech signal to the auditory signal resulted in 35% better recognition of words presented in carrier phrase context. Ultimately, children who receive speechreading training may be able to engage more easily and successfully in face-to-face conversations and be less hindered by the presence of background noise.

Whereas the benefits of speech perception training for children have not been well established to date, one theoretical framework that has drawn increasing interest regarding its applicability to different aspects of auditory training (Barcroft, Sommers, & Tye-Murray, 2007) is transfer appropriate processing (TAP) (Morris, Bransford & Franks, 1977). According to a TAP framework, the relative effectiveness of different types of tasks on human memory and learning depends upon the extent to which the types of cognitive processes engaged during learning match (or do not match) the processes engaged during testing. For example, completing a structurally oriented rhyming task at study improves performance more on a rhyming task at test; conversely, completing a semantically oriented pleasantness rating task at study improves performance more on a pleasantness ratings task at test. Numerous demonstrations of TAP-consistent effects across a wide array of different areas of memory and learning exist, including for auditory training (e.g., Barcroft et al. 2011), drawing into question how the TAP perspective may be useful in the design of different types of tasks for different types of learning goals in the area of speech perception training in general.

With TAP as a theoretical framework, the present investigation was designed to assess the benefits of computer-based speech perception training games for enhancing children’s speech recognition and to compare three groups for the effectiveness of different types of training. We implemented a well-populated, randomized control trial with a repeated measures A-A-B experimental mixed design, where children served as their own control for the comparison of overall benefit between training groups. The three training types were auditory training (AT), audiovisual training (AVT; i.e., speechreading training), and a combination of auditory and audiovisual training (AT/AVT), where AT and AVT were interwoven. We also assessed talker-specific learning and transfer of learning by including speech perception tests with stimuli spoken by the talker with whom a child trained and with talkers with whom the child did not train and by including speech perception tests that presented both auditory and audiovisual stimuli, regardless of whether a child received AT, AVT, or AT/AVT.

The rest of this introduction is divided into six sections. The first two sections review research about the effectiveness of auditory training and speechreading training. The third describes the possibility of implementing a hybrid approach, one that interweaves auditory and speechreading training. The fourth section focuses on the applicability of TAP theory, with a particular focus on comparing the three approaches to training of interest (AT, AVT, AT/AVT). The fifth discusses how principles related to TAP theory were pivotal in the development of the study’s speech perception training games. Finally, the sixth section outlines the specific research goals of the study reported.

Auditory Training (AT)

The research about the benefits of AT for children is sparse and often includes very small sample sizes. In addition, instruction often trains a variety of auditory skills in addition to speech perception, such as environmental sound recognition or music appreciation, so it is hard to extrapolate conclusions from these studies to instances where only speech-based AT may be provided. Even so, the results from published research are for the most part promising. For example Rochette and Bigand (2009) conducted a study with six participants and used voices, music, environmental sounds, and abstract sounds as training stimuli. The children received a total of ten hours of AT spread over 20 weeks. Results indicated that children improved in both their sound discrimination and identification. Roman, Rochette, Triglia, Schon, & Bigand (2016) conducted a study with children who were newly implanted with cochlear implants. Ten children received 10 hours of AT spread over 20 weeks that included environmental sounds, speech, and musical sounds. Nine children served as controls and received no intervention. The children who received the AT improved on discrimination, identification, and auditory memory tasks but not on an auditory scene analysis task. The children in the control group showed no improvement on any of the four types of task.

Two systematic reviews of the existing literature speak to the paucity of research, and reveal the need for further investigation about the benefits of AT for children. In a review of AT for children who received cochlear implants, Rayes, Al-Malky, & Vickers (2019) identified 96 published articles, of which only 9 met their inclusion criteria, and some of these articles had as few as nine participants (Kronenberger, Pisoni, Henning, Colson, & Hazzard, 2011) with 19.7 participants on average (SD=7.0). Only two implemented a randomized control trial experimental design (Ingvalson, Young, & Wong, 2014; Roman, Rochette, Triglia, Schon, & Bigand, 2016) and only three implemented a repeated measures design (Kronenberger et al., 2011, Wu et al., 2007; Welch et al., 2015). In the second review (Nanjundaswamy, Prabhu, Rajanna, Ningegowda, & Sharma, 2018), which concerned computer-based AT programs for children, only one of a possible 120 articles met the inclusion criteria (Glyde, Cameron, Dillon, & Hickson, 2014). Both reviews concluded that AT is beneficial and Rayes et al. (2019) also suggested that improvements can transfer to untrained tasks (found to be true in six of the studies). Both reviews ended with a call for more rigorous research. Rayes et al. (2019) wrote, “…a higher quality of evidence for examining outcomes of AT in pediatric CI [cochlear implant] recipients is still required” (pg. 1589) and Nanjundaswamy et al. (2018) wrote, “…there is a dire need for more research to establish efficacy measures for these [computer-based] programs” (pg. 92).

In addition to the paucity of published research, confounding any attempt to assess AT benefits is the heterogeneity of AT methodologies. Many different types of tasks fall under the rubric of auditory training and yet, in considering its value, the mistake of “lumping apples with oranges” is sometimes made. For example, in the review by Rayes et al. (2019), the researchers in the selected articles implemented training activities that required children to discriminate between nonsense syllables (Wu, Yang, Lin, & Fu, 2007), to receive singing and vocal exploration training (Welch et al., 2015), to take piano lessons (Good, Gordon, Papsin, Nespoli, Hopyan, Peretz, & Russo, 2017), to listen to pairs of notes on a keyboard (Yucel, Sennaroglu, & Belgin, 2009), to recognize environmental sounds (Roman et al., 2016), or to recall drumbeat sequences, distinguish between phonemes and syllables, and to recognize speech in noise (Ingvalson & Wong, 2013). Speech language pathologists who look to the literature for evidence-based guidance can be very challenged to make sense of the wide range of training approaches.

Speechreading or Audiovisual Training (AVT)

Most research about the effectiveness of training persons with hearing loss to utilize the visual speech signal has been performed with adults, often adults who have typical hearing, and has focused primarily on the benefits of providing vision-only training (i.e., the corresponding auditory speech signal is muted during training). Some studies suggest modest benefit of vision-only training, perhaps enhancing participants’ ability to recognize speech stimuli by about 10-15% (e.g., Sims, Dorn, Clark, Bryant, & Mumford, 2002; Walden, Prosek, Montgomery, Scherr, & Jones, 1977) whereas other studies have shown little if any measurable benefit (e.g., Lesner, Sandridge, & Kricos, 1987; Rishiq, Rao, Koerner, & Abrams, 2016).

Few studies have focused on children, but the findings are somewhat promising, the qualifier somewhat being used here because of the small sample sizes and lack of repeated measures or a control group. Massaro and Light (2004) administered vision-only training using a computer-based avatar to a group of seven children with hearing loss and found they improved their recognition of phonemes after 6 hours of training spread over 21 weeks. Kishon-Rabin, Haras, & Bergman (1997) found that AVT coupled with the use of a tactile aid improved closed-set word recognition of four children with profound hearing loss. Less promising is that some researchers suggest that the ability to utilize visual speech information is an innate ability that cannot be improved through training (e.g., Summerfield, 1987), which might rule out any possible benefit of AVT. Countering this suggestion are findings that children improve their ability to utilize the visual speech signal as they age (Tye-Murray, Hale, Spehar, Myerson, & Sommers, 2014) and that adults with prelingual hearing loss can read lips better than adults with typical hearing (Auer & Bernstein, 2007), suggesting that experience can modulate lipreading performance. These two sets of findings suggest that one’s ability to utilize visual speech information is malleable with experience, and hence, susceptible to improvement through targeted training.

Another hint that children may benefit from speechreading training come from studies that show that experience may affect speechreading and lipreading (vision-only) performance. For example, Lalonde and McCreery (2020) found that children with hearing loss and adults with typical hearing could better extract phonetic information from the visual speech signal than children with typical hearing and they received greater visual enhancement (i.e., the benefit when both seeing and hearing the talker as compared to just hearing the talker), suggesting that experience can affect performance, whether the experience comes from having to rely heavily on the visual speech signal for everyday communication or from maturation. Tye-Murray, Hale, Spehar, Myerson, and Sommers (2014) showed that lipreading performance improves in children ranging in age from 7 to 14 years of age and that children with hearing loss are better lipreaders on average than children with typical hearing. Again, these findings suggest that experience can affect how well children utilize visual speech information. One implication of this research is that children who experience speechreading training may have the potential to improve their speechreading performance.

To our knowledge, no group of researchers has assessed the benefits of AVT with a large group of children who have hearing loss, and no group has compared the benefits of AT with those of AVT. It could be that AT leads to improved recognition of speech that is presented in an auditory-only condition whereas AVT leads to improved recognition of speech presented in an audiovisual condition. It may also be that AT will enhance children’s speechreading performance and AVT will enhance children’s listening performance because children have learned to make the most of their residual hearing, whether this learning has occurred as a result of AT or AVT.

A Hybrid Training Approach (AT/AVT)

A relatively novel approach to provide speech perception training to children who have hearing loss is to interweave AT and AVT. This approach is inspired by practices found in second language learning. Second language learners acquire vocabulary more efficiently if instruction includes different types of acoustic variability, such as when multiple talkers instead of single talkers speak word tokens or when the speaking rate in which words are spoken is varied (e.g., Sommers & Barcroft, 2013). One potential implication of such findings is that children with hearing loss might benefit from stimulus variability in the form of presentation modality wherein training material varies between being presented in an auditory versus audiovisual modality.

Transfer-Appropriate Processing Theory and Transfer of Learning

One of the exciting new directions in AT is the application of principles derived from cognitive psychology and second-language learning to inform development of AT programs (Barcroft, Sommers, & Tye-Murray, 2007). For example, research in both learning and memory and second-language learning has demonstrated that learning is typically transfer appropriate -- the more similar the training is to the desired outcome, the greater will be the measurable training benefits. Studies with adults have consistently shown that AT benefits are tied to what is transfer-appropriate--greatest benefit is shown if the same talker and similar tasks are used in both training and assessing outcome (see Bronus & Pryce, 2011; Sweetow & Palmer, 2005 for systematic reviews). In previous research, we demonstrated that AT is most effective when it is targeted on specific outcomes (Barcroft, Sommmers, Tye-Murray, Mauze, Schroy, & Spehar, 2011; Barcroft, Spehar, Tye-Murray, & Sommers, 2016; Tye-Murray, Spehar, Sommers, & Barcroft, 2016). For example, Barcroft et al. (2011) showed that adults who have hearing loss and who received auditory training with a single talker improved on a test that presented stimuli spoken by that talker more than they improved on a test that presented stimuli by that talker plus an additional five talkers. Conversely, adults who received auditory training with all six talkers improved more on the test with the multiple talkers than on the test with a single talker. These findings are consonant with transfer-appropriate processing theory (e.g., Barcroft et al., 2007; Barcroft et al., 2016).

With respect to children, an example of transfer-appropriate learning might be to teach a child to recognize the speech of next year’s classroom teacher. As such, the child would receive speech perception training that utilizes that teacher’s recorded speech samples over the preceding summer. On the first day of class, the child would better recognize the teacher’s utterances than if the child had trained with utterances spoken by a “generic” actor.

In an ideal world, in addition to learning to recognize a particular talker’s speech, the benefits of speech perception training should extend to novel talkers and to novel situations, even if not to the same extent as in transfer-appropriate learning. In a review article that considered the benefits of auditory training for older adults who use hearing aids, Dubno (2013) noted, “Given that some [auditory training] tasks and stimuli may not be representative of real-world communication, it is important to assess the extent to which training with one set of tasks or stimuli results in improvements for novel (untrained) speech stimuli, novel talkers, and novel competing noises or messages” (pg. 337). In the hypothetical example of a child preparing for the new school year, although the child may not recognize the utterances of other unfamiliar adults (e.g., the gym teacher) as well as those of the trained-on teacher, the child might still be more successful because of the summer training. Work reported by Houston and Juscyk (2000) is consistent with this possibility. They found that infants who learned new words spoken by a male talker generalized to words spoken by a different male, and likewise for female talkers. Alternatively, one finding that is inconsistent with the TAP framework in the context of auditory training with children who have hearing loss are results from Sullivan et al. (2013). Specifically, Sullivan et al. reported that auditory training conducted in the presence of interrupted noise, produced equivalent gains for assessments conducted in both interrupted and continuous noise.

Incorporation of TAP theory into the Training Program

In this investigation, we evaluated both transfer-appropriate learning and transfer of learning to novel speech tasks. First, we trained children with samples produced by one talker and compared gains accrued with this talker to those accrued with unfamiliar talkers. If TAP theory applies, training and testing with the same talker should lead to greater improvement than training and testing with different talkers. Second, we also assessed the applicability of TAP by assessing the extent to which training in a given modality (AT, AVT, or AT/AVT) would lead to better performance in the corresponding modality after training. If TAP theory was applicable, then a given modality such as A only, should lead to better performance when the same modality was used at posttest, such as A only (listening). We also might expect that the combined AT/AVT training would yield comparable benefits in the listening and speechreading assessments, as the combined training incorporates conditions both with and without visual speech information.

Goals of the Present Study

In conjunction with assessing TAP theory, the goals of this investigation were as follows: 1) to assess the overall benefits to three groups who receive different types of speech perception training (i.e., AT, AVT, AT/AVT), 2) to demonstrate the benefit of talker-specific speech perception training, and 3) to determine if training benefits transfer to novel talkers and to conditions not trained (i.e., either a listening or a speechreading condition). We also assessed whether a child’s age, language level, or phonological awareness predicted their likelihood of benefiting from speech perception training, as findings might indicate which school chidren are most likely to benefit from this kind of intervention.

METHODS AND MATERIALS

All children completed two baseline sessions, 16 speech perception training sessions, and a final assessment after training was complete. Participants were quasi-randomly assigned to one of three training conditions (AT, AT/AVT, or AVT), and one of four talkers. To create similar group sizes, participants were assigned to group and talker based on the order they were recruited.

Participants

Ninety-nine children with sensorineural hearing loss were consented and enrolled in the “Enhancing Children’s Everyday Communication: Talker-Specific Speech Recognition Training” program. Of the 99 participants, three withdrew before the training was complete and therefore were not included in the analyses. Table 1 presents demographic and ethnic distribution information for the children along with the information split by group assignment. Child participants included 47 female children and 49 male children ages 5.8 to 12.0 years. (M=8.4 years, SD=1.9 years), an age range typical of children enrolled in USA elementary public schools. The racial distribution of the participants was generally representative of the St. Louis school district and included 50% White, 35% Black, 10% more than one race and 4% Asian. Eighty-five percent of the children were Not Hispanic while 9% were Hispanic or Latino and 5% were unknown or not reported. Socioeconomic status (SES) was obtained by using an 8 point educational index and an 8 point occupational index. The occupation of the primary income earner and their educational level was used to obtain the score. The two index scores were multiplied to obtain a score of 1-64. The average SES was 26.5 (SD=14.6).

Table 1.

Demographics by training group.

	Total	AT	AT/AVT	AVT
Female	47	12	18	17
Male	49	21	14	14
White	48	18	16	14
Black	34	11	13	10
Asian	4	2	0	2
More than one race	10	2	3	5
Not Hispanic	82	26	29	27
Hispanic or Latino	9	4	2	3
Ethnicity unknown or not reported	5	3	1	1
Avg Age (years)	8.4(1.9)	8.7(2.0)	8.4(1.9)	8.4(1.9)
Avg Age at Identification (months)	14.9(19.4)	12.3(14.9)	13.7(15.9)	14.6(25.9)
Avg age at Amplification (months)	24.4(21.9)	23.0(17.7)	21.3(17.0)	23.6(29.1)
Socioeconomic Status	26.5(14.6)	28.1(14.7)	23.4(15.6)	28(13.4)
Two Hearing Aids	46	18	14	14
Two Cochlear Implants	28	7	12	9
Bimodal	20	6	6	8
One Hearing Aid	2	2	0	0

^*(Standard Deviation)

The participants’ average age of hearing loss at identification was 14.9 months (SD=19.4 months). Ninety participants were confirmed as having been diagnosed as having hearing loss before the age of 36 months and five were confirmed by the age of 48 months. The average age at amplification was 24.4 months (SD=21.9 months). Forty-six participants were amplified with bilateral hearing aids, 28 were fitted with bilateral cochlear implants, 20 were bimodal, and two were amplified with one hearing aid.

All participants had bilateral hearing loss. Table 2 includes average aided threshold information for all the children, along with information split by group assignment. Seventy-nine participants had aided pure tone sound field testing. The average aided pure tone average (PTA) (500Hz, 1000Hz, 2000Hz) was 26 dB HL (SD=8.1). Sixteen participants were unavailable for aided soundfield testing but aided Ling Sounds Test results were accessed through their school records (see Glista, Scollie, Moodie, & Easwar, 2014). Sound awareness of the six Ling sounds (/m/, /u/, /a/, /i/, /sh/ and /s/) was averaged to obtain an aided Ling Sounds Test. The average aided Ling test was 19 dB HL (SD=11.4). Aided thresholds were not obtained or able to be accessed through school records on one child who wears bilateral hearing aids. Prior to participation, their listening devices were checked to ensure proper functioning.

Table 2.

Aided thresholds by training group.

Group	PTA (Avg of .5, 1, 2 Khz)	Ling Sounds Test (/m/, /u/, /a/, /i/, /sh/, /s/)
Total	26dB (8.1), N = 79	19.48 (11.4), N = 16
AT	24dB (7.7), N = 25	18dB (6.6), N = 7
AT/AVT	25dB (7.5), N = 26	19dB (8.4), N = 6
AVT	26dB (8.7), N = 28	26dB (22.8), N = 3

^*(Standard Deviation)

Each child’s current educational setting and communication mode was obtained using a questionnaire completed by a parent. Table 3 provides descriptive details related to educational settings and teaching approach as well as training groups. Sixty-five participants attended public schools, 28 attended private schools for children with hearing loss, two attended private schools, and one was home-schooled.

Table 3.

Educational setting by training group.

	Total	AT	AT/AVT	AVT
Public School	65	24	21	20
Private School for the Deaf	28	9	10	9
Private School	2	0	0	2
Home-Schooled	1	0	1	0
Fully Mainstreamed	24	9	6	9
Partially Mainstreamed	42	14	15	13
Special Classroom	29	10	10	9
Auditory Visual Emphasis	88	30	30	28
Auditory Verbal Emphasis	4	2	1	1
Total Communication (Equal Emphasis)	3	0	1	2
Total Communication (Speech Emphasis)	1	1	0	0

^*The homeschooled child was not mainstreamed or part of a special classroom

Participants were recruited through our laboratory database, Central Institute for the Deaf, Special School District of St. Louis, MO, and colleagues. We also posted flyers in audiology clinics throughout the St. Louis region. Each child received a hand-held tablet and audio speaker for completing the study as well as $10 an hour for their participation. The experimental protocol received IRB approval through the Washington University Human Research Protection Office.

Design

The design for the study included assignment to one of three training groups (AT, AVT, AT/AVT). Assessments were conducted using two types of talkers (trained versus untrained), and two modalities (listening versus speechreading). The dependent measure used to assess the effects of training was computed as the difference between baseline and post-training and was conducted immediately following completion of the training sessions.

Talkers:

The assessment stimuli recordings were spoken by four professional actors (two males and two females), who also spoke the training items. Recordings were created for each talker in a sound treated booth using a teleprompter. For the most natural recordings, talkers were asked to read the teleprompter and, once the words on the screen disappeared, speak the material directly into the camera. Stimuli were then edited and adjusted to the same long-term RMS across all four talkers. Each participant was assigned randomly to hear training material spoken by one of these four talkers. Thus, we were able to compare the gains on the speech recognition tests that participants made on their assigned training talker pre- to post-training with the magnitude of those gains made on talkers for whom they did not train. All talkers were white Midwest-English speaking professional actors in their mid-twenties. Acoustic analyses using Praat software determined the fundamental frequency for the talkers. Twelve words from each talker were analysed to calculate the fundamental frequency during vowel production of each talker’s voice. The words eat, bee, see, octopus, ostrich, dog, zoo, boot, shoe, bat, hat, map were used. The average fundamentals for the two male and two female talkers were 83.6 (min=76.8, max=85.9), 98.4 (min=95.0, max=119.9), 211.9 (min=181.0, max=233.0), and 212.8 (min=191.1, max=244.4) Hz respectively.

Predictive Test Measures

Prior to the first training session, the Peabody Picture Vocabulary Test (PPVT-4) (Dunn & Dunn, 2007) and the Expressive Vocabulary Test (EVT-2) (Williams, 1997) were either administered or obtained from the child’s parent or by release of information from the child’s audiology clinic if the results had been obtained within the previous year. Participants completed two subtests (Ellison, Blending Words) of the Comprehensive Test of Phonological Processing (CTOPP) (Wagner, Torgesen, Rashotte & Pearson, 1999) and a vision screening test.

Speech Perception Assessment Procedures

The speech perception assessments were administered over three test intervals in an A1-A2-B sequence, where A1 and A2 were baseline (pre-training) assessments (baseline one, baseline two) and B was the post-training assessment (post-training). By including two baseline assessments and comparing performance between them, we could determine whether any gains in performance seen post-training might have been due to the repeated exposure to the talkers and/or testing procedures. Participants completed the baseline one assessment at least one week before the baseline two assessment (M=12.1 days; SD=19.0). They completed the baseline two assessment shortly before beginning their training regime (M=3.5 days; SD=6.0).

At all three assessment intervals participants were tested using word-, sentence-, and phrase-level test lists. For each participant, scores from these test lists were combined to create a composite score. A composite score was used because conversation is a construct, one comprised of words, phrases and sentences that might occur when people engage in conversation. To explore individual differences at a later time, we avoided test order as a potential source of error variance and so the test order was the same for all participants.

Each test list included 144 presentations preceded by 12 practice items. Different test items were presented at each assessment interval. Items were taken randomly from the training material. No word, phrase, or sentence was repeated across the three assessments. Items were distributed equally among the four talkers and two modalities (A and AV). The phrases assessment was scored by a single target word correct and the sentences were scored by key words correct. Only exact matches to the target words were counted as correct. The final listening and speechreading composite scores were obtained by averaging performance across the three tests separately for each of the two modalities. The three different types of speech stimuli tests were not compared to each other because they were not counterbalanced to control for potential order and/or fatigue effects.

In the speechreading condition, the talkers’ head, neck, and upper shoulders appeared in the video. Participants were instructed to repeat the stimuli aloud. Tests were administered in the presence of 4-talker babble in order to prevent ceiling performance. The babble consisted of two female and two male talkers. The babble track was taken from the Iowa Phoneme and Sentence Tests (Tyler, Preece & Tye-Murray, 1986) and consisted of combined recordings of the four talkers reading from a newspaper. To avoid ceiling performance in the speechreading assessment condition, the SNR for each stimulus type (words, phrases, sentences) was set individually to obtain approximately 30% in an auditory-only condition. To determine the level of babble for each participant, Lists 13a and 13b of the Bamford-Kowal-Bench Speech-In-Noise test (BKB-SIN) (Bench, Kowal, & Bamford, 1979) were administered. For testing, we used the SNR50 from the BKB-SIN as a starting point to estimate a 30%-words-correct performance level in the listening condition of the words in isolation, phrases, and sentences. During the initial assessment, occasionally, if the tester felt that the child’s performance level was not close enough to the targeted 30% in the listening condition a small adjustment (± 2 dB steps in noise level) was made during the ten practice items to ensure a better estimate of the SNR needed. After that, the same SNRs, set separately for the different contexts, were used at all three assessment sessions and the same SNR was used in both the listening and speechreading conditions. Within a test, stimuli were administered in an auditory-only with babble condition (listening) and an audiovisual with babble condition (speechreading). The word, sentence and phrase tests were blocked; however, the order of conditions and talkers within each test was random.

All assessments were completed in a sound-treated booth in the Washington University School of Medicine Audiovisual Speech Perception Laboratory or in front of a sound-treated partition in a quiet room designated for the experiment within the child’s elementary school.

Training Program

The participants were randomly assigned to one of three training groups. The groups were AT (n=33), AVT (n=31), and AT/AVT (n=32). Participants within each training group were also randomly assigned to one of four talkers who spoke the training items that were presented in their speech perception training games. A child’s assigned talker is referred to as the child’s Trained Talker and the remaining three talkers are referred to as the child’s Untrained Talker.

Participants completed 16 training sessions that each lasted between 45 and 60 minutes, depending upon how long it took them to play the set of training games. Depending upon a participant’s school/summer schedule, they trained between one and four days a week over the course of four weeks. In each session, participants played a prescribed number of rounds of seven computerized training games while seated either before a 17-inch touch screen monitor in a sound-treated booth in the laboratory or between a 10.8 inch Surface Pro tablet and a sound treated partition in the participant’s elementary school (the same room where the participant’s assessment occurred). Audio was presented via two loudspeakers positioned ± 45 degrees at approximately 62 dB SPL. Noise levels and SNRs were adjusted programmatically using a PC and LabView control of Tucker Davis Technologies RP2.1(lab-based) RM1 (school-based) processors and an amplifier to drive the speakers. The training program adjusted the SNR programmatically during training. Computer-based speech perception training was performed in the form of 7 games. We opted for computer-based training as opposed to “live” training so we could control the quantity and quality of training and so we could ensure that participants in the three groups trained with the same activities and stimuli and for approximately the same amount of time. In a typical game, children either heard or speechread an assigned talker speak a stimulus and then responded by selecting a picture from a set of competitors. Positive reinforcement that was integrated into each game occurred following a correct response.

Rationale for the Training Games

The development of the current training games was based on the framework developed in several commercially available curricula. Many formal speech-based auditory training programs such as the SPICE-- --Speech Perception Instructional Curriculum and Evaluation (Manley, 2017), the CHATS- --The Miami Cochlear Implant Auditory and Tactile Skills Curriculum (Vergara & Miskiel, 1994), and the Developmental Approach to Successful Listening II (DASL II, Stout & Windle, 1992) follow a three-or-four level hierarchical progression in developing listening skills, in accordance with the developmental hierarchy of listening skills proposed by Erber (1982). In the first level, which primarily pertains only to babies and very young children, training activities require students to detect sound and be aware of its presence or absence. Once that skill is mastered, the next level of training requires children to discriminate sounds and words and decide whether two stimuli are the same or different, even if students cannot understand the words or attach meaning to the stimuli. In the final level (which in some curricula are two distinct levels), training activities require children to identify the meaning of words and to comprehend connected speech.

The participants in our study had prelingual hearing loss, wore appropriately fitted listening devices, and were of elementary school age so we bypassed the traditional first and second levels and developed auditory training games that were appropriate for the final level(s). Training games focused participants’ attention on identifying everyday words and comprehending connected speech in the form of phrases and simple sentences. In addition, some games were designed to tap into auditory processing speed (the speed with which a child could recognize a word), auditory memory (the ability of a child to hold spoken words in memory), and auditory attention (the ability to attend to speech in the presence of background babble). Increasingly, contemporary speech perception training programs include training for those cognitive skills that are necessary to comprehend speech, no matter what is being said (e.g., Mishra, Boddupally, & Rayapati, 2015; Kronenberger, Pisoni, Henning, Colson & Hazzard, 2011).

The training games each presented either word, phrase, or sentence stimuli. No training item was presented more than four times during the 16 hours of training, and typically occurred only once. As part of game development, we selected for inclusion training words from lists of familiar children’s words and children’s vocabulary tests and asked two teachers of children with hearing loss to review the selections. We then selected illustrations from the Internet that corresponded with the words. To ensure that illustrations unambiguously depicted the words, we asked two 7-year-old children with typical hearing to name the selected illustrations. Any misnamed illustration was replaced with an alternative.

The seven training games were identified arbitrarily by number and are described in Figure 1.

Figure 1.

Instructional design for each of the seven games and gives examples of what a child might experience if he or she was assigned to the AVT AT/AVT, or AVT groups. The top row indicates the type of training a game provided

Each game required the child to perform either an identification or comprehension task. Also, whenever possible, cognitive processes were used as a foundation for the auditory training activity in the games. For example, Game 1 was designed to utilize auditory processing speed, Games 3 and 5 used auditory memory, and Games 1, 2, and 3 were directed at auditory attention activities. Games 1, 2, 3, 4 and 6 were played in four-talker babble. For Games 1, 2, 4, and 6 the level of the babble was adaptively varied in order to maintain a correct-response-rate of approximately 70%. The audio for games 5 and 7 were presented in quiet. The games allowed participants to accumulate points for every correct response and some games provided visual prizes at the end of a round.

RESULTS AND ANALYSES

Baseline Assessments

The three assessment intervals included two baseline assessments along with a post-training assessment. Figure 2 shows the average scores at the three different assessment times split by the modality of the assessment. Comparing the two baseline assessments determined if potential gains were a result of repeated exposure to the talkers at the time of the assessments or other non-training related improvements (e.g., participants becoming increasingly comfortable with the testing environment and procedures). The two sets of baseline scores were compared using paired-samples t-tests for each type of assessment condition separately (listening or speechreading). Baseline listening and speechreading scores changed by an average of 1.2 and 0.4 percentage points, respectively. There was no change from baseline one to baseline two for either assessment modality (95% CI of the Mean Difference; listening = −2.6 – 0.2; speechreading −0.97 – 1.7). The most recent baseline score was taken as the pre-training performance level, therefore all subsequent analyses of training benefit were conducted using the baseline two scores for the pre-training assessment and are referred to as baseline scores.

Figure 2.

Average percent word correct scores listening and speechreading at each assessment interval. Error bars indicate standard error.

Transfer Appropriate Processing (TAP) Analyses

To address the general issue of TAP, planned analyses focused on two specific comparisons, the modality of the training and the talker used during training. First, to determine if there was a difference between listening and speechreading scores depending on which training group the child was assigned, we looked at whether there was an effect of group assignment (AT, AT/AVT, AVT) on the relative amount of improvement in the test modality (listening versus speechreading). TAP processing would be evident if participants trained in the AT condition exhibited greater improvement on the listening than on the speechreading assessment and, those who trained in the AVT condition exhibited greater gains on the speechreading assessment. Those trained in AT/AVT were expected to show similar amount of gain on both types of assessment.

Figure 3 presents the scores at the pre- and post-training assessments for each group separately. Scores were analyzed using a mixed design three-way ANOVA with assessment modality (listening versus speechreading) and training benefit (pre- versus post-training) scores entered as repeated measures variables and group assignment as a between-subjects variable. Results indicated that performance was significantly better following training compared with pre-training scores (F(1,93) 207.1, p<.001, pt eta sq = .690) and that performance was higher in the speechreading compared with the listening assessment modality (F(1,93) 222.5, p<.001, pt eta sq = .705).

Figure 3.

Average percent word correct scores for listening and speechreading) for each of the three training groups (i.e., AT, AVT, AT/AVT) at baseline and post-training. Error bars indicate standard error.

Consistent with a TAP perspective, training-related gains for those trained in the A condition were significantly higher for listening assessment than for the speechreading assessment (10.6 versus 6.5 percentage points; see Table 4 for confidence intervals and results of statistical analyses). Furthermore, training gains for those trained in the AVT condition were higher for the speechreading assessment than for the listening assessment (9.3 versus 8.6 respectively). Interestingly, training gains on the mixed modality (AT/AVT) training were higher than either of the pure modalities for both assessment types (12.9 and 9.4 for listening and speechreading respectively). Post-hoc analyses with a Bonferroni correction for multiple comparisons indicated that the children who trained in the AT group had greater gains when tested in the listening than in the speechreading assessments (p<.01). Similarly, the children trained in the combined group (AT/AVT) had greater gains in the listening than in the speechreading assessment (p<.05). Finally, the gains seen by individuals trained in the AVT group were similar for both types of assessment.

Table 4.

Benefit by training group.

Group	Mean Listening Benefit (SD)	Mean Speechreading Benefit (SD)	Mean Difference in Percentage Points
A	10.6 (7.7)	6.5 (7.0)	4.1 (SD=7.9) CI of the Mean Diff = 1.3 – 6.8 F(32)=8.9, p<.01, pt eta sq=.217
A/AV	12.9 (7.6)	9.4 (7.2)	3.5 (SD=7.8) CI of the Mean Diff = 0.7 – 6.3 F(31)= 6.4, p=.02, pt eta sq=.170
AV	8.9 (8.4)	9.3 (8.0)	−0.4 (SD=7.9) CI of the Mean Diff =−3.3 – 2.5 F(30)<1.0, p=.797, pt eta sq=.002

The second TAP-centered analysis focused on talker familiarity. The aim was to determine if those who trained on a specific talker showed more benefit for their assigned talker than for the three other untrained talkers. According to TAP, we would expect more benefit with the trained talker than for the untrained talker. Figure 4 shows the scores at the baseline and post-training assessments split by talker familiarity. Listening and speechreading scores are combined in Figure 4. The combined listening and speechreading scores were analyzed using a repeated measures three-way ANOVA with talker familiarity (trained versus untrained) and pre-versus post-training scores entered as repeated measures variables. As we saw in the training group analyses above, scores improved from pre-to post training. There was, however a difference between the amount of benefit obtained between the trained and the untrained talkers (F(1,92)=74.2, p<.001, pt eta sq = .446). To investigate the interaction, the amount of benefit in percentage points for the trained (M=12.7, SD=7.8) and the untrained talker scores (M=6.6, SD=7.1) were compared using a paired t-test. Results showed that the amount of overall benefit for the trained talker was greater than the benefit of the untrained talker (Mean diff = 6.1, SD=7.0, t(95)=8.5, p<.001, 95% CI of the Mean Diff = 4.7 - 7.5).

Figure 4.

Average percent word correct scores averaged across listening and speechreading for trained and untrained talkers at baseline and post-training. Error bars indicate standard error.

Predicting Benefit

As a final analysis, we evaluated whether we could predict who would most likely benefit from training based on age, language skills, and phonetic awareness. Hierarchical regression was used to measure the potential predictive value of the raw scores from the PPVT-4, EVT-2, CTOPP-Elision, CTOPP-Blending Words, along with the participants’ age. Two regression models were created, one for each modality (listening and speechreading). The stepwise hierarchical regression predicted post-training scores by first removing the variance associated with their baseline scores, then using a stepwise approach to determine if a participant’s age, language, or phonological processing variables were predictive of any remaining variance. The stepwise approach provided independent estimates of variance accounted for by each predictor variable. Results showed that none of the potential predictor variables were able to predict a significant amount of variance in performance in either listening or speechreading.

DISCUSSION

Overall, the findings of this study are largely consistent with the applicability of TAP theory to speech perception training. Most consistent with TAP were our findings that training with a given talker led to more improvement when the same talker was used for testing. Importantly, however, improvements were also observed for different talkers, albeit not to the same extent; “importantly” because this latter finding suggests that generic speech perception training can enhance children’s ability to understand the speech of various talkers in their everyday linguistic environment. Second, we also observed that children who received AT showed greater improvement on the listening test than on the speechreading test, which is highly consistent with TAP.

Results for the other two training conditions were less clearly aligned with the predictions of TAP. We predicted children who received AVT would improve more on the speechreading assessment than the listening assessment, but they improved equally on both. We predicted that those who received AT/AVT would improve in equal amounts on both, but they improved more on the listening than the speechreading assessment. In hindsight, however, these findings might be viewed as being substantially consistent with TAP theory in that AVT and AT/AVT both present the auditory speech signal, which could lead to improved performance on a listening assessment. In addition, because the AT/AVT condition included stimuli that were auditory only, this could account for why children in this group showed greater gains on the listening than on the speechreading assessment.

In addition to performing a general evaluation of TAP theory, we performed analyses to address three more fine-grain goals. First, we wanted to assess the benefits of speech perception training per se. Results showed that 16 hours of computer-based gamified speech perception training, regardless of presentation type, led to significant improvements in participants’ ability to recognize speech. This was true whether or not improvements were assessed in a listening-only or a speechreading test condition. On average, participants improved their listening scores by 9% words correct and their speechreading scores by 7%. The latter finding complements previous work (e.g., Lalonde & McCreery, 2020; Tye-Murray et al., 2014) suggesting that experience can affect how well children can utilize visual speech information.

A second goal was to assess the benefit of talker-specific speech perception training for children who have hearing loss. The results clearly indicate that children who train with stimuli spoken by a particular talker will learn to perceive the speech of a trained talker to a much greater extent than if they were to train with “generic” talkers, i.e., talkers with whom they would not interact with at a later date or experience during a post-training test session. This conclusion is demonstrated by the significant differences seen in the gains that were accrued between the baseline two and post-training test scores for the trained talker versus the untrained talkers. Participants improved between 10 and 15% words correct on their trained talker as compared to between 5 and 11% words correct on their untrained talkers. Of particular importance is that the increased benefit of trained versus untrained talkers was observed for all four of our talkers. Thus, it is not the case that the advantage for trained talkers is a consequence of differential intelligibility of one or more talkers. One potential application of this finding is that if children receive teacher-specific speech perception training prior to starting a new school year, they will better recognize that teacher’s speech during the first days and weeks of their classroom experience.

For the third goal, to see if improvements generalized to untrained talkers, we found that even though the amount of improvement was greater for the trained talker versus the untrained talker, participants did significantly improve their speech perception scores for the untrained talkers. The implications of this finding is that commercially available speech perception training programs, even though they provide training with generic talkers, potentially will benefit children’s ability to recognize speech during their everyday conversations.

In a final analysis, we found that age, language level, and phonological awareness did not predict training benefits to any meaningful extent. One interpretation of this finding is that most children with hearing loss of elementary school age stand to benefit from speech perception training, whether it be auditory or audiovisual training or a hybrid combination, and that a requisite age, level of language proficiency, or stage of phonological awareness is not necessary for benefits to accrue.

Conclusions

Overall, the findings provide support for the practice of providing children who have hearing loss with structured speech perception training and they suggest that future aural rehabilitation programs might include teacher-specific speech perception training to prepare children for an upcoming school year.

In their systematic review, Rayes et al. (2019) bemoaned that very few investigations considered how long training benefits are maintained and very few considered participants’ subjective impressions of training benefits. In subsequent reports, we will consider the extent to which training benefits are maintained over time and the value of ongoing speech perception training and will consider children’s and parents’ subjective opinions about the value of training programs of this nature.