New language outcome measures for Mandarin speaking children with hearing loss

Abstract

Objective

The paper discusses recent evidence on the assessment of language outcomes in children with hearing loss acquiring oral language.

Methods

Research emphasizes that language tests must be specific enough to capture subtle deficits in vocabulary and grammar learning at different developmental ages. The Diagnostic Receptive and Expressive Assessment of Mandarin (DREAM) was carefully designed to be a comprehensive standardized Mandarin assessment normed in Mainland China.

Results

This paper summarizes the evidence-based item design process and validity and reliability results of DREAM. A pilot study reported here shows that DREAM provided detailed information about hearing impaired children's language abilities and can be used to aid intervention planning to maximize progress.

Conclusion

DREAM represents an example of translational science, transferring methods from empirical studies of language acquisition in research environments into applied domains such as assessment and intervention. Research on outcomes in China will advance significantly with the availability of evidence-based comprehensive language tests that measure a sufficient age range of skills, are normed on Mandarin speaking children in mainland China, and are designed to capture features central to Mandarin language acquisition.

1. Background

Medical advances, as well as technological inventions such as digital hearing aids and cochlear implants, have made significant changes in the likelihood that a child with hearing loss will achieve adequate speech perception, intelligible speech and language competency, though none of these is yet guaranteed for all children (Duchesne, 2015). Despite these advances, it is probable that a child with hearing loss faces a significant delay in exposure to auditory stimulation, leading to delays in the normal course of oral language development through audition. Research findings suggest that children who received an implant even before 12 months had a one year expressive language delay (Manrique et al., 2004). Others (Anderson et al., 2004, Duchesne et al., 2009, Tomblin et al., 2005) found variable degrees of delay with children who received implants before age 2. Lowering the age of implantation does seem to improve the prospects of normal language acquisition but the evidence does not suggest that children with cochlear implants perform like hearing peers on average (Blamey and Sarant, 2011, Duchesne, 2015).

It is critical to assess different components of language achievement, for example, receptive and expressive language skills, as well as vocabulary and grammar. For example Duchesne et al. (2008) revealed that children who received a cochlear implant below the age of 3 had different degrees of delay across different components of language. Markman et al. (2011) studied age of implantation in relationship to language development in different domains (such as vocabulary, receptive syntax and expressive syntax) and found that age of implantation before 18 months had a larger positive impact on the development of receptive and expressive syntax than on vocabulary acquisition.

In addition, the age at which the language performance is assessed can also affect the outcome results. Language assessments done at 36 months of age tend to reveal a significant advantage of early implants (Duchesne, 2015). These early advantages might not continue when children are assessed later: Dunn et al.'s (2014) follow-ups at school age (8–10 years) reveal much less difference in language for early (under 2) or later (2–4 years) age of implantation. In a systematic review of results from children receiving a cochlear implant before the age of three, Duchesne et al. (2008) found residual delays in both vocabulary and grammar for most of their subjects, even after five years with the implant. Geers et al. (2016) find that of their sample of children implanted between 12 and 38 months, one third scored within normal range on a battery of standardized language assessments by age 4.5 years, one third caught up by age 10.5 years, and the remaining third had persistent language delay. Therefore a language assessment that covers a wide age range will be valuable for monitoring language acquisition of children with hearing loss over time.

The demands of school language set new criteria for success for children with hearing loss. A school age child needs to have sufficient linguistic knowledge to learn new words from limited contexts, to make inferences across discourse, and to have a metalinguistic awareness about language. Metalinguistic awareness is the ability to reflect on language, such as recognizing that two words rhyme or have different meanings (Nittrouer et al., 2014). The requirements to produce longer narratives or expository text make new demands for coherence and cohesion that can stretch the demands on a child in new directions (de Villiers, 1991). For all these reasons, assessment of the language proficiency of children with hearing loss must be continued past the preschool level, as subtle difficulties might emerge with the new demands of schooling (Moeller et al., 2015). Vocabulary tests are not completely sufficient in this regard, serving only as a proxy for how much language the child has received, but neglecting the linguistic devices in syntax, morphology, and semantics that make complex human communication possible.

Norm-referenced standardized language assessments are commonly used in outcome research studies for children with hearing loss in western countries (Liu, 2015). Some examples of these assessments include vocabulary tests, such as the MacArthur Communicative Development Inventory (Fenson et al., 2007) and the Peabody Picture Vocabulary Test (PPVT) (Dunn and Dunn, 2007), parent questionnaires, such as Reynell Developmental Language Scale (RDLS) (Edwards and Reynell, 1997), comprehensive receptive and/or expressive language tests, such as Clinical Evaluation of Language Fundamentals (CELF) (Wiig et al., 2003), Preschool Language Scales (PLS) (Zimmerman et al., 2011), Comprehensive Assessment of Spoken Language (CASL) (Carrow–Woolfolk, 1999), Test of Auditory Comprehension of Language (TACL) (Carrow–Woolfolk, 1998), and reading tests, such as Test of Reading Comprehension (TORC) (Brown et al., 1995), and Woodcock Reading Mastery Test (Woodcock, 1987). Other assessments are not standardized, such as transcribed spontaneous language samples, which are time-intensive (Koehlinger et al., 2013; Hammer et al., 2014).

In China, the Mandarin MacArthur-Bates Communicative Development Inventories (Tardif et al., 2008) was adapted from MacArthur-Bates Communicative Development Inventories. It is a vocabulary checklist and normed for the Beijing region exclusively for children age 0–30 months. Lu et al. (2013) developed the Mandarin Expressive and Receptive Vocabulary Test (MERVT). MERVT was normed on 245 normal-hearing children ranging in age between 1;6 to 3;11 (1 years 6 months to 3 years 11 months) in Beijing, China. These regionally normed standardized assessments only measure early vocabulary skills in Mandarin. However, there is a significant need for a standardized comprehensive language assessment that covers a wide age range of language skills that could be used for Mandarin speaking children with hearing loss as well as children with other forms of language delay. The availability of such standardized comprehensive language measures in western countries allows careful comparison of language skills in different domains in children with hearing loss who are users of either hearing aid or cochlear implants against a large group of age-matched hearing peers who make up the standardization sample (Ching et al., 2013, Tobey et al., 2013, Nicholas and Geers, 2013, Yoshinaga-Itano et al., 2010). It is also important to note that standard scores in a norm-referenced assessment should only be used in the regions where the norm was developed. Therefore, standardized language assessments that are normed outside of mainland China or only for a specific city in China have limitations in use for Mandarin speaking children in different regions in mainland China.

2. Introduction

The development of the Diagnostic Receptive and Expressive Assessment of Mandarin (DREAM) was fueled by the need for a comprehensive assessment that was normed on Mandarin speakers aged 2;6 to 7;11 in mainland China, and designed to evaluate language difficulties in children with various etiologies, including hearing impairment. The full details of the development of the test are included in the DREAM test manual (Liu et al., 2015a). Here we provide an outline of the rationale and process by which it was created, report the major results so far that establish its reliability and validity, and evaluate its suitability for research in the field of language outcomes in Mandarin speaking children with hearing loss.

2.1. Test design and content validity

The design process aimed for high content validity through several efforts.

First, the design process included the involvement of a team of researchers representing test design, psychology, statistics, and most especially linguists and speech-language pathologists with extensive knowledge of Mandarin.

Second, a large variety of test items were designed with linguistic experts from mainland China and piloted in the Beijing/Tianjin area. Research suggests that it is imperative in designing language tests that the starting point be the properties of that particular language. Translation of items from tests in a very different language like English is a mistaken procedure that can lead to distortion of the estimation of children's language ability (Peña, 2007). The items were selected to represent the variety of linguistic forms and structures that are acquired by typically developing children from age 2 through 8 who speak Mandarin (Lee, 1982, Lee, 1986, Lee, 1992, Cheng, 1988, Zhou, 2002, Zhou, 2004, Lee and Naigles, 2008, Zhou and Crain, 2009, Zhou and Crain, 2011, Liu, 2009, Liu and Ning, 2009, Li et al., 2010).

Third, items were not only chosen by close attention to the empirical and theoretical literature on language development, but also by evidence on the nature of language deficits in childhood. Although children with mild to moderate hearing impairment can sometimes achieve syntactic competence (Briscoe et al., 2001), most research finds significant delay in syntax in children with severe to profound hearing loss. In English, de Villiers et al. (1994) and de Villiers (1988) report significant delays in comprehension and production of complex embedded forms such as relative clauses in children with severe to profound hearing loss. In Hebrew, Friedmann and Szterman (2006) examined understanding and use of phrasal movement as in relative clauses and topicalized sentences (Szterman and Friedmann, 2014). In other work on children's understanding of questions in two-clause sentences, Schick et al. (2007) found that 6- through 10-year-old children with hearing loss had significant impairment in understanding such structures well beyond 4 years of age, when typically hearing children understand them. The DREAM included items central to assessing language skills, including those highlighted above as difficulties for children with hearing loss as well as children with specific language impairment, such as fast mapping of novel words (Rice et al., 1990, de Villiers and Johnson, 2007), wh-questions (van Der Lely and Battell, 2003, de Villiers et al., 2008), tense/aspect markers (Duchesne, 2015, Rice et al., 1995, Leonard, 1995), and embedded clauses (van der Lely, 1998, Friedmann and Novogrodsky, 2004).

Fourth, it is important to measure not just what children have acquired, that is, the products of learning, but also to assess the process of acquiring new forms (Hirsh-Pasek et al., 2005). The skills required to acquire new forms of language from linguistic context emerge quite early in typically hearing children, but seem to be delayed in children with specific language impairment (Rice et al., 1990; Johnson and de Villiers, 2009) and in children with hearing impairment (Lederberg et al., 2000), partially as a function of their reduced vocabulary size (Pittman et al., 2005). For these reasons, this type of process item is now a part of at least two standardized tests in English, the DELV (Seymour et al., 2005; for data on fast mapping, see Johnson et al., 2009) and a new touchscreen assessment for English and Spanish (de Villiers et al., 2014). Items that tap the process of learning were included in DREAM.

Fig. 1 provides an example of such an item in comprehension. This is an illustration of fast mapping a novel word via cues from the sentence context. It is a process-type item. The classifier (群) that precedes the novel noun dafu implicates that dafu must be animate, ruling out the other unknown entities in the picture because they are artifacts, and ruling out the ducks because they have an existing name. Classifiers in Mandarin have an important role grammatically, more so than English words such as “flock”, which was used as the nearest translation equivalent to make the point. This also counts as a semantics item, together with more standard vocabulary items and more sophisticated phenomena concerning meaning.

Fig. 1.

An example of a Fast Mapping item (From Liu et al., 2015a).

看, 河边有一群dafu。哪张图里的是dafu?

kan4, he2 bian1 you3 yi4 qun2 da1fu1. na3 zhang1 tu2 li3 de shi4 da1fu1?

“There is a flock of dafu by the river side. Which picture shows dafu?”

Fifth, most importantly, however well-founded in theory the type of item was, an item was only considered usable if it proved discriminating for children with more or less language ability, so great care was taken in the item design to maximize discriminability across the age range. After extensive piloting and tryout in different regions within mainland China, a final set of items was selected by subjecting the data to Rasch analyses (a variety of Item Response Theory IRT; see e.g. Embretson and Reise, 2000).

In addition to the item design and selection criteria discussed above to ensure DREAM's content validity, close attention was paid to potential confounds with different dialects influencing a child's acquisition of Mandarin in different parts of China (Liu et al., 2015a).

2.2. Test design and quality control

Based on the scope of practice for the professionals working with individuals with language disorders (American Speech-Language-Hearing Association, 2007), the publishers of standardized language assessments in the West usually only allow people with rigorous qualifications, such as advanced degrees in speech-language pathology, to administer the tests and interpret the results. Since trained personnel in this discipline are in short supply in China, two strategies were used in the design of DREAM. First, certificate training that is associated with DREAM evaluation and speech-language therapy is required for any personnel before they may administer DREAM. Second, a computerized test was developed, with the presentation of stimuli fully standardized, and the child's responses automatically recorded via a touchscreen for comprehension, or for production, by a menu of simple options for the test administrator. The program automatically generates a professional report. These designs assist Chinese professionals to meet the highest international standards in administering a comprehensive standardized language assessment.

2.3. Standardization

The sample for standardization of the instrument consisted of 969 Mandarin-speaking children between the ages of 2;6 and 7;11, with about equal numbers of boys and girls. Between 2;6 and 5;11 years, half-year age groups were distinguished, with year-long age groups for 6 and 7 year olds. Sampling included multiple cities and suburbs in both the Northern and Southern regions of China, and was stratified by multiple variables such as age, gender, urban versus suburban, region, and parental education level according to the most recent census data. All data are automatically recorded and scored through a computer system. The norms derived by age group are Receptive and Expressive skills, as well as Semantics and Syntax skills.

2.4. Reliability

The details of the various reliability indices are to be found in Liu et al. (2015b, submitted) but we report them briefly here. With any standardized test it is important to establish the internal consistency of the test, that is, the extent to which the items cohere, measured by Cronbach's alpha. The value of .94 shows that the items have considerable inter-correlation and therefore measure a common construct, language ability. Test-retest results show good consistency, namely .85 between tests.

2.5. External validity

A further sample of 230 children aged 2;6 to 7;11 tested at Shanghai Children's Medical Center, was used to study external validity. That sample included 94 children identified by pediatricians as language delayed. Children with other sensory impairments or autism and children who did not speak Mandarin when playing with peers were excluded from this study (children whose parents spoke a dialect at home were not excluded for this reason alone) (Liu et al., 2015b; submitted). It is necessary to show adequate convergent validity by determining convergence between the measure and some other recognized indices of language growth. In the absence of another satisfactory standardized test of Mandarin normed in mainland China, we compared it to indices such as diversity of sentence types, morphemes and vocabulary use, coded in spontaneous speech elicited within a specially designed play session for children up to age 4;5 (compare e.g. Geers et al., 2016 and Pearson et al., 2014). For children ages 4;6 through 7;11, well-founded indices of cohesion in elicited short narratives were adopted (Burns et al., 2012, Seymour et al., 2005). The DREAM scores showed highly significant correlations with these independent and quite different measures (for details see Liu et al., 2015b; submitted). The external validity study also included a study to evaluate the assessment's sensitivity and specificity. DREAM distinguished the children satisfactorily given that the a priori division was from the judgment of skilled pediatricians based only on detailed parent report, not by another established gold standard language assessment.

The results of standardization were also used to recognize markers representing specific areas in which a given child showed weakness relative to his or her peers. These provide guides for the speech-language therapists on precise areas of difficulty for further dynamic assessment and appropriate individualized language therapy.

Here we present a small pilot study to test the viability of using the DREAM tool for children with hearing loss who speak Mandarin. The goal was to see if the children could take the computerized test and respond to its receptive and expressive demands. It was expected given work on children with CIs tested in other languages that there would be some scatter in results across the domains, and that all scores would be well below the standard scores for typically hearing children matched in age.

3. Methods

3.1. Participants

A small pilot study (N = 9) was also conducted to test the feasibility of the instrument for children with hearing loss. Participants in this study were well-defined: they were between 3;4 to 6;4 and had a bilateral severe to profound or profound sensorineural hearing loss. No other medical diagnosis was reported other than the hearing impairment. Age of implantation was between 1;2 to 4;11 (Mean = 3;0). All children started aural rehabilitation within two months after the implantation. The hearing age of these children was between 0;11 to 3;5 (Mean = 2;1). Table 1 shows the characteristics of this pilot group.

Table 1.

Characteristics of the subjects in the pilot study.

Age	Gender	Degree of HL -L	Degree of HL —R	Age at implantation	CI ear	CI processor	HA	Type of HA	Hearing age at DREAM test
3;4	F	Profound	Profound	2;2	R	Cochlear Freedom	no		1;2
4;8	F	Profound	Profound	3;9	R	Cochlear Freedom	left	Naida-3	0;11
4;4	F	Profound	Profound	1;2	R	AB Harmony	no		3;1
4;10	F	Profound	Profound	3;0	R	Med El COMBI40+	no		1;10
4;5	F	Profound	Severe	2;8	L	Cochlear 3G	no		1;9
5;10	M	Profound	Profound	3;8	R	Cochlear 3G	no		2;1
6;2	M	Profound	Profound	2;9	R	Cochlear 3G	no		3;5
6;4	F	Profound	Profound	4;11	R	Cochlear 3G	no		1;5
6;0	M	Profound	Severe	2;11	R	Cochlear 3G	no		3;2

3.2. Procedure

The instrument is implemented on a touchscreen, with a standardized narration provided by a female Mandarin speaker who works in a professional capacity on a children's radio program. The child responds directly to the touchscreen for the comprehension items, and for the expressive items, the examiner chooses from a menu of possible answers for entering the child's responses. The entire test takes about 45 min to complete. All children were tested by trained examiners.

4. Results

The data from the children with hearing loss are shown in Fig. 1, using z-scores adjusted for comparison with their chronological age peers. The small sample of CI children consistently showed index scores − 1 to −2 SDs below the mean across all the language domains: receptive, expressive, syntax and semantics. The impact of hearing loss is evident in lower scores across all the language domains. The expressive score was impacted more than the receptive score, and semantics was impacted more than syntax (Fig. 2).

Fig. 2.

Performance of nine children with cochlear implants in pilot study across the language indices.

Inspection of the performance in different linguistic subtypes also reveals the predicted pattern of considerable variation in performance, not readily attributed to age at implantation or hearing age. On different linguistic subtypes, such as fast mapping new words, answers to wh-questions, or producing relative clauses, there were individual children who scored in the normal range, while the majority scored very poorly. But as the graph also indicates, a given child who achieved a normal score on some linguistic type/s then did not score well on the others.

5. Discussion

These preliminary results are in keeping with research that children with hearing loss learning English have a reduced vocabulary size (Pittman et al., 2005) and have difficulty with the process of learning new words (Lederberg et al., 2000). However, our finding of better syntax than semantic skills is not in keeping with a small study by Young and Killen (2002), who found in the language of 7 children who had used a cochlear implant for 5 years that their semantic skills were stronger than their syntactic and morphological skills. They used for testing the Clinical Evaluation of Language Fundamentals-3, the Peabody Picture Vocabulary Test-Revised, and the Expressive Vocabulary Subtest of the Test of Word Knowledge (Young and Killen, 2002). Variability in performance occurred between subjects and within subjects across subtests. More work will be needed with larger samples and a close comparison of test contents to resolve this discrepancy.

In a longitudinal study by Geers et al. (2009) comparing spoken language between cochlear implant users and matched hearing peers, 47% of the CI users achieved age-appropriate scores on receptive language, and 39% of the same children achieved age appropriate scores on expressive language.

While general trends of overall language scores are similar, it is clear that more research is needed to understand the rate and developmental course of language acquisition for children with hearing loss, especially with respect to particular linguistic skills. In current research studies it is common to find considerable scatter of language outcomes for the same level of hearing impairment, suggesting that many other factors would have to be matched in future studies (Blamey and Sarant, 2011, Ching et al., 2013, Duchesne, 2015, Moeller et al., 2007, Tobey et al., 2013).

Having reviewed outcome studies in the US and Europe on children with hearing impairment (Liu, 2015), it is clear that comparison studies in China will be enhanced by the availability of a comprehensive standardized assessment. The pilot study provides initial evidence that the computer and touchscreen procedures are suited to the testing of children with hearing loss. Furthermore the data are in keeping with other studies that demonstrate a variety of outcomes with children with CI, with average performance falling below the mean of hearing peers. Although this is a very small study, the results suggest that there may be differential impact across different indices of language.

In the review, it was noted that several studies of children with hearing loss find marked disparities in their performance in different domains of language (receptive language, expressive language, syntax, semantics, and total language). Standard language assessments that examine multiple language domains, such as DREAM, allow different language domains to be researched in outcome studies. There is much more to be learned about how children with hearing loss develop competence in Mandarin. A standardized test is just one step towards this goal. Much intensive research needs to be conducted to help understand the impact of hearing loss on the growth of linguistic competence in diverse languages. Only when we understand the barriers to full mastery and can identify missed or disordered steps, can successful remediation be achieved.

DREAM is normed on hearing children who speak Mandarin from ages 2;6 to 7;11 years. The age range overlaps with the age of beginning of grammar skills (usually 2 and half to 5 years old), but more assessments normed in China are needed that tap earlier skills such as symbolic play and shared attention. At the upper end, the assessment is extended into the age of beginning literacy skills (usually 6–8 years old), which depends heavily on the successful achievement of oral language skills in the hearing population (Storch and Whitehurst, 2002, National Early Literacy Panel (NELP), 2008) and in children with hearing loss (Nittrouer et al., 2012, Szterman and Friedmann, 2014). It is recognized that children with hearing loss may be delayed by a year or more in achieving oral language skills relative to hearing peers. In fact, it has been argued that standardized tests may underestimate the limitations that children with hearing loss have in language for school, where they are in competition with hearing peers and under greater communication pressure (Tomblin et al., 2015). For the child who has hearing loss to be successful in education, it is necessary to evaluate and remediate language at school age as well as at the preschool age. In future work, one research goal should be to design and validate additional tests of more advanced skills such as narrative use and reading skills, to complement the measures in DREAM.

With the availability of a standardized instrument for language testing in China, what other kinds of studies might be done? A large body of research in the West has looked at age of implant effects on language outcomes. A standardized test (such as DREAM) makes it possible to assess Mandarin speaking children with hearing loss in a way that is both standardized and linguistically sophisticated, complimenting vocabulary tests and parental checklists. Using a standardized outcome measure makes it feasible to consider differences in language outcome in children with hearing loss in China as a function of external variables such as age of implantation, hearing technology (e.g. types of hearing aids and cochlear implants, unilateral versus bilateral implantation, sequential versus simultaneous implantation), type of therapy intervention, and type of education programs. Additionally, standardized language tests allow comparison of language profiles across etiologies of hearing loss, including the broad new range of genetic defects being identified (Torriello et al., 2004, Usami, 2015). Hawker et al. (2008) tried to explain the “disproportionate language impairment (DLI)” of children who performed more poorly at 7 years post-CI than a closely matched group with similar CI experience. Hawker et al. made the suggestion that the DLI group may in fact be genetically distinct, perhaps sharing features of specific language impairment (SLI) seen in hearing children. The Childhood Development after Cochlear Implantation (CDaCI) studies reported in their large scale, multicenter longitudinal studies, that factors such as earlier age of implantation, greater residual hearing prior to CI, higher ratings of parent–child interactions, and higher SES were associated with greater rates of growth in comprehensive and expressive language (Niparko et al., 2010). With the availability of a comprehensive standardized language assessment normed in mainland China, multicenter longitudinal studies such as the CDaCI studies can be conducted to investigate language outcomes for children with hearing loss in Mainland China. The types of outcome studies that can be conducted using a standardized language instrument are indeed varied and extensive in nature.

Conflict of interest

This paper was funded by Bethel Hearing and Speaking Training Center.