Are we slipping them through the cracks? The insufficiency of norm-referenced assessments for identifying language weaknesses in children with hearing loss

Abstract

Children with hearing loss who use listening and spoken language increasingly reach performance within or above the average range on norm-referenced assessments of language ability prior to entering school; however, they continue to perform below expectations on language-based academic skills, such as reading. The purpose of this article was to identify limitations of making service provision decisions primarily on the basis of norm-referenced assessments for children with hearing loss. We offer evidence from a group of children with hearing loss who scored within 1.5 standard deviations of the mean on a norm-referenced omnibus language measure. These children with hearing loss performed more poorly than age-matched children with normal hearing on several measures derived from a naturalistic language sample. Based on the limitations of norm-referenced assessments and the findings of this study, we propose that language sample analysis should be used as primary evidence of language weakness for children with hearing loss.

Introduction

It is widespread knowledge that many children with hearing loss (CHL) identified in infancy and fit with modern-day listening devices (digital hearing aids and/or multi-channel cochlear implants) prior to the age of two years can achieve spoken language proficiency levels, as determined by norm-referenced assessments, comparable to same-aged peers without hearing loss before or by the end of their preschool years (Geers, Nicholas, Tobey & Davidson, 2016). A child’s language proficiency is the characterization of his or her language skills relative to some benchmark or expectation (Spaulding, 2012). When assessing the language proficiency of CHL who are learning spoken language, this comparison traditionally has been made relative to their same-aged peers without hearing loss using standardized assessments (Geers, Moog, Biedenstein, Brenner & Hayes, 2009; Geers et al, 2016). These language proficiency determinations are derived based on their performance during highly controlled, linguistic-specific tasks on norm-referenced tests (see Hresko, Reid, & Hammil, 1999; Semel, Wiig & Secord, 2004). Qualified professionals who work with CHL and administer such assessments, such as speech-language pathologists and/or deaf educators, report the results to determine the need for, or to justify denying, specialized services (Skahan, Watson & Loff, 2007). Instead of looking for mismatches between a child’s performance on standardized assessments and naturalistic phonologic, morphologic, syntactic, semantic, and pragmatic use in realistic contexts, the need for specialized language services are often determined primarily by those who achieve standard scores that are 1.5 standard deviations below the mean on a norm-referenced language assessment (e.g., Nowland, 2009).

The practice of using psychometric test results as the predominant method for determining language status and eligibility criteria, however, has been viewed as limited for decades (Dunn, Flaxx & Sliwinski & Aram, 1996; Spaulding, 2012; Spaulding, Plante, & Farinella, 2006). It has been criticized from a clinical perspective because of (1) a lack of adequate reliability and validity of assessment instruments, (2) the tendency to focus on single aspects of language, (3) the reliance on a false sense of objectivity, (4) the failure to use qualitative information, and (5) the psychometrically incorrect practice of comparing test scores measuring language abilities with test scores measuring other abilities (Dunn et al., 1996). Continuing this practice undermines appropriate clinical and educational interpretations, and it is not consistent with US law (Pena, Spaulding, & Plante, 2006; IDEA, 2004). For instance, the law says a variety of informal and formal assessment tools should be utilized to determine eligibility (IDEA, 2004). In practice, however, published state guidelines often suggest that non-standardized testing procedures, such as language sample analysis, should be used to verify the results of norm-referenced scores (e.g., Noland, 2009; Minnesota Administrative Rules, 2007; North Carolina Guidelines, 2006). State guidelines such as these indicate that assessments (e.g., language sample analysis) should be used to determine if low language proficiency has a negative effect on educational performance only after a student has scored below the cut-off on a norm-referenced language measure.

Using norm-referenced language assessments as the gold standard in diagnosing language impairment is problematic. A child’s performance on a standardized norm-referenced language assessment may not represent the child’s use of language structures in his or her spontaneous, conversational language (Dunn et al., 1996). This is because a limited number of items measures structures of interest under highly controlled, operant conditioning procedures in a cloze task format (e.g. “This boy is a soccer player. They are all soccer _____ [players]”). Children may be able to use language structures in such highly controlled environments but not in their everyday spontaneous language. Additionally, particularly in the case of CHL, children repeatedly are administered the same norm-referenced measures, which may result in their learning of the task demands and perhaps even test items. A child with HL who has nonverbal intelligence in the average range or higher, and who has been in spoken language therapy for many years, may respond to the subtest items correctly because he/she is conditioned to the cueing procedures, and not because a particular test item is representative of their typical language use. Indeed, achievement on these test items neither provides proof of, nor indicates proficient use of, such language knowledge in their spontaneous, conversational language. Dunn and colleagues (1996) concluded that language sample analysis was more effective than norm-referenced assessments for identifying language impairments.

For the purposes of this paper, we focus on Brown’s grammatical morphemes. Brown (1973) identified 14 morphemes typically mastered or demonstrated in conversational language in children with normal hearing (CNH) by the age of four years. Grammatical morphology has been identified as a clinical marker for language impairment in preschool children (Leonard, Eyer, Bedore, & Grela, 1997; Rice & Wexler, 1996) Additionally, CHL are notoriously delayed in grammatical morphology (Moeller, Tomblin, Yoshinaga-Itano, McDonald-Carter, & Jerger, 2007; Tye-Murray, 2013). In addition to the widespread tense-marking difficulties experienced by children with language impairments (Leonard et al., 1997; Rice & Wexler, 1996), grammatical morphology difficulties experienced by CHL likely are a result of the unstressed, low intensity (dB) and quick timing acoustical properties of English morphemes. These difficulties make careful assessment of the understanding and use of this fine feature of spoken English particularly warranted for CHL. Language delay that persists beyond the preschool years is highly predictive of poor reading and academic outcomes; therefore, current assessment methods must validly expose any pertinent linguistic deficits (Geers, et al, 2016; Scarborough, 2001; Zambrana, Pons, Eadie, & Ystrom, 2014). The primary use of standardized scores, particularly for CHL, may not be the most appropriate method of making eligibility decisions and could reasonably have the potential to sorely mask these children’s continuing needs. However, at this time, we have no evidence in the extant literature on CHL who use hearing aids and/or cochlear implants to support this concern.

The purpose of this paper was to identify potential limitations of making service provision decisions primarily on the basis of standardized assessments for a group of CHL who use listening devices, such as digital hearing aids and/or multi-channel cochlear implants. Measures taken from spontaneous language samples of preschool CHL were compared to a matched group of typically developing children without hearing loss. Both groups demonstrated standardized scores on norm-referenced assessments that would not typically qualify for services through standard psychometric criteria. We asked the following research question: Does grammatical morpheme language use in spontaneous conversational language reveal significant differences between CHL who use hearing aids and/or cochlear implants and children with typical hearing? We hypothesized that the grammatical morpheme language analysis of spontaneous conversational language samples would reveal differences in expressive language use between the two groups that were not identified by standardized norm-referenced language measures.

Method

Study procedures were approved by the University of South Carolina Institutional Review Board. Language sample assessments were completed with 13 preschool CHL and 13 CNH matched within one month of chronological age. The children were recruited from preschools and speech-language pathology/audiology clinics from six states across the Southeastern and Midwestern United States (Florida, Illinois, North Carolina, Ohio, South Carolina, Tennessee). The two groups did not differ on standardized test scores for nonverbal intelligence, as measured by the Primary Test of Nonverbal Intelligence (Ehrler & McGhee, 2008; CHL mean = 115.69, SD = 12.45; CNH mean = 112.23, SD = 10.80; p = .456). The two groups also did not differ on overall spoken language as measured by the Test of Early Language Development – Third Edition (TELD-3; Reid, Hammil, Hresko, 1999; CHL mean = 103.62, SD = 16.22; CNH mean = 109.92, SD = 12.89; p = .283). The TELD-3 uses brief, simple receptive and expressive tasks to assess the broad picture of a child’s language development specifically in the areas of semantics, syntax, and morphology. On the TELD-3, all CHL had standardized language assessment scores within 1.5 standard deviations of the test mean. Table 1 provides additional demographic information of the CHL.

Table 1.

Demographic Information of Children with Hearing Loss

Child	Overall Language Standard Score	Gender	Chronological Age	Amplification Type	Age at Identification (in months)	Age at Amplification (in months)
1	77	Girl	4;2	Bilateral CI	0	2
2	113	Girl	4;3	Bimodal	1.75	3.75
3	116	Boy	4;3	Bilateral CI	1.5	36
4	108	Boy	4;5	Bilateral HA	0	1.5
5	82	Boy	4;8	BAHA	0	not reported
6	88	Girl	4;2	Bilateral CI	3	3
7	115	Boy	3;9	Bilateral CI	1.5	3
8	80	Boy	5;0	BAHA	0	12
9	119	Girl	4;0	Bilateral HA	0	9.5
10	110	Girl	4;7	Bilateral HA	0	4
11	103	Boy	4;4	Bilateral HA	14	19
12	125	Boy	4;4	Bimodal	0.5	3
13	111	Girl	4;10	Bilateral HA	36	36

A 12-minute language sample was elicited from each child, using the Hadley (1998) protocol for conversational interview samples. Language samples primarily included the child and the examiner; occasionally a parent was also in the room. The Hadley (1998) protocol includes three four-minute blocks. The first block elicits personal narratives; for example, the child is asked to tell about a birthday party or recent holiday, as well as siblings. The second block elicits expository explanations; the child is asked to explain how they take care of a pet and how to play their favorite sports or games. The third block elicits story retelling; the child is asked to tell what happens in their favorite movies and television shows. Because the Hadley (1998) protocol was developed for use with school-age children, we utilized pictures to support conversation in each block for preschool children. We chose to use the Hadley protocol, because interviews that contain expository and story retelling contexts for language samples elicit longer, more varied utterances and measure a greater range of linguistic capabilities (e.g., Evans & Craig, 1992; Masterson & Kamhi, 1991).

Language samples were audio-recorded to allow for accurate transcription of dialogue. There were three steps in the transcription process. First, a trained undergraduate student made an initial pass, transcribing the child and adult utterances. The goal of this pass was to get on paper what was said during the interaction. Next, a trained lab employee “cleaned up” the transcription. During this step, the employee divided utterances, marked utterance overlap, and added gloss and contextual lines to aid interpretation of the interaction. Appendix A contains a description of how utterance boundaries were determined. Finally, a senior lab member who was not involved in the previous steps performed a final pass. The purpose of the final pass was to ensure accuracy. If errors were discovered, the transcript was sent back to the second step for correction.

After transcriptions were finalized, we coded the language samples for Brown’s 14 grammatical morphemes (Brown, 1973). Table 2 illustrates this coding scheme. Errors, including errors of Brown’s morphemes, as well as any other grammatical, semantic, or pragmatic errors, were marked with an [err] code. Omissions of bound morphemes, as well as articles and prepositions, were marked with an asterisk (*). For example, if a child said, “she talk the phone three time yesterday,” the utterance would be coded as “She talk/*ed *on the phone three time/*s yesterday.” The first step of coding each transcript involved a trained lab member making a first pass of grammatical morpheme coding. Next, a second trained lab member checked the coding. Any disagreements were resolved by discussion, resulting in 100% agreement of the final transcripts.

Table 2.

Brown’s Grammatical Morphemes Coding Scheme

Morpheme	Code	Example
Present Progressive	/ing	We’re read/ing the book.
Preposition in	in	Dad put the cup in the sink.
Preposition on	on	The book is on the table.
Regular Plural	/s	The cat/s drank milk.
Irregular Past Tense	[ptirr]	I caught [ptirr] the ball.
Possessive	/z	My mom/z car is outside.
Uncontractible Copula	[unconcop]	He was [unconcop] happy.
Articles	a an the	I had a chocolate cake. Mom gave me an apple. We are going to the zoo.
Regular Past Tense	/ed	The boy jump/ed in the puddle.
Regular Third Person Singular	/3s	The girl like/s dogs.
Irregular Third Person Singular	[3irr]	Daddy says [3irr] no.
Uncontractible Auxiliary	[unconaux]	They were [unconaux] sleeping.
Contractible Copula	[concop]	He is [concop] hungry.

After the coding process was complete, we utilized SALT Software (Miller & Iglesias, 2012) to analyze each sample. First, we used the Standard Measures function to calculate number of total utterances, number of complete and intelligible utterances, mean length of utterance in morphemes (MLU), and number of different words (NDW). Second, we used the Explore function to search for each coded feature. Appendix B contains our Explore lists for grammatical morpheme analysis. Finally, we used the SALT output to calculate percent correct, percent omissions, and percent errors for each grammatical morpheme.

Results

In contrast to the similar performance between the groups on the norm-referenced standardized language measure, the grammatical morpheme language analysis of spontaneous conversational language revealed differences in expressive language use between the two groups. CHL performed lower than CNH on many measures of their productive conversational language use. The CHL had an average MLU of 4.63 (SD = 1.73), compared to 5.66 (SD = 1.14) for CNH (p = .084; d = 0.71). Additionally, CHL exhibited less lexical diversity than CNH. The CHL had an average NDW of 150.38 (SD = 47.59), compared to 188.23 (SD = 41.12) for CNH (p = .040; d = 0.85). CHL also had a higher percentage of utterances that contained omissions of words or morphemes (mean = 15.09%, SD = 8.34) compared to CNH (mean = 3.61%, SD = 2.70; p < .001; d = 1.85). Finally, CHL and CNH did not differ on the total number of utterances produced during their 12-minute sample (CHL mean = 136.15, SD = 27.98; CNH mean = 137.15, SD = 27.55; p = .975).

When examining specific grammatical morphemes, several differences between the groups also emerged. As a group, the CNH reached performance levels typically considered “mastery” (i.e., 80%) in language intervention for all of Brown’s grammatical morphemes. In contrast, the CHL reached such levels only for present progressive, prepositions, and articles. Cohen’s d effect sizes additionally revealed small group differences for present progressive and prepositions, indicating minimal clinical significance. Large group effects were observed for all tense-marking morphemes, as well as regular plurals, possessives, articles, and some Basic English (BE) verbs (uncontractible copula and contractible auxiliary). Other BE verbs had medium group effects (contractible copula and contractible auxiliary). Table 3 displays percent correct of attempts for each group on each grammatical morpheme.

Table 3.

Percent Correct of Brown’s Grammatical Morphemes

Morpheme	CHL Mean (SD)	CNH Mean (SD)	p	d
Present Progressive	100.00 (0.00)	95.83 (14.43)	.339	0.41
Preposition in	86.00 (25.40)	93.92 (9.41)	.348	0.41
Preposition on	82.83 (30.26)	91.54 (15.94)	.372	0.36
Regular Plural	74.85 (31.62)	98.54 (3.26)	.019	1.05
Irregular Past Tense	66.62 (25.34)	85.34 (13.24)	.030	0.93
Possessive	45.60 (41.85)	93.08 (16.56)	.063	1.49
Uncontractible Copula	62.70 (40.28)	92.33 (10.39)	.047	1.01
Articles	87.69 (15.63)	97.08 (3.62)	.054	0.83
Regular Past Tense	35.90 (35.43)	80.09 (26.13)	.004	1.42
Regular Third Person Singular	45.63 (32.71)	86.08 (12.54)	.001	1.63
Irregular Third Person Singular	44.44 (45.13)	83.24 (21.43)	.016	1.10
Uncontractible Auxiliary	73.89 (36.38)	92.56 (16.92)	.190	0.66
Contractible Copula	72.08 (27.65)	88.62 (19.53)	.091	0.69
Contractible Auxiliary	65.00 (42.98)	93.75 (15.87)	.088	0.89

Discussion

The purpose of this paper was to expose any limitations of making service provision decisions primarily on the basis of standardized assessments for a group of CHL who use listening devices, such as digital hearing aids and/or multi-channel cochlear implants. We compared measures taken from spontaneous language samples of preschool CHL matched to a group of typically developing children without hearing loss. Both groups demonstrated standardized scores on norm-referenced assessments that would not typically qualify for services through standard psychometric criteria. Our hypothesis was correct in that despite similar performance between the groups on the norm-referenced standardized total language measure, the grammatical morpheme language analysis of spontaneous conversational language revealed significant differences in expressive language use between the two groups. CHL performed significantly lower than CNH on many measures of their productive conversational language use, including mean length utterance, the number of different words, and the use of Brown’s grammatical morphemes. Clearly, for some CHL, these language forms are not developing similarly to CNH. This dissimilarity is despite the presence of intense, spoken-language intervention and participation in rich, linguistic environments. Additionally, the primary use of a total language standard score to determine comparable achievement did not expose these deficits. The careful assessment of Brown’s morphemes in a child’s spontaneous language is important because difficulty in this area could negatively impact a child’s future academic-linguistic ability to effectively describe events and make logical comparisons (Semel et al., 2004). Limitations to the current paper include small group sizes and the use of one norm-referenced language assessment. Our current concerns can only be related to results of the TELD-3. Future research should include a deeper look at other standardized morpho-syntactic assessments to determine their sensitivity to the listening and spoken language needs of CHL, as compared to their conversational performance on spontaneous language samples. Finally, we would like to replicate this study to compare any differences in similar linguistic performance of those who use specific hearing devices, including the effects of specific sound coding strategies.

Conclusions

Although current practices for the determination of eligibility for special services is often primarily based on performance of standardized assessment (Geers et al, 2008; Skahan et al, 2007), the use of objective measures from spontaneous language samples appears to be a more valid approach to the identification of CHL who might continue to need specialized services despite standard scores that indicate performance within or above the average range for their chronological age on psychometric measures. Skipping this process (by choosing assessment activities that take the least amount of time) puts the clinician in the unfortunate position to misdiagnose and undermines recommendations intended to achieve maximal benefit. Although some may argue that language sample analysis procedures are too time consuming, such assessments can be especially efficient if the practitioner collects and analyzes language samples throughout the year (Douglas, 2016). To further mitigate time and fidelity challenges associated with language sample analysis procedures, we also recommend that clinicians become familiar with elicitation protocols before utilizing them with children. The use of audio or video recording should be used to ensure accurate transcription of child utterances and errors.

Based on the results of this study, the benefit of deeper assessment for the CHL outweighs the cost of time. We propose that scores above 1.5 standard deviations below the mean should not be an indicator for the refusal to provide services or the need for dismissal. Instead, language sample analysis provides better insight into the everyday language use of CHL and represents a more valid assessment strategy than norm-referenced tests. By utilizing language sample analysis and examining specific grammatical morphemes therein, skilled service providers could lower the risk for persistent delays in pre-academic language skills. With the goal of maximizing each child’s benefit from their listening device(s) so that competitive linguistic performance is utilized in mainstream environments, it is the hope of these authors that readers seriously consider this information when making appropriate eligibility decisions regarding the continuation of service provision for CHL.

Appendix A

How to Divide Speaker Talk into Utterances

adapted from Schuele (2009)

1. Each utterance begins on a new line, even when spoken by the same speaker.

2. Pauses and intonation can provide general guidelines for determining the end of an utterance. Importantly, this information cannot be used as an unerring strategy for dividing utterances. Syntactic information must also be considered.

3. Utterances should be divided at syntactic boundaries, even if intonation or pause information does not indicate the end of an utterance. For example, my dog’s big he likes to run is divided into two utterances even if there is no pause (or intonation change) to mark utterance boundaries.

4. Sometimes speakers pause before completing an utterance or another speaker interrupts them as they complete an utterance. Maintain syntactic integrity of utterances by following the guidelines below:

   • 4a. If a speaker pauses and then completes an utterance, maintain as one utterance and mark the pause with a colon.

     • C My favorite one in frozen is : anna.
   • 4b. If a speaker is interrupted but then completes the utterance, maintain as one utterance and mark the interruption with <>.

     • C My favorite one in frozen is <> anna.
     • E <elsa>?
     •   NOT: C My favorite one in frozen is>
     •     E elsa?
     •     C anna.

5. Recognize the differences in spoken and written language. Many utterances are acceptable in spoken language that we would not generally use in written language – for example, olaf, he’s a silly snowman should be considered one utterance, not two.

Dividing speaker talk that contains complex syntax into utterances.

1. Do not subdivide utterances with embedded clauses.

   • C I like the dog that is barking (2 clauses).
2. If a speaker joins more than two independent clauses, break the utterance after the second independent clause. Note the break by including the code [bu] at the beginning of the broken utterance:

   • C elsa made a snow monster and olaf spread a lot of fire.
   • C [bu] and then he made olaf disappear and he melted.
3. Follow these rules for subordinate clauses:

   • 3a. If a speaker joins one main clause and one subordinate clause, leave this as one utterance.

     • C I like olaf because he’s silly.
   • 3b. If a speaker produces multiple subordinate clauses, break the utterance after two subordinate clauses.

     • C because he’s silly when he falls.
     • C [bu] when they are singing.
   • 3c. If a speaker produces a main clause and more than one subordinate clause, allow the main clause and one subordinate clause to constitute one utterance. Break the utterance and mark with [bu] for the second subordinate clause.

     • C I like olaf because he’s silly.
     • C [bu] when he falls.
   • 3d. If a subordinate clause is embedded within another clause, do not break the utterance.

     • C I like olaf because when he falls he’s silly.

Appendix B

SALT Explore List for Grammatical Morpheme Analysis

/ing

/*ing

/s

/*s

/z

/*z

/3s

/*3s

/ed

/*ed

a

*a

an

*an

the

*the

in

*in

on

*on

[3irr]

[ptirr]

[concop]

[unconcop]

[conaux]

[unconaux]