Abstract
Hypothesis
Undesirable behaviors in young children with cochlear implants suggest device soft failure.
Background
Cochlear implant soft failure refers to non-optimal performance not detectable with routine hardware checks. Pediatric failures may delay language development, but failure detection is difficult. A 2005 soft failure consensus statement recommended a checklist for suspected device malfunctions. That checklist included the appearance of "bad" behaviors and aggression (externalizing behaviors) or self injury and inattentiveness (internalizing behaviors) as indicators of soft failure. Accordingly, these behaviors should predict slowed language growth, and the analyses reported here sought evidence of that predictive power.
Methods
Data from a longitudinal study of 80 children with cochlear implants collected at six times between 18 and 48 months were re-examined. Language measures included auditory comprehension, expressive vocabulary, and unstructured language. A parent questionnaire, the Child Behavior Checklist, examined externalizing and internalizing behaviors. Behavior measures were correlated with language measures in a series of analyses.
Results
Externalizing and internalizing behaviors did not consistently correlate with language at the ages tested. Additionally, early behaviors did not predict later language abilities. Individual language measures correlated best with overall language development 12 months later.
Conclusions
This study fails to support the hypothesis that externalizing and internalizing behaviors in pediatric cochlear implant users correlate with slowed language advance. These behaviors should not be seen as evidence of declining language performance as may be seen with device soft failure. Instead clinical assessments of language abilities are necessary.
Introduction
Cochlear implant (CI) “soft” failure is a working diagnosis that is supported by clinical, audiologic, and radiographic criteria (1–2). Typically, patients present with a deterioration in their previous implant performance, a failure to progress in language development, or new symptoms such as facial nerve stimulation or pain with use of the implant. Four criteria identified in a 2005 consensus statement on identification of CI soft failures included 1) the exclusion of detectable hardware- or software-related causes, 2) exclusion of medical problems that could lead to device failure, 3) radiographic evidence of proper device and electrode array placement, and 4) improved function or symptom alleviation with device re-implantation (2).
Diagnosis of pediatric soft failure may be significantly more challenging than for adults, and even defining the term soft failure in children is difficult. It has been stated, “[Soft] device malfunction is suspected but cannot be proven. It is a working diagnosis, based on characteristics such as shocking sensations, popping sounds, intermittency, or unexplained progressive decrement in performance” (2). Marlowe et al. (2009) described 18 cases of pediatric device reimplantations for device soft failure (there termed “suspected failures”), which was defined as “diminished or stagnant speech perception.” Many young children are not able to verbalize their experience of new symptoms or a deterioration in speech perception. Moreover, the rate of language development is variable among pediatric CI users (3), so it may not be apparent if a child is straying from his or her previous trajectory of language development. Device failures may be missed in children unless an observant caregiver or teacher detects a decline in language abilities or poorer-than-expected progress. Sensitive clinical tools are not readily available to identify these children, so the identification of device soft failure continues to equate with poor language development or a decline in language performance.
Several authors have reported their institutions’ incidence of soft failure (sometimes referred to as suspected device failure) in pediatric CI users. Marlowe et al. reported a revision pediatric cochlear implantation rate of 12.9% in 482 CI surgeries, 29% of which were performed for suspected device failure (4). In another large series of pediatric implants, Brown et al. reported a 7.3% implant revision rate with 23% performed for soft failure (5). Cullen et al. reported a 11.2% revision rate in 952 pediatric CI users, with a 15% soft failure rate (6). In each series, soft failure was diagnosed by identification of aversive symptoms, a decrement in language performance, or a failure to progress appropriately in language development. Considering the difficulty posed by trying to identify language decrement or failure to progress in pediatric patients, it is conceivable that actual soft failure rates may be even higher than those reported.
A suggested Soft Failure Assessment Checklist was developed by the 2005 consensus panel consisting of leaders in the implant field to assist clinicians in recognizing device soft failures (Appendix A) (2). Included in this checklist were a number of behavioral factors that were thought to be useful in identifying at-risk individuals: an increase in “bad” behaviors or aggressiveness (“externalizing” behaviors), and unwillingness to wear the device, head hitting, or inattentiveness (“internalizing” behaviors). It is reasonable to believe that these behaviors could serve as “red flags” for device malfunction. Unfortunately, there is little data to date to support the relationships between these behaviors and language development or device failure.
The purpose of this study was to evaluate whether the presence of externalizing or internalizing behaviors such as those listed above can be used as a warning sign of poor language development and should rouse suspicion of a device soft failure. Identification of these behaviors in a child with a CI would be a much simpler task for caregivers, teachers, and clinicians than identifying a general language decline or failure to progress in language development. The reported evaluation was accomplished by re-analysis of a subset of previously published data (7) to gauge the strength of relationship between suspect behaviors and language skill.
Materials and Methods
Participants
Participants were 80 children with cochlear implants who were part of an ongoing longitudinal study (7), and a full report of demographic factors can be found there. Table 1 presents group means (and SDs) for selected demographic measures. All children in the study had profound sensorineural hearing loss. All were identified before two years of age, and most before one year. Children received their CIs early, which for most meant at or before two years of age. Consequently, these children had considerable experience with their CIs. Forty-two of the children wore CIs bilaterally by the age of 48 months. Sixty-one percent of the children wore devices by Cochlear Corporation, 35% wore devices by Advanced Bionics, and 4% wore devices by Med-El. To be in the study, children were required to have been full-term newborns and have no medical problems other than hearing loss that could be expected to delay language acquisition. English was the only language spoken in the home, and parents were required to have normal hearing. Socio-economic status was indexed using a two-factor scale on which both the highest educational level and the occupational status of the primary income earner in the home is considered (8). These scores suggest that all children came from middle-class families, so had reasonably rich language environments in the home. All children had received intervention starting shortly after their hearing loss was identified at least once per week until the age of 36 months and then attended preschool programs designed for children with hearing loss for at least 16 hours per week after age 36 months. These programs emphasized spoken language. Data collected from these children between 18 and 48 months are reported.
Table 1.
Mean values for demographic measures.
| N | Mean (SD) | |
|---|---|---|
| Age of Identification (months) | 80 | 6.13 (7.13) |
| Age of First Amplification (months) | 80 | 7.80 (6.96) |
| Age of First Implant (months) | 80 | 17.09 (6.85) |
| Age of Second Implant (months) | 42 | 35.14 (10.75) |
| Better-Ear Pure Tone Average | 80 | 103.76 (13.83) |
Equipment
Each test session was video and audio recorded using a SONY DCR-TRV19 video recorder. Sessions were recorded so scoring could be done at a later time. Children wore SONY FM transmitters in specially designed vests that transmitted speech signals to the receivers, which provided direct line input to the hard drives of the cameras. This procedure ensured good sound quality for all recordings. All children were tested wearing their customary auditory prostheses (unilateral CI, bilateral CI, or unilateral CI with contralateral hearing aid), which were checked at the start of testing.
General Procedures
Testing took place at multiple sites across the United States as described by Nittrouer (7), with all examiners having undergone two 2-day training sessions prior to testing. Measures collected during those sessions and used in this report are described here. They include children’s auditory comprehension, expressive vocabulary, and unstructured language.
Task-Specific Procedures
Multiple measures of language, including auditory comprehension, vocabulary, and children’s use of unstructured language, were collected and were correlated with measures of internalizing and externalizing behaviors. Task-specific procedures will be described in brief in this report. See Nittrouer (7) for a more complete description of each task.
Auditory comprehension
The Auditory Comprehension subscale of the Preschool Language Scales-4 (PLS-4) was used to measure this ability (9). This test is a standardized assessment tool generally used for children between birth and 7 years of age. Most items require the child to demonstrate spoken language comprehension by performing specific commands provided by the examiner. Items on this test are organized hierarchically, as older children would be expected to understand more complex linguistic structure than younger children. The Auditory Comprehension subscale has excellent reliability to evaluate specific components of comprehension, including lexical, syntactic, and grammatical elements. Standard scores were used in analyses.
Expressive vocabulary
Two measures of expressive vocabulary were used for this study. For children between the ages of 18 months and 30 months, the Language Development Survey (LDS) was used (10). The LDS has been shown to correlate well with direct measures of expressive vocabulary. This test consists of 310 words, and parents filled out a survey by circling the words that their children say spontaneously and with consistent pronunciation. Number of words spoken was used in analysis.
For children between 36 and 48 months of age, the Expressive One-Word Picture Vocabulary Test (EOWPVT) was administered (11). For this test, the child is shown a picture and is asked to name the picture with the appropriate vocabulary word. Standard scores were used in analyses.
Unstructured language
At every test age, a 20-minute language sample was obtained and scored to provide a comprehensive measure of children’s productive language abilities. Four measures that were made in order to index the form and function of children’s early communication are reported here. First, the number of real-word utterances was calculated as the number of utterances consisting of real words in the entire 20-minute language sample. Second, the number of answers children produced in 20 minutes was obtained as a metric of how responsive children were in a meaningful way. Third, the total number of vocalizations produced during this 20-minute sample was calculated. Finally, the total number of imitations was calculated during the same sample.
Two measures of productive syntax obtained from the 20-minute language sample are also reported. Systematic Analysis of Language Transcripts, Version 9 (SALT) was used to obtain these metrics (12). First, mean length of utterance (MLU) was used, defined as the mean number of morphemes per utterance. The number of pronouns used correctly in those 50 utterances is the other measure reported.
Latent language measure
In order to more comprehensively evaluate language development, a composite measure of language was developed (7). This composite measure, referred to as a latent language measure, allowed the characterization of developmental trajectories of individual children. Details on calculating a composite language score using latent growth modeling are found in Nittrouer (7). In brief, an estimate of the effects of multiple predictor variables on a dependent language measure can be made. Eight measures were used in the construction of the latent language measure: auditory comprehension, expressive vocabulary, real-word utterances, answers, imitations, number of vocalizations, mean length of utterance, and the Leiter International Performance Scale-Revised (LIPS-R) classification score, which is a measure of cognitive development (14).
Internalizing and externalizing behaviors
To evaluate internalizing and externalizing behaviors, the Child Behavior Checklist (CBCL) (13) was used. The CBCL is appropriate for children between 18 months and 5 years of age and examines 100 separate behaviors. It is a questionnaire that is completed by parents, who respond to each item by circling a number from 0 to 2, indicating whether ascribing the behavior to their child would be not true (0), somewhat or sometimes true (1), or very true or often true (2). A weighted sum of responses is obtained, with results loaded on seven clusters: Emotionally Reactive (I), Anxious/Depressed (II), Somatic Complaints (III), Withdrawn (IV), Sleep Problems (V), Attention Problems (VI), and Aggressive Behavior (VII). The sum of scores on clusters I through IV serves as a general “internalizing index,” and the sum on clusters VI and VII serves as a general “externalizing index.” Raw scores were used in statistical analyses. Clinical significance on this measure is suspected when a raw score exceeds 18 for internalizing behaviors and 24 for externalizing behaviors.
Correlational Analyses
In order to examine the relationship of behaviors and language development, several series of correlation analyses were performed. In each analysis, Pearson product-moment (i.e., zero-order) correlation coefficients were computed between each behavioral variable (internalizing index and externalizing index) and each language measure (auditory comprehension, vocabulary, unstructured language measures, and latent language measure).
Results
The reported analyses examine the language and behavior measures of a group of pediatric CI users. For an in-depth analysis of the language outcomes of these patients as compared with children with normal hearing or hearing aids, the reader is encouraged to review the results of the ongoing longitudinal study (7). In brief, no significant differences were found between the internalizing or externalizing behavior index among the groups for CI users, hearing aid users, or children with normal hearing. On the other hand, CI users on the whole performed approximately one standard deviation below the mean for normal-hearing children on all language measures, including the latent language measure. For a sense of the range of scores for the CI users, mean scores (and standard deviations) at 48 months were the following: internalizing behavior index, 6.2 (SD, 5.5); externalizing behavior index, 9.3 (SD, 7.4); and latent language measure, 7.5 (SD, 1.9).
Correlating Behaviors with Language
The first analysis was performed to examine how well internalizing and externalizing behaviors correlated with language development at that same age. In other words, would the presence of internalizing and externalizing behaviors serve as a red flag that language was not developing appropriately? Figure 1 and Figure 2 display zero-order correlation coefficients between each behavior index, internalizing or externalizing, respectively, and the language measures at each age. By taking the square of the correlation coefficients, it is possible to index the amount of variance in language development that can be explained by each behavioral index. The dotted lines show the points above and below which more than 10 percent of the language measure is explained by the behavioral index measure, with 10 percent generally accepted as the amount of variance that is sufficient to be of interest. Below 10%, the behavioral index measure is unlikely to be worthwhile in explaining variance in language development. Examination of these coefficients reveals that neither internalizing nor externalizing behaviors consistently explained greater than 10% of the variance in the language measures, including the latent language measure.
Figure 1.
Zero-order correlation coefficients between internalizing behavior index and language measures at each age. The dotted lines show the points above and below which more than 10 percent of the variance in the language measure is explained by the behavioral index measure. LDS: Language Development Survey; EOWPVT: Expressive One-Word Picture Vocabulary Test
Figure 2.
Zero-order correlation coefficients between externalizing behavior index and language measures at each age. The dotted lines show the points above and below which more than 10 percent of the variance in the language measure is explained by the behavioral index measure. LDS: Language Development Survey; EOWPVT: Expressive One-Word Picture Vocabulary Test
Behaviors as Predictors of Later Language Development
The second analysis was performed to examine how well internalizing and externalizing behaviors at 36 months could predict language development at 48 months. This analysis examined whether the earlier presence of internalizing and externalizing behaviors might suggest an impending problem with language development, which would become evident at a later age. The older ages were chosen because they would best represent ages of more complex language and behavior. Mean length of utterance (MLU) was chosen as representative of unstructured language for this analysis because it is the most commonly evaluated measure of syntactic development. Figure 3 shows zero-order correlation coefficients calculated between each behavior index (internalizing or externalizing) at age 36 months and the variable of the latent language measure at age 48 months. No significant correlations were found (internalizing index r = .040, p = .83; externalizing index r = −.132, p = .48). Similar correlation coefficients were calculated between each behavior index at 36 months and each separate language measure (auditory comprehension, expressive vocabulary, and mean length of utterance) at 48 months, and no correlations were significant.
Figure 3.
Zero-order correlation coefficients between internalizing and externalizing behavior indices at age 36 months and language measures at age 48 months. The dotted lines show the points above and below which more than 10 percent of the variance in the language measure is explained by the behavioral index measure. Comprehension: Auditory Comprehension; Vocabulary: Expressive One-Word Picture Vocabulary Test score
Examining the Poorest Language Performers
The above findings suggested there was no significant relationship between the presence of internalizing or externalizing behaviors and poor language skills in the group of children with CIs as a whole. However, it was possible that the individual children with the poorest language skills would still show a larger number of these behaviors. Therefore, the ten children with the poorest latent language measure value at 48 months (all worse than one standard deviation below the mean for the group) were examined independently. As seen in Figure 4, wide variability in internalizing and externalizing behaviors was seen for these children with the poorest language skills. These findings suggest that even for the children with the poorest language skills, behavior scores varied greatly. In addition, none of these children had behavior indices reaching clinical significance for either internalizing or externalizing behaviors.
Figure 4.
Individual data from the ten CI users with the poorest latent language measure scores. Latent language, internalizing behavior index, and externalizing behavior index are plotted for each subject as number of standard deviations from the mean for the entire group of CI users. A negative number for the latent language measure signifies worse performance than the mean, while a positive number for internalizing or externalizing behavior signifies poorer performance.
Examining Children with Slowed Language Development
In addition to failure to progress appropriately in language development, device soft failure has been defined as deterioration of previous performance. Therefore, the developmental trajectory of each child was examined in order to identify children who showed deterioration in performance over time and to determine whether they showed an increase in internalizing or externalizing behaviors around the time of language deterioration. Individual latent language measure data were examined across ages for each child. No child showed a decline in latent language measure over time.
Next, individuals were identified who showed a slowed rate of growth in latent language measure score. For the group as a whole, the average increase in latent language measure score over each 6-month period was approximately 1.0. A “slowed” language growth rate was defined as an increase in latent language measure score of less than 0.5 over at least one 6-month period. Eight children met this criterion. Among these children, no consistent increase in internalizing or externalizing behaviors was seen around the time period of their slowed language growth. These findings support the idea that increased internalizing or externalizing behaviors are not suggestive of a period of slowed language growth.
Examining Children with Device Hard Failures
If an association exists between device failure and an increase in bad behaviors, this would likely be seen for children with device hard failures (a hardware-related failure identified by diagnostic testing of the device in vivo), as well as those with device soft failures. Performance of children in this study who had undergone reimplantation for device hard failures was examined individually to see if there was any increase in internalizing or externalizing behaviors around the time of their device failures. Four children met this criterion, and none of them showed an increase in internalizing or externalizing behaviors prior to reimplantation. Only one child showed a decrease in externalizing behaviors (from a score of 10 to a score of 1) after reimplantation. Although not conclusive based on the small number of subjects included, these findings imply that hard device failure is not associated with an increase in internalizing or externalizing behaviors.
Separate Language Measures as Predictors of Later Language Development
Because internalizing and externalizing behaviors at 36 months did not significantly predict language development at 48 months, a final analysis was performed to examine which separate early language measures at 36 months served as the strongest predictors of language development at age 48 months. Figure 5 shows zero-order correlation coefficients calculated between each language measure at 36 months and the dependent variable of the latent language measure at 48 months. Each independent language measure at age 36 months was a strong predictor of the latent language score at 48 months. Each language measure predicted about 50% or greater of the variance in latent language score, found by taking the square of the correlation coefficient, with auditory comprehension and vocabulary being the strongest predictors.
Figure 5.
Zero-order correlation coefficients between separate language measures at 36 months of age and the latent language measure at age 48 months. The dotted lines show the points above and below which more than 10 percent of the variance in the latent language measure is explained by each separate language measure. Comprehension: Auditory Comprehension; Vocabulary: Expressive One-Word Picture Vocabulary Test score
Discussion
In children, a diagnosis of CI soft failure is difficult to make but is often considered when a child shows declining language performance or a failure to make satisfactory gains in language skills. It has been suggested that the presence of “bad” behaviors in young children should alert the clinician of a possible soft failure, as described in the soft failures consensus statement (2). The purpose of the re-analysis reported here was to see if there was any evidence to support this relationship of “bad” behaviors with poor implant performance suggestive of device failure in a group of children between the ages of 18 and 48 months. The results revealed that internalizing and externalizing behaviors were neither good at explaining variance in language outcomes at each age nor good at predicting later language outcomes. The children with a period of slowed language growth did not have associated increases in internalizing or externalizing behaviors. Device hard failures were also not associated with an increase in bad behaviors, although only a few patients with hard failures were presented in this series. The findings of this study suggest that assessment of internalizing and externalizing behaviors in pediatric CI users is not a sensitive tool for identifying children who are at risk for a possible device soft failure. On the other hand, separate language measures, such as auditory comprehension, vocabulary, and unstructured language measures, were found to strongly predict the overall language development a year later. These findings reinforce the conclusions of previous work, showing that to best predict the language skills of children with CIs, individual measures of language are essential (15–16). Because language performance is likely one of the strongest indicators of device integrity in the pediatric population, language development should be thoroughly and carefully monitored.
As intuitive as it seems, the findings of this study suggest that in order to identify a failure to progress in language development by children with cochlear implants, such as may be seen with a device soft failure, clinicians need to focus on directly evaluating aspects of language development. Tests of auditory comprehension, vocabulary, and language production should be developed for use by clinicians to better assess overall language development of children with implants and to identify those who are failing to progress appropriately.
Acknowledgments
Disclosure of funding: This work was supported by Grant No. R01 DC006237 to Susan Nittrouer from the National Institute on Deafness and Other Communication Disorders, the National Institutes of Health.
Appendix A
A suggested checklist and assessment guideline for clinicians that the panelists felt could be useful in evaluating patients with suspected device malfunction. Reprinted from “Cochlear Implant Soft Failures Consensus Development Conference Statement,” by T. Balkany et al., 2005, Otology & Neurotology. 26, p. 817. Copyright 2005 by Lippincott Williams & Wilkins. Reprinted with permission.
| Suggested Checklist for Assessment of Soft Failures | ||
|---|---|---|
| ADULT/OLDER CHILDREN | YOUNG CHILDREN | |
| AUDITORY | MAPPING | BEHAVIORAL |
| □ atypical tinnitus | □ Changes in levels over time | □ Increase in "bad" behaviors |
| □ Buzzing | □ Changes in pulse wdth/dur | □ Aggressiveness |
| □ Roaring | □ Loss of Channels | □ Unwilling to wear device |
| □ Engine-like | □ Changes in impedance | □ Head hitting |
| □ Static | □ Shorts/open circuits | □ Inattentiveness |
| □ Popping | □ Regression in language/speech | |
| □ Other | ||
| TEACHER/THERAPIST CONCERNS | ||
| NONAUDITORY | HARDWARE | □ Intermittent Responsiveness |
| □ Pain over implant site | □ Replacement of all externals | □ Frequent appearance of being "off-task" |
| □ Pain down neck | □ Deterioration in grades/school performance | |
| □ Shocking | □ Plateau in performance | |
| □ Burning | □ Fails to meet appropriate expectations | |
| □ Itching | ||
| □ Facial stim | ||
| OTHER FACTORS | ||
| PERFORMANCE | OBJECTIVE ASSESSMENT | □ Educational placement |
| □ Sudden drop | □ Surface potential testing | □ Type and amount of therapy |
| □ Decrement over time | □ Neural response measures | □ Family involvement |
| □ Fails to meet expected performance | □ Stimulus artifact | □ Puberty |
| □ Evoked potentials | In addition, the adult checklist should be applied to a child whenever possible. | |
| □ Intermittent performance | ||
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Chung D, Kim AH, Parisier S, et al. Revision cochlear implant surgery in patients with suspected soft failures. Otol Neurotol. 2010;31:1194–1198. doi: 10.1097/MAO.0b013e3181f0c631. [DOI] [PubMed] [Google Scholar]
- 2.Balkany TJ, Hodges AV, Buchman CA, et al. Cochlear implant soft failures consensus development conference statement. Otol Neurotol. 2005;26:815–818. doi: 10.1097/01.mao.0000178150.44505.52. [DOI] [PubMed] [Google Scholar]
- 3.Peterson NR, Pisoni DB, Miyamoto RT. Cochlear implants and spoken language processing abilities: Review and assessment of the literature. Restor Neurol Neurosci. 2010;28:237–250. doi: 10.3233/RNN-2010-0535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Marlowe AL, Chinnici JE, Rivas A, et al. Revision cochlear implant surgery in children: The Johns Hopkins experience. Otol Neurotol. 2009;31:74–82. doi: 10.1097/MAO.0b013e3181c29fad. [DOI] [PubMed] [Google Scholar]
- 5.Brown KD, Connell SS, Balkany TJ, et al. Incidence and indications for revision cochlear implant surgery in adults and children. Laryngoscope. 2009;119:152–157. doi: 10.1002/lary.20012. [DOI] [PubMed] [Google Scholar]
- 6.Cullen RD, Fayad JN, Luxford WM, et al. Revision cochlear implant surgery in children. Otol Neurotol. 2008;29:214–220. doi: 10.1097/MAO.0b013e3181635e9a. [DOI] [PubMed] [Google Scholar]
- 7.Nittrouer S. Early Development of Children with Hearing Loss. San Diego: Plural Publishing; 2010. [Google Scholar]
- 8.Nittrouer S, Burton LT. The role of early language experience in the development of speech perception and phonological processing abilities: Evidence from 5-year-olds with histories of otitis media with effusion and low socioeconomic status. J Commun Disord. 2005;38:29–63. doi: 10.1016/j.jcomdis.2004.03.006. [DOI] [PubMed] [Google Scholar]
- 9.Zimmerman IL, Steiner VG, Pond RE. Preschool Language Scale. 4th ed. San Antonio: The Psychological Corporation; 2002. [Google Scholar]
- 10.Achenbach TM, Rescorla LA. Manual for the ASEBA Preschool Forms and Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families; 2000. [Google Scholar]
- 11.Brownell R. Expressive One-Word Picture Vocabulary Test. 3rd ed. Novato: Academic Therapy; 2000. [Google Scholar]
- 12.Miller J, Chapman R. Systematic Analysis of Language Transcripts (SALT): Version 9. Madison: University of Wisconsin-Madison, Language Analysis Laboratory; 2006. [Google Scholar]
- 13.Achenbach TM, Rescorla LA. Manual for the ASEBA preschool forms and profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families; 2000. [Google Scholar]
- 14.Roid GH, Miller LJ. Leiter International Performance Scale-Revised (LIPS-R) Wood Dale, IL: Stoelting; 2002. [Google Scholar]
- 15.Nittrouer S, Caldwell A, Holloman C. Measuring what matters: Effectively predicting language and literacy in children with cochlear implants. Int J Ped Otorhinolaryngol. 2012;76:1148–1158. doi: 10.1016/j.ijporl.2012.04.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Nittrouer S, Chapman C. The effects of bilateral electric and bimodal electric-acoustic stimulation on language development. Trends In Amplif. 2009;13:190–205. doi: 10.1177/1084713809346160. [DOI] [PMC free article] [PubMed] [Google Scholar]