Intelligibility in Context Scale: Sensitivity and specificity in the Jamaican context

Abstract

The present study investigated the sensitivity and specificity of the English Intelligibility in Context Scale (ICS) and the ICS-Jamaican Creole (ICS-JC) translation with bilingual preschool-aged Jamaican children. Participants in this study were 262 English-Jamaican Creole simultaneous bilingual children (aged 3;3 to 6;3, M=4;11, SD=7.8). The ICS and ICS-JC were administered to parents in auditory form, rather than written form. Although recent evidence has demonstrated the validity and reliability of the ICS as an assessment tool in various languages, further data are needed to determine diagnostic accuracy of the ICS and ICS-JC in Jamaican children. The sensitivity and specificity of both tools were high in this cohort of children, indicating that in the Jamaican context, these versions of the ICS could be used as screening tools to identify children who require further assessment of speech sound disorders. A cut-off score of 4.12 was used for both tools to achieve high sensitivity (0.84) and specificity (0.70) values for the ICS, as well as high sensitivity (0.84) and specificity (0.71) for the ICS-JC. The results of this study also demonstrate that administration of the auditory ICS is a valid way of collecting parent reports about children’s speech intelligibility, which has implications for use of the ICS in languages with no written form or with parents who have a low level of literacy in the languages they use. This investigation is relevant not only to this underserved population but broadens knowledge of research-based tools for working with bilingual children.

Introduction

Reduced speech intelligibility in children, a known consequence of developmental speech sound disorders (SSD), presents a significant barrier to children’s academic and social participation. The negative consequences of SSD are well-documented and include communication breakdowns and misunderstandings in the short-term (McCormack et al., 2010), as well as lifelong impacts on an individual’s educational and occupational outcomes (McCormack et al., 2009; Ruben, 2000). Diagnosis of SSD is more challenging when working with children from multilingual backgrounds. These challenges relate to (a) a lack of evidence-based assessments that are available to assess the languages a child uses, (b) a lack of evidence-based assessments for assessing bilingual children, and (c) developmentally normal interaction between the child’s languages that differ from monolingual children (Fabiano-Smith & Goldstein, 2010; Goldstein & Gildersleeve-Neumann, 2015; Paradis et al., 2011; Washington et al., 2019). The consequences of these assessment challenges can be both over- and under-diagnosis of SSD in bilingual children. Over-diagnosis results from determining that a child has a disorder (i.e., significant discrepancies in language skills across all the child’s languages compared to the expected benchmark), when in fact they may be better characterized as having a difference (i.e., rule-governed and typical within their culture). Under-diagnosis results from determining that a bilingual child demonstrates a difference when in fact a disorder is present. As the proportion of speech-language pathologists (SLPs) who work with bilingual children is increasing steadily, the issues the SLPs face in providing culturally and linguistically appropriate assessment and diagnosis are also increasing. While culturally and linguistic minority groups in the US have increased over 30% from 2000–2015 (American Speech-Language-Hearing Association [ASHA], 2013; 2019), tools for diagnosing SSD in bilingual speakers have not kept pace with this growing need (McLeod et al., 2017; Skahan et al., 2007).

The Intelligibility in Context Scale (ICS; McLeod et al., 2012a) is a validated tool for assessing speech intelligibility and screening for SSD that has been translated into more than 60 languages (https://www.csu.edu.au/research/multilingual-speech/ics). The ICS utilizes the bio-psycho-social model of health of the International Classification of Functioning, Disability, and Health - Children and Youth Version (ICF-CY; World Health Organization [WHO], 2007). The ICF-CY provides a framework for defining and establishing children’s functionality when a disability is present by acknowledging the factors (Body Structures and Functions, Activities, Participation, Environmental, Personal Factors) that can contribute to overall functioning. The ICS provides a method of assessing children’s functionality using these guidelines. Although factors from Body Structures and Functions, Activities, and Participation can all influence intelligibility, the ICS was developed with a focus on Environmental Factors (McLeod et al., 2012a; WHO, 2007). It is a seven-item assessment that a child’s parents complete to report on functional intelligibility across seven different communication partners with varied familiarity and authority and in varied contexts. Being understood by different listeners and in different settings can affect communication success, which is critical for social engagement and participation (McLeod & Threats, 2008). For example, if a child’s speech is not well understood by their peers, their invitations to participate in play may be reduced, while if a child’s speech is not well understood by their teacher, engagement in learning activities may decrease.

In contrast to most measures of children’s speech, which only consider the SLP’s judgment from an initial assessment in a clinical context (Berisha et al., 2013), the ICS gathers information from a parent’s perspective and incorporates many contexts of a child’s life. In completing the ICS, parents use a five-point scale (1=never, 5=always) to rate how they perceive their child’s speech is understood by speakers in seven different contexts: parents, immediate family members, extended family members, acquaintances, teachers, and strangers. Questions are phrased simply, such as “Do you understand your child?” and “Do your child’s teachers understand your child?”. While the inclusion of parents’ perspectives in assessment is important for all areas of practice, it is especially important in gathering assessment information describing performance in language/s that the child speaks but the SLP does not (Restrepo, 1998).

Previous studies of the ICS have provided evidence of its utility for assessing speech intelligibility across a range of languages and cultural contexts. Initially validated with 120 Australian English-speaking children, the ICS demonstrated good sensitivity and, construct validity, and excellent internal consistency (α=0.93). Criterion validity was also established using three objective severity measures: percentage of phonemes correct (PPC), percentage of consonants correct (PCC), and percentage of vowels correct (PVC) (McLeod et al., 2012a). Further studies investigating the ICS have found it to have good psychometric properties such as high discriminant accuracy, internal consistency, and test-retest reliability in languages such as English (McLeod et al., 2015), Cantonese (n=72; Ng et al., 2014), German (n=181; Neumann et al., 2017), Vietnamese (n=181; Phạm et al., 2017), Italian (n=364; Piazzalunga et al., 2020), and European Portuguese (n=76; Lousada et al., 2019). There is also emerging evidence for use of the ICS with bilingual populations (McLeod, 2020), such as, Fiji English, Fiji Hindi, and Standard Fijian (n=65; Hopf et al., 2017), English and Korean (n=33; Kim et al., 2016), and in English-speaking children who use a different language at home (n=31; McLeod et al., 2015).

The Jamaican Context

English and Jamaican Creole (JC) is an under-studied language pairing in the area of research of children’s speech and language development. JC is a creole language originally formed from West African languages and English but is now a language that is considered to be independent of its heritage languages (Cassidy, 1966; Washington et al., 2017). One main characteristic of the Jamaican language context is that most Jamaicans are described as simultaneous bilinguals as they are exposed to English and JC from birth or shortly after and code-switching is not only present, but expected (Paradis et al, 2011; Washington et al., 2019). In Jamaica, English is used in formal settings, such as schools, and is used in spoken and written forms. JC is primarily an oral language and is commonly used with friends, family, or in situations that are perceived as informal. Standardization of written JC occurred in 2002, meaning that this language has a primarily oral tradition with few people using its written form (Jamaican Language Unit, 2009; Washington, 2012; Washington et al., 2017). English and JC is a language pairing that is also increasingly common outside of Jamaica. There were more than 700,000 Jamaican-born individuals in the United States in 2017, and Jamaicans represent approximately 20% of the Caribbean-born population living in the United States (U.S. Census Bureau, 2017). With this growth, the possibility of SLPs having English-JC bilinguals on their caseloads increases, making it important to understand the features of the typical language profiles of these children for diagnostic and treatment purposes.

The ICS was translated into JC by Washington and Devonish in 2014 (ICS-JC; McLeod et al., 2012b) and later validated (Washington et al., 2017). This validation study examined the ICS scores of 145 typically developing English-JC speaking bilinguals in Jamaica. This study demonstrated that the ICS and ICS-JC were valid and reliable for assessing speech intelligibility in English and JC, respectively, showing high internal consistency, inter-rater reliability, and test-retest reliability. This study was the first to use an auditory administration of the ICS and ICS-JC to accommodate for parents’ having low levels of literacy in JC, due to the brief history of its orthographic form. While validation of the ICS and ICS-JC for typically developing Jamaican children has been successfully completed, evaluation of the accuracy of these tools in differentiating typical from disordered development is needed.

The Current Study

In this study we sought to investigate the ability of parent-reported speech intelligibility scores on the ICS and ICS-JC to accurately distinguish typical development from suspected SSD in bilingual English-JC speaking children. This study extends on previous research (Washington et al., 2017) in three ways. First, we investigate the appropriateness of the ICS and ICS-JC in the Jamaican context. Second, this study considers a larger sample size of bilingual participants than any previous study using the ICS. Third, this study is the first to examine bilingual children who are known to be typically developing as well as children with suspected SSD using the ICS. We hypothesize the tools will demonstrate high discriminant accuracy as evidenced by fair-to-good sensitivity and specificity rates (80%−90%). The following research questions were addressed:

1. Do the ICS and ICS-JC demonstrate satisfactory sensitivity and specificity levels?

2. Do the ICS and ICS-JC demonstrate satisfactory internal consistency?

3. Do the ICS and ICS-JC demonstrate criterion validity with objective measures of speech production?

Materials and Methods

Study approval

Ethical approval for this study was obtained from the Institutional Review Board, University of Cincinnati. Support and permission were also obtained from the Early Childhood Commission, Government of Jamaica, and each participating early learning centre (n=5). Licensure for the practice of speech therapy in Jamaica was obtained from the Council for Professions Supplementary to Medicine. Parental consent and child assent were also obtained.

Data sources

This study included data from four sources: preschool children (speech production data using audio and video recordings), children’s parents (ICS, ICS-JC, and report of speech development concerns), and children’s teachers and SLPs (report of speech development concerns). Data relating to 303 children were collected between 2013 and 2019, with the ICS-JC being available starting in 2014. Due to corrupt and/or missing files, complete data were available for 262 of these children and their data served as the corpus for the present analysis.

Participants

Children

Data from 262 children were included in this analysis. Participants were located in Kingston, Jamaica (n=252) and New York City, United States (n=10). Preschoolers were all English-JC simultaneous bilingual children aged 3;3 to 6;3 (M=4;11, SD=7.8), consisting of 148 females (56.5%) and 114 males (43.5%). Participants came from dual-income (n=176, 67.2%), single-income (n=72, 27.5%), and unemployed (n=4, 1.5%) family homes, with ten families not reporting their status (3.8%). All participants met the following inclusion criteria: (a) parents and teachers reported that all participants used English and JC at home and at preschool; (b) binaural hearing at 25dB for 1kHz, 2kHz, and 4kHz; (c) parent-reported absence of neurological deficits or pervasive developmental disorders; (d) achieved age-based criterion on the Oral Motor subtest of the Diagnostic Evaluation of Articulation and Phonology (DEAP; Dodd et al., 2006); and (e) standard scores of ≥72 (M=105.03, SD=16.70, range=72–149) on the Primary Test of Nonverbal Intelligence (PTONI; Ehrler & McGhee, 2008).

Children’s Parents

Most caregivers who reported on their children’s speech were parents (n=238, 90.9%). Reports were provided by mothers (n=199, 76%), fathers (n=39, 14.9%), and other family members (e.g., grandmothers and aunts; n=9, 3.5%). Caregiver relationship was left unanswered for fifteen participants (5.7%). All caregivers are referred to as parents in this study.

Children’s teachers and SLPs

The preschool teachers completed a scaled questionnaire describing any speech and language concerns for each child participating in the study. Eighteen teachers from 18 different classrooms across five preschools filled out the questionnaire every year. It should be noted that the same teachers participated each year for each new cohort of participants. Two research SLPs (first and second authors), who are experienced with bilingual children and SSD, categorized each child as concerning for suspected SSD or no concern. These classifications were based from the child’s direct speech productions in English and JC, using the DEAP.

Instruments

ICS and ICS-JC

Parents were asked to complete the ICS and the ICS-JC forms in separate sessions (McLeod et al., 2012b; Washington et al., 2017). Parents used a five-point scale (1=never, 2=rarely, 3=sometimes, 4=usually, 5=always) to rate how their children’s speech was understood by seven communication partners (parents, immediate and extended family members, child’s friends, acquaintances, child’s teachers, and strangers) in the last month. Both ICS and ICS-JC questions were presented via audio-recordings on laptop computers (e.g., “Do you understand your child?”/“Yu andastan wen yu pikni chat?”). Parents listened to the instructions and questions for each language and provided their responses using a paper and pen form written in English (ICS) or JC (ICS-JC).

Diagnostic Evaluation of Articulation and Phonology (DEAP)

Recordings of participants’ single-word speech productions were made using the Articulation (30 words) and Phonology (50 words) subtests of the DEAP (Dodd et al., 2006). Although not normed for this population, these subtests provided a means of consistently eliciting word-level speech samples in each language. The 80 stimulus words were phonetically transcribed and PPC, PCC, and PVC were calculated for each child in English and JC. Guidelines for scoring the DEAP based on adult-productions of English and JC were used to score children’s productions (Washington et al., 2017). Furthermore, the researcher SLPs used the single-word productions in both languages to classify the children as possible SSD or typically developing.

Questionnaires

Demographic information regarding parent education, family socioeconomic status, and language use were gathered through a questionnaire completed by children’s parents. Teachers answered questions about their concerns for each child’s communication development. Their response to the question, “Has there been any concerns regarding this child’s communication (e.g., talking and sound production)?” was recorded and used for group classification.

Procedure

Sample and group classification

In the absence of well-validated criteria on how to identify SSD in bilingual preschoolers, participants were identified as typical or disordered based on consensus of concerns about speech intelligibility (Restrepo, 1998) from (a) parent and (b) teacher or SLP. The first and second authors provided the SLP response after reviewing DEAP assessments for the children in both languages. To be placed in the ‘disordered’ group, a “yes” response to the question, “Are you concerned about this child’s talking?” (adapted from the Parent Evaluation of Developmental Status; Glascoe, 2000), must have been obtained from both sources and in both languages. This approach, which includes responses from key respondents (i.e., parent AND teacher OR SLP), is considered the gold-standard for use with bilingual populations to establish typical versus suspected SSD diagnosis. Parental and teacher concerns about each preschooler’s speech development were gathered through the screening questionnaires. The criterion, using two sources of concern, was applied to the sample, resulting in the identification of 25 preschoolers as presenting concerns for SSD in both languages.

Data Collection

Direct measures of speech

Parent consent and child verbal assent were collected prior to any data collection. Preschoolers completed sessions for the collection of demographic information (e.g., the PTONI, hearing screening, DEAP Oral Motor) that lasted for 50-minutes and for speech sound production data in each language, which lasted for about 50-minutes (25-minutes in each language). The participants completed the elicited picture-based task for speech sound production using the DEAP (Dodd et al., 2006) in English and in JC at their preschools. To support the children in distinguishing and responding in English and in JC, two trained SLPs administered the DEAP; one assessed the child in English, and the other in JC. From the sample, 262 children completed the DEAP in English, and 215 children completed the DEAP in JC. The ICS-JC was not available until after 2014, which is why there is less data in JC than English. These data were transcribed, scored, and analysed for preschoolers’ speech sound productions at the single-word level in both languages.

Indirect measures of speech

Parents completed a questionnaire, along with the ICS and ICS-JC, which took approximately 40 minutes to complete. The collected ICS (n = 262) and ICS-JC (n = 215) responses were scored using a previously established criteria for this study (Washington et al., 2017). This difference in numbers is due to the ICS-JC not being available at the beginning of the study.

Data Analysis

The IBM Statistical Package for the Social Sciences Version 23 (SPSS) was used to analyse all data. To answer the first research question (RQ), sensitivity and specificity of the ICS and ICS-JC were evaluated using Receiver Operating Characteristics (ROC). The area under the ROC curve (AUC) provided overall measurement accuracy of the screeners (Obuchowski, 2000). ROC curve analysis is a commonly used and supported method when examining the discriminant accuracy of diagnostic tools, especially in cross-sectional designs (Liu et al., 2005; Mandrekar, 2010). Accurate classification indicates clinically useful tools in identifying speech disorders. To address the remaining research questions, internal consistency was analysed using Cronbach’s alpha (RQ 2) and correlation between ICS and ICS-JC items was analysed using Spearman’s rho (RQ 3). Pearson correlation analyses and bivariate correlations were completed to establish criterion validity (RQ 3).

Results

Descriptive statistics

To facilitate comparisons with other ICS studies, descriptive statistics for each language are included separately. The mean scores for the ICS (M = 4.48) and ICS-JC (M = 4.48) showed that parents’ ratings differed by communication partners; with ratings highest for parents and lowest for strangers in both languages. These findings are consistent with other ICS studies. Table 1 presents parents’ ratings on each of the seven items for English and JC, with slight differences in the ranking of items in each language. For the ICS, parents were rated the highest, followed by teachers, and then immediate family. However, for the ICS-JC, ratings were highest for parents, then immediate family, followed by teachers. Overall, the results reveal that conversational listeners play a role in the intelligibility of a speaker.

Table 1.

Parent ratings for the ICS and ICS-JC

	Ratings Items	M	SD	Always (5) %	Usually (4) %	Sometimes (3) %	Rarely (2) %	Never (1) %
ICS	1. Parents	4.69	(0.57)	74.8	20.2	4.6	0.4	0.0
	2. Immediate family	4.59	(0.65)	67.4	23.8	8.8	0.0	0.0
	3. Extended family	4.37	(0.79)	56.1	25.2	18.3	0.4	0.0
	4. Friends	4.54	(0.71)	64.9	26.0	8.0	0.8	0.4
	5. Acquaintances	4.34	(0.81)	54.2	26.7	17.9	1.1	0.0
	6. Teachers	4.62	(0.65)	68.7	26.3	4.2	0.4	0.4
	7. Strangers	4.19	(0.91)	47.3	29.0	19.5	3.8	0.4
	Total	31.33	(4.29)	-	-	-	-	-
	Mean	4.48	(0.65)	-	-	-	-	-
ICS-JC	1. Parents	4.73	(0.54)	77.2	19.1	3.3	0.5	0.0
	2. Immediate family	4.62	(0.61)	69.3	23.7	7.0	0.0	0.0
	3. Extended family	4.37	(0.77)	54.4	28.4	16.7	0.5	0.0
	4. Friends	4.52	(0.69)	61.9	29.3	8.4	0.5	0.0
	5. Acquaintances	4.33	(0.80)	53.5	27.9	17.7	0.9	0.0
	6. Teachers	4.60	(0.62)	67.0	27.4	4.7	0.9	0.0
	7. Strangers	4.13	(0.94)	45.1	29.3	20.5	4.2	0.9
	Total	31.31	(4.09)	-	-	-	-	-
	Mean	4.48	(0.63)	-	-	-	-	-

Note. ICS=Intelligibility in Context Scale (McLeod et al., 2012a); ICS-JC=Intelligibility in Context Scale – Jamaican Creole (McLeod et al., 2012b).

English

Parents were more likely to report that they “always” understood their child compared to all other groups. The mean ICS scores for the 262 children differed by communication partners. Ordered from most to least often understood were: parents (M = 4.69), teachers (M = 4.62), immediate family (M = 4.59), the child’s friends (M = 4.54), extended family (M = 4.37), acquaintances (M = 4.34), and strangers (M = 4.19) (see Table 2). Mean scores for each item were compared using an ANOVA with repeated measures. Mauchly’s test indicated that the assumption of sphericity had been violated (χ²(20) = 227.39, p<0.001); therefore, degrees of freedom were corrected using Huynh-Feldt estimates of sphericity (ε = 0.79). Statistically significant differences were revealed, showing that communication partner had an effect on parent ICS ratings, F(4.731, 255) = 43.188, p= 0.001.

Table 2.

Participant Descriptions and Performance (ages shown in years; months).

	ICS				ICS-JC
	Typical (n=237)	SSD (n=25)	Total Sample (n=262)		Typical (n=190)	SSD (n=25)	Total Sample (n=215)
Age range	3;3–6;3	3;7–6;0	3;3–6;3		3;4–6;2	3;7–6;0	3;4–6;2
Female (%)	134 (57%)	14 (56%)	148 (56%)		111 (58%)	14 (56%)	125 (58%)
NVIQ M(SD)	105.24 (16.54) Range: 72–149	103.12 (18.39) Range: 73–145	105.03 (16.70) Range: 72–149		106.64 (16.81) Range: 74–149	103.12 (18.39) Range: 73–145	106.23 (17.00) Range: 73–149
ICS M(SD)	4.57 (0.58) Range: 3–5	3.66 (0.71) Range: 2–5	4.48 (0.65) Range: 2–5		4.59 (0.54) Range: 3–5	3.71 (0.70) Range: 2–5	4.48 (0.63) Range: 2–5
PCC M(SD)	97.73 (3.10) Range: 81.99–100	91.39 (9.74) Range: 64.88–100	97.12 (4.57) Range: 64.88–100		97.72 (2.60) Range: 85.24–100	88.17 (11.98) Range: 62.50–100	96.61 (5.61) Range: 62.50–100
PVC M(SD)	99.31 (0.93) Range: 95.65–100	98.04 (2.74) Range: 88.98–100	99.19 (1.28) Range: 88.98–100		99.44 (0.85) Range: 95.69–100	97.65 (2.51) Range: 90.98–100	99.23 (1.30) Range: 90.98–100
PPC M(SD)	98.30 (2.12) Range: 88.15–100	93.83 (7.03) Range: 73.68–100	97.87 (3.21) Range: 73.68–100		98.35 (1.80) Range: 89.88–100	98.35 (8.26) Range: 73.03–100	97.57 (3.88) Range: 73.03–100

Note. Standard scores are reported for NVIQ (Nonverbal Intelligence) as measured by the Primary Test of Nonverbal Intelligence. ICS=Intelligibility in Context Scale (McLeod et al., 2012a); ICS-JC=Intelligibility in Context Scale – Jamaican Creole (McLeod et al., 2012b; Washington et al., 2017). PCC=Percentage of Consonants Correct; PVC=Percentage of Vowels Correct; PPC=Percentage of Phonemes Correct. The classification of groups was determined using the respondent method recommended (Restrepo, 1998).

Jamaican Creole

Like the English ICS, Table 2 shows that parents were more likely to report that they “always” understood their child on the ICS-JC. The mean scores for the 215 children whose parents completed the ICS-JC questionnaire were highest for themselves (M = 4.73), followed by immediate family (M = 4.62), the child’s teacher (M = 4.60), the child’s friends (M = 4.52), extended family (M = 4.37), acquaintances (M = 4.34), and then strangers (M = 4.13). Differences among responses were analysed using an ANOVA with repeated measures. Mauchly’s test indicated that the assumption of sphericity had been violated (χ²(20) = 233.37, p < 0.001); therefore, degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity (ε=0.71). Statistically significant differences were observed, indicating that communication partners had an effect on parent ICS-JC ratings, F(4.244, 209) = 44.064, p = 0.001.

English and Jamaican Creole

Spearman’s rho was conducted to provide bivariate nonparametric correlation analyses between items in the ICS and for the items in ICS-JC. There were moderate to strong correlations between items of the ICS (r = .52-.82, ps < 0.001) and between items of the ICS-JC (r = .46-.78, ps < 0.001).

ICS and ICS-JC sensitivity and specificity levels

Participants were classified as typically developing or disordered based on Restrepo’s (1998) recommendations. ICS scores were significantly lower for the disordered group (M = 3.66, SD = 0.71) compared with the typically developing group (M = 4.57, SD = 0.58), t(260) = 6.17, d = 0.89. Similarly, ICS-JC scores were significantly lower for the disordered group (M = 3.71, SD=0.70) compared with the typically developing group (M = 4.59, SD = 0.54), t(213) = 6.05, d = 1.01. These differences in group mean scores indicate the ICS and ICS-JC scores can both be used to distinguish children with and without SSD in their respective languages. Group performance scores are provided in Table 2.

In order to conduct sensitivity and specificity analyses of the ICS and ICS-JC, ROC curve analysis was conducted and complemented with AUC under the ROC values. Sensitivity levels indicate that the screeners accurately identify children who may present with SSD, whereas specificity indicates the accuracy of the screeners to identify children who do not present with SSD. Using Euclidean distance, optimal threshold ROC values for ICS sensitivity (0.84) and specificity (0.70) and for ICS-JC sensitivity (0.84) and specificity (0.71) were identified (see Figures 1 and 2). The corresponding cut-off ICS score for the sensitivity and specificity pair was 4.12 for both ICS and ICS-JC, meaning that a mean score of 4.12 can be used to accurately distinguish preschoolers who are typically developing from those with SSD.

Figure 1:

Receiver Operating Characteristic (ROC). ICS cut-off score: 4.12 (sensitivity: 0.84, specificity: 0.70).

Figure 2:

Receiver Operating Characteristic (ROC). ICS-JC cut-off score: 4.12 (sensitivity: 0.84, specificity: 0.71)

To further test the appropriateness of this score for accurate classification of preschoolers as typically developing or disordered, AUC values and related Cohen’s d values were calculated. AUC values provide a summary of the ROC curve and the overall accuracy of the diagnostic tools with ≤ 0.5 indicating no discrimination ability, 0.7–0.8 indicating acceptable ability, 0.8–0.9 indicating excellent ability, and ≥ 0.9 considered outstanding (Mandrekar, 2010). Cohen’s d scores facilitate effect size comparison across studies (Rice & Harris, 2005; Salgado, 2018; Schuele & Justice, 2006) with a value ≥ 0.8.

ICS and ICS-JC internal consistency

High internal consistency indicates correlation among the items and their reliability in measuring speech intelligibility. Cronbach’s alpha was calculated to measure internal consistency of the items in the ICS (n = 262, α = 0.92) and ICS-JC (n = 215, α = 0.92). Alpha values of 0.7 to 0.8 are considered satisfactory; however, 0.90 to 0.95 are preferable in clinical use (Martin, & Altman, 1997). Both ICS and ICS-JC scores indicated high internal consistency in parents’ responses to the seven items of the scale.

ICS and ICS-JC criterion validity

The third aim of this study was to demonstrate the criterion validity of the ICS and ICS-JC through correlations with three objective measures of speech sound production: PPC, PCC, and PVC. The strength of the relationship between ICS and ICS-JC scores and PPC, PCC, and PVC scores for each language describes the extent that the ICS and ICS-JC can accurately measure speech intelligibility. Descriptive statistics for English (n = 262) and JC (n = 215) PPC, PCC, and PVC are presented in Table 2.

Objective Assessment of Speech Sound Production (PPC, PCC, PVC)

Participants’ speech production showed high accuracy levels in both languages. Out of a possible score of 100, over 50% of participants scored ≥ 98.8 for PPC, ≥ 98.0 for PCC, and 100 for PVC. Overall, the scores for PPC, PCC, and PVC were all high and showed low variability with a negatively skewed distribution, likely due to ceiling effects (skewness statistics for English = PPC: −3.63, PCC: −3.48, PVC: −3.44; skewness statistics for JC = PPC: −3.72, PCC: −3.69, PVC: −3.23). This observation is in line with the Washington et al. 2017 publication that described JC-English speaking preschoolers’ speech sound productions.

Mean scores for PPC, PCC, and PVC varied based on participants’ categorization as typically developing or suspected SSD for both English and JC. Comparison of typically developing and SSD groups in English showed that on all measures the disordered group scored significantly less on speech production measures: PPC t(260) = 3.12, d = 0.45; PCC t(260) = 3.23, d = 0.89; PVC t(260) = 2.28, d = 0.62. The disordered group also scored significantly lower on the speech production measures in JC: PPC t(213) = 4.01, d = 1.32; PCC t(213) = 3.97, d = 1.31; PVC t(213) = 3.55, d = 0.99. Descriptive statistics for each group are presented in Table 2. It is important to note that while severity measures (PCC, PVC, PPC) were high in the suspected SSD group, this was expected given the age range of the children. This finding is supported by earlier work (suspected SSD PPC M = 89.4, SD = 8.53; Ng et al., 2014).

Criterion validity was examined to identify the predictive ability of the ICS and ICS-JC to identify SSD. Bivariate correlation analyses (using Pearson correlation analyses) showed that the ICS in English was significantly and positively correlated with PPC (r = .44, p < 0.001), PCC (r = .44, p < 0.001), and PVC (r = .30, p < 0.001). The ICS-JC was also significantly and positively correlated with PPC (r = .45, p < 0.001), PCC (r = .46, p < 0.001), and PVC (r = .29, p < 0.001). These moderate (r = .3-.5) to low (r = .1-.3) associations suggest that severity of speech difficulties in both English and JC are linked with parental ratings on the ICS and ICS-JC (Cohen, 1988).

Discussion

The number of bilingual children on SLPs’ caseload is increasing, yet there is a shortage of valid diagnostic tools for these populations (McLeod et al., 2017; Skahan et al., 2007), especially tools that can be used by an SLP who does not use the same language as their client. In the present study we addressed these concerns by analysing the effectiveness of screening tools (ICS and ICS-JC) in identifying suspected SSD in bilingual English and JC-speaking preschoolers when these screening tools were administered to parents using an auditory format. A total of 262 ICS and 215 ICS-JC questionnaires were completed for preschool children who were classified as typically developing or with suspected SSD. This is an important contribution since earlier published works for this linguistic group only included typical populations (Washington et al., 2017; 2019). Our findings demonstrate promising evidence for the clinical utility of the ICS and ICS-JC with English-JC bilingual preschoolers.

To answer our first research question, we determined that the ICS and ICS-JC were able to accurately distinguish typical from disordered communication for the Jamaican population (ICS sensitivity=0.84 and specificity=0.70; ICS-JC sensitivity=0.84 and specificity=0.71). Satisfactory levels were expected and are in line with previous studies (Lousada et al., 2019; McLeod, 2020). To answer our second research question, we determined that the items in the ICS and ICS-JC conceptually captured intelligibility. Our results align with previous studies that also showed high internal consistency in bilingual populations: Washington et al. (2017) (English, Jamaican context α = 0.91; JC α = 0.91) and Hopf et al. (2017) (Main Language α = 0.83, Fiji English α = 0.79). Our results also align with monolingual studies: McLeod et al. (2012c) (English, Australian context α = .93), McLeod et al. (2015) (English, normative Australian sample α = 0.94), and Pham et al. (2017) (Vietnamese α = 0.94). Although examination of internal consistency may not seem immediately relevant to clinical practice, these findings suggest that the ICS and ICS-JC questions are beneficial in providing clinicians with the information needed about children’s speech intelligibility.

For our third research question, we sought to determine if subjective parent report scores (i.e., responses on the ICS and ICS-JC) were correlated with the objective measures of speech production (i.e., PPC, PCC, PVC). Pearson’s correlation was used to establish the relationship amongst the variables and identify their strength and direction. The magnitude of the correlation coefficients (Cohen, 1988) were classified as moderate with only one meeting the criterion for the low range (JC PVC: r = .29). Values in the low range may be indicative of some spread in the data set. The statistical significance of the correlations suggested that the accuracy of speech sound production in both English and JC was linked with parental ratings of how well children’s speech was understood by different communication partners. Significant positive correlations between the ICS, ICS-JC, and children’s objective speech measures suggest scores on the questionnaires are an accurate reflection of children’s speech intelligibility, a similar finding in previous studies (McLeod, 2020).

In the ICS and ICS-JC scoring, parents were rated highest for understanding their child in both languages. There was a slight but noticeable difference across languages in the order of the rated listeners who followed. In the ICS, teachers were second best in understanding the child, followed by immediate family. In the ICS-JC, however, these two groups were inverted; immediate family were second best in understanding the child, followed by teachers. This may be due to the changes in language use by Jamaicans based on the social context. For example, English is used mainly in settings that are perceived as formal; thus, teachers are more familiar with the children’s use of English. On the other hand, JC use is more common with immediate family members and may indicate why their understanding of children is higher than teachers’ on the ICS-JC. These examples support how the familiarity of the listener can have a strong influence on how a child is understood.

The ICS and ICS-JC provide a quantitative evaluation of speech intelligibility, which appears to accurately distinguish typical from disordered speech. The ability of these tools to distinguish these two groups of speakers, while not penalizing children who have differences in their speech production due to being bilingual, is an important feature. The information gathered from the screeners allow for a holistic view of the children’s communication abilities and may improve diagnostic and therapeutic decisions (McCormack et al., 2009). Results gathered in this study are comparable to other related ICS studies and broaden the use of the ICS to an understudied bilingual population. Of importance to our study is the auditory presentation of the tools and that its use did not appear to impact the validity of the results. We believe these findings serve as support for researchers and SLPs to continue considering language context when screening for intelligibility and to adapt the ICS to auditory formats when appropriate (i.e., with cultures with a strong oral tradition) (McLeod, 2020; Pascoe & McLeod, 2016; Washington et al., 2017).

Limitations and Future Research

The present study is an extension of Washington et al. (2017) and although some limitations from that study were addressed, there is a continuation of others. While direct assessment of speech production was conducted using native speakers of English and JC, the English dialect used by some assessors did not match that of the participants. The majority of native speakers of English in the Jamaican context are also bilingual users of JC, making the separation of languages that was important to this study not possible. The use of different English dialects may have caused effects that were not controlled. There were also limitations specific to the population sample. A large sample of Jamaican preschoolers who were typically developing and had SSD were recruited from Jamaica, with a smaller sample of Jamaican preschoolers from the United States; yet, our sample does not represent all Jamaican preschoolers. A larger sample, which was more representative of children living in different regions of Jamaica and children with Jamaican parentage living in different countries, would increase the study’s external validity. Future research should also consider Jamaican preschoolers that are dominant in either English or JC. This study included only simultaneous bilinguals, meaning that other bilingual typologies are not represented in this sample. Future research should also consider the utility of the ICS and ICS-JC for Jamaican children with developmental differences, who were intentionally excluded from this study, such as those with oromotor difficulties and neurological, developmental and/or audiological concerns.

Clinical Implications and Conclusions

The findings from the present study contributes knowledge about diagnostic tools to address the paucity of available, valid, and sensitive tools in understudied bilingual contexts. The findings of this study also increase the possibilities for SLPs to be more culturally responsive in their practice. Our study demonstrates that both the ICS and ICS-JC are informative screening tools to guide SLPs in their evaluative and diagnostic decision-making when working with preschool-aged children of Jamaican heritage who use English and JC. Gathering information about children’s functioning in varied settings is an important, but often challenging, task to complete in an initial assessment session. Use of the ICS and ICS-JC provide insight into different contextual factors that may impact on children’s functional use of speech in multiple contexts (as outlined in the ICF-CY). Use of these tools can also inform SLPs of how their clients’ lives are affected by poor speech intelligibility in various contexts. The ICS and ICS-JC apply the lens of the ICF-CY and allow for quick documentation of communication function. This practice is aligned with recommendations of speech-language pathology organizations around the world, such as the American Speech-Language-Hearing Association (ASHA, 2016) and Speech Pathology Australia (Speech Pathology Australia, 2015). Further, the possibility of administering the ICS quickly and easily in an oral format, rather than the traditional written format, means that the ICS can be administered by a speaker of the target language in languages that do not have a written form or where parents have a low level of literacy in the target language. In sum, our data show promising evidence that the ICS and ICS-JC are effective screening tools for three- to six-year-old English and JC-speaking preschoolers that can be administered in a culturally sensitive manner.