Adapting the Early Communication Indicator as a Social Communication Outcome Measure for Young Autistic Children: A Pilot Study

Abstract

Purpose:

We sought to conduct a pilot investigation of the reliability and administration fidelity of a new play-based measure of social communication for infants and toddlers with an autism diagnosis.

Method:

Our team adapted an existing measure, the Early Communication Indicator (ECI), for use with young autistic children in clinical and research contexts. In this brief report, we detail our adaptation process including administration and scoring of the final adapted measure based on data from a two-phase pilot study with young autistic children (N = 17).

Results:

This adapted measure, the Early Communication Indicator–Autism (ECI-A), captured a range of scores for the ECI, Initiation of Joint Attention, and Directed Communication in pilot testing. Interrater reliability was moderate to strong across the scored behaviors. Finally, parents were able to administer the ECI-A with high fidelity with support from the research staff.

Conclusions:

This two-phase pilot study demonstrated promise for the ECI-A as a brief measure of social communication that can be administered by parents and reliably scored by trained staff with limited background in autism assessments. Validation of the ECI-A is presently underway.

Supplemental Material:

https://doi.org/10.23641/asha.26042077

Most of the available validated social communication measures for autistic children were designed to diagnose autism or document social communication skills at a given time rather than to detect change in response to intervention. The few measures that were designed for progress monitoring have significant training and time burdens and preclude clinical use (e.g., Brief Observation of Social Communication Change [BOSCC], Grzadzinski et al., 2016; Grzadzinski & Lord, 2018) or rely on parent report, which is subject to bias (e.g., Social Communication Questionnaire, Rutter et al., 2003). The lack of a time-efficient, validated social communication treatment outcome measure has delayed progress in both autism intervention research and clinical practice, because there are few tools to monitor the effects of intervention programs. It is challenging to distinguish if lack of treatment progress is truly the result of ineffective interventions or, rather, poorly validated social communication measures for young autistic children.

To address this need, our interdisciplinary group of clinical researchers is working to validate a brief play-based observational measure that can easily be used for progress monitoring in both research and clinical practice. The Early Communication Indicator (ECI) is part of a collection of measures called Individual Growth and Development Indicators. It was designed to monitor the expressive communication progress of young children between 6 and 42 months of age and is norm-referenced with live scoring via mobile phone application (Buzhardt et al., 2018, 2020; Greenwood et al., 2013, 2020). The ECI is validated for general early intervention use and shows initial promise for intervention progress monitoring in autistic children (Buzhardt & Meadan, 2022).

The ECI is scored by counting how often a child uses expressive communication during a 6-min play-based session in each of four key areas: gestures, vocalizations, single words, and multiple words. This scoring makes the ECI uniquely valuable for capturing the communication skills and patterns of young autistic children, many of whom show floor effects on standardized expressive language testing or are not able to participate in standardized testing (e.g., do not yet reliably point to pictures or respond to others). Children who communicate using signs/gestures, augmentative communication systems, repetitive vocalizations, scripted phrases, echolalia, and other means common to autistic language learners can all be scored on the ECI, minimizing floor effects and allowing for individual communication patterns over time to be documented (e.g., if a child's frequency of repetitive vocalizations decreases and is replaced with gestures or phrases).

Joint attention (JA) is known to be a critical precursor of later social and language skills in the development of autistic children (Charman, 2003; Poon et al., 2012). Delays in JA skills are one of the earliest traits of autism (Werner & Dawson, 2005), which interact with other developmental processes to have a cascading impact on later language and social communication outcomes (Baranek et al., 2018; Mundy & Jarrold, 2010; Nowell et al., 2020). Considering that 25%–30% of autistic children do not develop functional expressive language (Tager-Flusberg et al., 2005) and over 60% of autistic children have co-occurring language disorders (Levy et al., 2010), monitoring JA is of particular importance for this population.

Our team is adapting the ECI to include probes for Initiation of Joint Attention (IJA) and validating this measure with an autistic sample to capture the social communication skills of autistic children and monitor their development and response to intervention. In this research note, we report our pilot data on the adaptation of the ECI to the Early Communication Indicator–Autism (ECI-A). The aim of this two-phase pilot of the ECI-A validation study was to add IJA probes to the existing ECI assessment. We proposed to do this by determining the best tasks for measuring IJA during the ECI. Specifically, we collected data on items that integrated best with the existing ECI, items likely to be sensitive to change, items that were easiest for parents to administer, and items that were most engaging and enjoyable to young autistic children.

Method

The following methods and procedures were approved by the institutional review board at The University of Kansas Medical Center. The pilot study occurred in two phases. First, four forms of the ECI-A were created, which had IJA probes either embedded into the ECI or before/after the ECI administration. These four forms of the ECI-A were piloted with a sample of 10 children with autism and their parents in North Carolina and Kansas. The goal was to narrow down the four forms in Phase 1 to two forms in Phase 2, aligning with the two forms of the ECI. Data were collected on child communication and IJA, parent-reported ease of use in administering the probes, researcher ratings of the administration success, child enjoyment of IJA probes, and parent fidelity of IJA probe administration rated by researchers. On the basis of the results of Phase 1 and feedback from an advisory board of experts in measurement development and autism in early childhood (see the Acknowledgments section), the research team created two final forms of the ECI-A (Forms A and B; see Table 1), which include both before/after and embedded IJA probes. The revised ECI-A forms were then piloted with seven additional families in Phase 2.

Table 1.

Final Early Communication Indicator–Autism form and activities.

Probe timing	Form A: house playset		Form B: barn playset
	IJA probe	Description	IJA probe	Description
Before ECI	Marker box	Parent establishes coloring routine and then offers the child a feather pencil in the box of markers and waits for the child to initiate JA.	Puzzle	Parent establishes a puzzle routine and then offers the child a pop bracelet instead of a puzzle piece and waits to see if the child initiates JA.
During ECI (at 3 min)	Bean bag	Parent places sensory bean bag filled with soft-colored beads into playset and waits for child to initiate JA for 1 min then removes the object.	Water log	Parent places a water log into the playset and waits for child to initiate JA for 1 min then removes the object.
After ECI (at 6 min)	Massager	Parent turns on a back massager and places it in the playset to see if the child initiates JA.	Light up wand	Parent turns on a spinning LED light wand and places it in the playset to see if the child initiates JA.
After ECI	Surprise bag	Parent offers child a bag of toys and objects to explore.	Dress-up	Parent offers child a box of dress-up accessories to explore.

Note. IJA = Initiation of Joint Attention; ECI = Early Communication Indicator; JA = joint attention.

Autistic children and their parents participated in two study visits in their home approximately 2 weeks apart. Informed consent and demographic information were collected from parents via REDCap survey prior to the first visit. At each visit, parent–child dyads completed a different form of the ECI-A, and parents were asked to give feedback about their experience. Parents were given a choice to do these visits virtually (via Health Insurance Portability and Accountability Act–compliant Zoom) or in person due to the COVID-19 pandemic. Forty percent of Phase 1 visits were in person, and 80% of Phase 2 visits were in person. For both virtual and in-person visits, parents were sent a handout via e-mail prior to each visit with information about what to expect. A member of the research team brought the toy set to the family home. For in-person visits, the masked researcher video-recorded the ECI-A on a tablet while coaching the parent. For virtual visits, the toy set was left at the doorstep, and the parent took the toy set and tablet inside to virtually meet with another researcher who coached and recorded the session over Zoom. The research team members who coached and coded the ECI-A at both sites included a bachelor's-level research assistant, a board certified behavior analyst, two speech-language pathologists, a master's-level ECI training coordinator, and two doctoral-level speech-language pathologists who were also study investigators and authors.

The original ECI requires minimal parent coaching (see administration at guidelines https://igdi.ku.edu/what-are-igdis/how-do-i-administer-igdis/). Coaching for the ECI-A was extended to prepare parents to introduce IJA probe materials into the ECI-A and to assist with the timing of the probe activities so that parents could focus on playing rather than timing. ECI-A coaching involved a researcher reviewing the written and pictorial instructions with the parent before ECI-A administration and modeling how to use the ECI-A materials (e.g., “This is the button you'll push to turn it on”). During the ECI-A, coaching was provided to tell the parent when to administer each probe (e.g., “You can put the first one in now”). If parents asked for or needed more support, it was provided but only as needed (e.g., if the parent asked, “Should I say something?” the researcher would answer, “Wait and see what he does on his own first”). Our goal was to make coaching and administration as simple as possible for use in the field by staff of varying training levels. This is similar to the original ECI, as another person is typically present to coach the play partner and score the ECI.

Participants

We aimed to recruit English-speaking children from 12 to 60 months of age diagnosed with autism using the University of North Carolina at Chapel Hill Autism Research Registry and the University of Kansas Medical Center Autism Clinic. The Phase 1 sample included 10 autistic children (three girls; one African American, one Latino) aged 30–60 months (M = 46 months) with a range of expressive communication skills. The Phase 2 sample included seven autistic children (two girls; one Asian, one African American, one mixed race) aged 24–51 months (M = 38 months). Most were nonspeaking (43%) or used phrase-level speech (29%). Due to COVID-19 delaying and complicating clinical autism diagnoses (see Dahiya et al., 2021; Jang et al., 2021), our recruiting research registries and clinics were not receiving many toddlers, and therefore, our pilot sample was on the older end of our 12- to 60-month target age range.

Measures

The ECI measures expressive communication development in young children (Luze et al., 2001). A parent or other familiar adult administers the ECI by engaging the child with a standard playset, either the Fisher Price House or Barn, for 6 min. Following an administration checklist, adult play partners are encouraged to follow the child's lead (see Hart & Risley, 1975, for more on this incidental teaching technique), minimize questions, allow the child to play as they typically would with the playset, and respond to their child as they normally would throughout the ECI. Four communicative skills are tallied during the ECI: gestures, vocalizations, single words, and multiple words. The total weighted communication score is calculated by summing the sums of the four skills and weighting spoken or signed words by multiplying single words by 2 and multiple words by 3. Although the ECI can be coded live, the ECI-A was coded from video recordings for this pilot study.

IJA probes added to the ECI were based on items from the Joint Attention Protocol (Watson et al., 2003; available open access at https://cdr.lib.unc.edu/concern/generals/x059c9640), a validated measure of JA for autistic children (Nowell et al., 2018) that includes eight play-based activities, or probes, for IJA. Probes generally involve setting up a routine in play and then introducing something unexpected. We developed a coding definition of IJA with examples and counterexamples: The child independently attempts to get the play partner to notice an object/event with intent to share interest. IJA must clearly not be for the purpose of requesting an item or action. A clear instance of IJA involves at least two modalities (gesture, vocalization, or gaze shift). IJA probes were coded from video recordings of the ECI-A. IJA was scored as present “+” or absent “−” for each probe and then averaged to provide a mean IJA score with a possible range from 0 to 1 (i.e., proportion of IJA). See Supplemental Material S1 for complete coding definition and examples.

A Directed Communication rating was developed in order to capture the child's level of interest in the probe materials as communicated to their play partner. This score was intended to encompass communication directed to the adult play partner about the probe materials more broadly than our definition of IJA. Since the ECI provides only frequency counts of the expressive communication a child produces during each session (which can include repetitive words, phrases, or nondirected vocalizations), Directed Communication allows the coder to rate the quality of the child's socially directed expressive communication within the IJA activity probes. Moreover, children with lower levels of communication may direct communication for other purposes (e.g., requests); thus, this quality rating may capture variability prior to the emergence of JA. In addition, we observed in trialing the activities with the children of research team members prior to the pilot studies that as children's language level increased, they were less likely to demonstrate IJA with multiple modalities with a familiar play partner but more likely to verbally share interest (e.g., “Mom, look at this” with no gesture or gaze shift) and engage in back-and-forth conversational exchanges with their play partner about the probe materials. Directed Communication was scored on a 4-point scale from 0 (no related communication/action) to 3 (two or more reciprocal communicative turns related to the probe materials). The Directed Communication rating was coded from video recordings of the ECI-A. See Supplemental Material S1 for complete coding criteria and examples.

The parent interview was administered at the end of each ECI-A assessment. Parents were asked to rate each IJA probe by ease of use on a 5-point scale from 1 (very easy to administer) to 5 (very difficult to administer). They were also asked how much their child enjoyed the activities in the ECI-A from 1 (not at all) to 5 (very much). Finally, parents were asked three open-ended questions: (a) Please share any ideas for what we could do differently to better support families in engaging in the assessment activities, (b) Please share any comments about the virtual assessment experience, and (c) Is there anything else you want us to know about your or your child's experience today?

The researcher interview was completed after each ECI-A administration. Researchers were asked to rate the family's experience administering each IJA probe from 1 (very easy) to 5 (very difficult). At the end of the survey, researchers responded to an open-ended prompt to add comments about the session related to the overall study goals: Is there anything you want to note about this assessment related to: Things that influenced fidelity? Child response to materials/activities? Parent response to materials/activities? Other aspects of the assessment we should know about for the larger study?

Results

Phase 1

Due to the small sample size, we carefully examined descriptive data for each participant dyad on every variable and also mean scores on IJA, communication, and parent fidelity across participants. The final forms are composites of IJA probes from all of the four Phase 1 forms (some embedded probes and some pre/post, which were separated in Phase 1). The IJA tasks dropped were those that resulted in minimal to no IJA or communication used by any of the Phase 1 participants. Parent fidelity was examined and found fairly consistent across IJA probes. No notable differences were found on any measure for dyads who completed the ECI-A in person compared to virtually. Examining the data by form of ECI (house vs. barn), placement of IJA probes (embedded vs. pre/post), parent ease-of-use ratings, child enjoyment (rated by parents), and parent fidelity of administration, we had five major takeaways from Phase 1 of the pilot, which informed the final ECI-A forms (see Table 1):

1. Small tactile sensory IJA probes performed best as embedded probes within the ECI administration, because they were usually not disruptive to the ECI (i.e., child communication scores similar before and after introduction of probe), easy for parents to administer, and consistently engaging to children. These were included in the final ECI-A.

2. Longer joint interactive IJA probes (e.g., dress-up) performed well as pre/post probes for the ECI, because they were easy for parents to administer and allowed for multiple opportunities for IJA. These probes were included in the final ECI-A.

3. More subtle IJA probes embedded into the ECI (e.g., replacing a vehicle in the playset with a vehicle with a missing wheel) were not usually noticed by autistic children and rarely led to IJA. These probes were not included in the final ECI-A.

4. More intense multisensory IJA probes (e.g., remote-controlled car, vibrating ball) also did not result in IJA from children. When children enjoyed these probes, the interest in the object was often intense and not shared with the play partner. In some cases, these probes resulted in negative reactions (e.g., ear covering, hiding). These probes were not included in the final ECI-A, but we found similar sensory items that were less disruptive to pilot-test in Phase 2.

5. Parent fidelity was relatively high (above 70%) for IJA probes whether parents were supported in person or over videoconferencing. We continued to offer both options in Phase 2.

Parents provided qualitative feedback on the measure by responding to the open-ended interview questions. In response to what could be done differently to support families in administering the ECI-A, it was recommended that parents be informed: They do not have to be responsible for remembering when to introduce IJA probes; that all types of play and interaction with the materials were “ok,” particularly for children who were at exploratory play levels; and that the handouts given in advance will help with completing the play interaction. In response to the question on virtual assessments, parents were very positive, and some reported that the virtual option allowed for a more accurate assessment of their child's communication abilities. Finally, parents said that their children really liked the materials, but some of them struggled with being observed for 6 min. Some parents also reported difficulties with engaging their child for 6 min, especially if the child typically plays alone and does not want the parent to play. One parent who self-identified as autistic shared that they preferred building activities with their child compared to pretend play like the dolls and dress-up activities provided.

Researcher comments on fidelity in Pilot 1 included capturing some unanticipated child strengths. These included children who could read the parent handouts and labels on the materials and those who were socially attuned to their parents such that they anticipated the introduction of the IJA probes. Overall, parents were observed to be skilled in flexibly engaging their children, even when their children were not interested in some materials or struggled to transition away from highly preferred materials. One child was more interested in the computer recording the session than the toys themselves, but there were otherwise no challenges noted with fidelity of virtual assessments.

Phase 2

ECI-A

The ECI-A averaged about 12.5 min (M = 12:26, range: 08:43–19:00), which included the four IJA probes. The ECI-A was scored beginning at the time the house (Form A) or barn (Form B) toys were presented for a consecutive 6-min segment—using the standard ECI scoring procedures with count data tallied on child gestures, vocalizations, single-word utterances, and multiple-word utterances. Very low rates of communication were seen overall across the seven participants (e.g., average of 3.29 single words in a 6-min play interaction) but with a fairly wide range (see Table 2). The intraclass correlation coefficients (ICCs) represented data across six to eight coders for each video, some of whom were training on the measure during the pilot study. Overall, there were high intraclass correlations.

Table 2.

Phase 2 pilot descriptive and reliability data.

Measure	Metric	M (SD)	Range	Reliability (ICC)
ECI gestures	Count	3.50 (2.21)	1–9	.65
ECI vocalizations	Count	16.57 (8.25)	6–31	.65
ECI single words	Count	3.29 (5.15)	0–15	.84
ECI multiple words	Count	4.71 (7.86)	0–22	.79
ECI total score	Weighted count	40.79 (30.72)	9–112	.88
IJA	Proportion (±score)	0.14 (0.25)	0–0.75	ICC on M = 0.59; 89% agreement on item-level ratings
Directed Communication	M (0–3 scale)	1.52 (0.72)	0.50–3.00	ICC on M = 0.86; 76% agreement​ on item-level ratings
Parent fidelity	% items correct	89.7% (10.2%)	71.4%–100%
Parent ease of use	M (1–5 scale)	4.00 (0.78)	3–5
Researcher ease of use	M (1–5 scale)	3.27 (0.93)	2–5
Parent rating of child enjoyment	M (1–5 scale)	3.92	2–5

Note. ICC = intraclass correlation coefficients; ECI = Early Communication Indicator; IJA = Initiation of Joint Attention.

IJA

The mean proportion of IJA (times the child initiated JA out of four probe opportunities) was 0.14 (see Table 2). There was a range, although none of the children exhibited JA during every probe activity. We did see floor effects in which three of the participants did not use IJA at all during any of the four probes during either administration. Item-level agreement was high for IJA probe scoring, but the ICC on the mean was only moderate.

Directed Communication

We calculated mean ratings for each of the child participants across the four probe activities. The overall mean was 1.52, which is halfway between the rating of 1 (nondirected communication) and 2 (directed communication). There was a wide range of mean scores and strong reliability between raters. See Table 2 for additional data.

Parent Fidelity

Parent fidelity of ECI-A administration was coded from video by research staff. Mean fidelity was about 90%, and all parents were above 70% for fidelity of administration (see Table 2).

Parent-Rated Ease of Use

All activities were individually rated on a 1–3 scale, where 1 = very easy, 2 = easy, and 3 = not easy. The average parent rating was 1.75 (0.29) with a range from 1 to 3. Most parents rated the puzzle activity and the embedded sensory probes (e.g., placing a water log in the playset) as very easy. The coloring, dress-up, and mystery bag activities were variably rated, which seemed to depend on child interest in engaging with the materials. Parents had to work harder to administer the probes with fidelity when their child showed minimal interest. They also rated their ease of use overall on a scale from 1 (these activities were very difficult to use with my child) to 5 (these activities were very easy to use with my child; M = 4). Parents also rated their child's enjoyment of the ECI-A assessment overall on a scale from 1 (not at all) to 5 (very much). Overall, parents rated their child's enjoyment between somewhat and quite a bit (M = 3.92).

Qualitatively, many parents reported that their children liked “physical play” and were not yet engaging with pretend play materials like the little figures in the ECI-A set. One parent reported that they were not sure whether to do “hand over hand” with the puzzle or just let the child play how they wanted to play. Placement of the camera was reported to be the most challenging part of the virtual assessments in Phase 2. Some parents were surprised by how well their child engaged in the assessment (e.g., “I learned some things about him—actually picking up the crayon and coloring, seeing how interested he is in different things, noticing the toy—seeing him notice things”; “He had fun!“), and others noted situational challenges such as their child having just woken from a nap or being really dysregulated on the day of the ECI-A administration.

Researcher-Rated Ease of Use

Researchers rated the family ease of use overall from 1 (very difficult) to 5 (very easy; M = 3.27; see Table 1). Researcher ratings were slightly lower than parent ratings, likely due to their ECI training and knowledge of administration fidelity. Qualitatively, researchers reported that this Phase 2 sample, which included children with very low expressive language levels, had more trouble engaging with the playsets compared to Phase 1. Some children played repetitively with preferred toys in the home that were not ECI-A materials (e.g., going down a slide in the living room) or struggled to transition between toys in the toy set (e.g., one child really liked the horse in the barn set). As in Phase 1, parents were skilled with overcoming these obstacles and co-regulating with and then engaging their child. One parent was noted to sing and use physical touch throughout the ECI-A to keep their child calm and engaged. In some cases, these child behaviors interfered with the ECI-A in that it took longer to capture the 6 min of ECI play since the ECI is paused whenever the child is upset. Whenever possible, we allowed children the time they needed to enjoy preferred materials and transition smoothly.

Comparing scores on measures across the different forms of the ECI-A (A and B), mean IJA (0.14) and communication scores (1.5) were nearly identical. We also looked at retesting differences between scores at the two administrations (2 weeks apart). Communication mean scores were slightly higher at Time 1 (1.8) compared to those at Time 2 (1.3) in this small sample. There were no differences in scores between virtual and in-person visits, but anecdotally, parents seemed to prefer having in-person support when given the choice.

Discussion

In this two-phase pilot study, our team successfully adapted the ECI for young autistic children and their parents. In an ongoing project, we are validating the ECI-A against existing measures and investigating the degree to which the ECI-A is sensitive to this population's communication development. The ECI-A captured a range of scores for the ECI, IJA, and Directed Communication scores in pilot testing. Overall, interrater reliability was moderate to strong across the scored behaviors from video recordings of the ECI-A. Finally, parents were able to administer the ECI-A with high fidelity with support from the research staff.

If validated, the ECI-A could meet a clinical and research need as a brief observational progress monitoring measure of social communication for young autistic children. First, the average time of administration for the ECI-A was 12.5 min for this pilot study, which is shorter than other measures available (e.g., ESCS [Mundy et al., 2003] and Communication and Symbolic Behavior Scales [CSBS; Wetherby & Prizant, 2003] take 15–25 min) and similar to the administration time for the BOSCC (about 12 min). Furthermore, the ECI-A was administered with high fidelity by parents (rather than clinicians) and could be administered in person or virtually. As discussed previously, the open-ended play aspect of the ECI-A allows for scoring of children who often struggle with standardized assessment procedures, while scoring based on frequency counts can reduce floor effects in young autistic children with minimal expressive communication. Although scored from video rather than live for this pilot study, ECI-A coding burden was substantially lower for observers compared to other behavior coding measures of social communication. For instance, the ECI-A and BOSCC are being coded for the validation study, and the ECI-A takes coders around 15–25 min to code, while the BOSCC takes an average of 90 min to code. Training time and expertise needed for administration is also lower for the ECI-A. For example, the ECI-A can be administered and coded by undergraduate and naïve research assistants, unlike other measures that were designed to be administered and interpreted by licensed clinicians (e.g., ESCS, CSBS). Meanwhile, the BOSCC may be coded by undergraduate and naïve research assistants, but the initial training and ongoing coding to achieve and maintain reliability are quite time consuming. The ECI-A would not replace these other valuable measures of social communication in autism, but rather offer another option for intervention progress monitoring needs in both research and clinical settings.

Further validation of the ECI-A is necessary before it is used in research or clinical practice. Our team is currently collecting data from a diverse sample of autistic and neurotypical children on the ECI-A and other established measures. In response to parent qualitative feedback from this pilot work, we ask parents in advance about their child's play level and what supports they may need to engage in the ECI-A (e.g., visual schedule, play mat to define physical boundaries, first–then board, visual countdown). We will be able to assess concurrent and divergent validity, sensitivity to change, and interrater reliability of the adapted measure as well as report feedback from families with a range of backgrounds and lived experiences.

Data Availability Statement

Much of the data supporting the results of this study are available in the National Database for Autism Research. Full data sets (including qualitative responses) and coding definitions are available at https://nda.nih.gov/edit_collection.html?id=3625 and by request from the corresponding author at nowell@unc.edu.

Funding Statement

This study was funded by National Institute of Child Health and Human Development (NICHD) Grant R01 HD100364-01A1 (awarded to Brian Boyd). Assistance for this project was provided by the University of North Carolina (UNC) Intellectual and Developmental Disabilities Research Center (NICHD; P50 HD103573; PI: Joseph Piven) and the UNC TEACCH.