Balancing Standardization and Ecological Validity in the Measurement of Social Communication Intervention Outcomes

Abstract

Purpose:

Caregiver-mediated communication intervention outcomes are inconsistently measured, varying by assessment settings, materials, and activities. Standardized materials are often used for measuring outcomes, yet it remains unknown whether such standardized contexts equitably capture caregiver and child intervention outcomes representative of dyads' typical interactions. This within-subject study investigates how intervention outcomes differ between family-selected and standardized interactional contexts for autistic toddlers and their caregivers.

Method:

Following an 8-week caregiver-mediated telehealth intervention delivered to 22 dyads, caregiver outcomes (fidelity of using responsive communication facilitation strategies) and child outcomes (total spontaneous directed communicative acts) were measured during two interactional contexts using (a) family-selected activities and (b) a standardized toy set. A routines checklist surveyed the activities dyads value, enjoy, complete frequently, and/or find difficult with their child.

Results:

Caregiver outcomes and child outcomes did not significantly differ between the family-selected and standardized interactional contexts. Descriptive results suggest that the types of toys commonly included in standardized toy sets are representative of the materials many families choose when playing with their child at home. However, during the family-selected interactional context, the majority of dyads also chose materials or activities that were not available to them during the standardized context.

Conclusion:

It is necessary to carefully consider a more expansive approach to standardization in which intervention outcomes are measured in ecologically valid contexts, which meaningfully, accurately, and equitably capture caregiver and child functional outcomes, and the translation of interventions to families' everyday routines.

Social communication interventions, particularly those delivered jointly by therapists and parents, have been shown to effectively support autistic children's communication skills, as early as toddlerhood (Hampton & Kaiser, 2016). However, communication outcomes in studies evaluating the efficacy of interventions for autistic children are inconsistently measured, with significant variability in assessment settings, materials, and activities (Heidlage et al., 2020; Sandbank et al., 2020). Limited research examines the impact of this inconsistency on intervention outcomes, despite the importance of outcome measures, which are critical to establishing evidence-based practices (Roberts & Kaiser, 2011). Given that outcomes are commonly measured during caregiver–child interactions (e.g., play, daily routines), the context of these interactions is likely to impact not only the measurement of specific child and caregiver behaviors but also the interpretation of intervention effects (Heidlage et al., 2020). To account for extraneous variability in contexts, researchers often use standard sets of toys and other standardized conditions when characterizing caregiver–child interactions (Bottema-Beutel & Kim, 2021). However, it remains unknown the extent to which such standardized contexts equitably capture caregiver and child intervention outcomes that are representative of each dyad's typical interaction.

Background on the Measurement of Caregiver-Mediated Intervention Outcomes

Caregiver-mediated interventions are considered evidence-based practice for improving young children's communication and caregivers' use of communication facilitation strategies (Deniz et al., 2022; Heidlage et al., 2020; Roberts et al., 2019). Caregiver-mediated interventions employ a cascading, family-centered service delivery model, in which a speech-language pathologist (SLP) coaches a caregiver to use communication facilitation strategies with their child during play and other daily routines (Roberts et al., 2014). In contrast to clinician-implemented interventions, caregiver-mediated interventions seek to enhance the quantity and quality of caregiver–child interactions in order to optimize child communication (Jeong et al., 2021). The theoretical basis for caregiver-mediated interventions stems from transactional models of child development and social-interactionist theory. Together, these theories suggest that children learn language through reciprocal and bidirectional exchanges with adults in their immediate environments (Sameroff & Mackenzie, 2003; Vygotsky, 1978). As such, caregiver-mediated interventions are frequently delivered in naturalistic settings to maximize the relevance of the intervention and increase the frequency of communication facilitation strategy use during everyday caregiver–child interactions (Sone, Lee, & Roberts, 2021). Although caregiver-mediated interventions are commonly delivered in families' homes and caregivers practice the strategies with the toys and activities available in their home, intervention outcomes are frequently measured in less familiar interactional contexts (e.g., Barrett et al., 2020; Gengoux et al., 2019; Roberts et al., 2022).

A variety of methods have been used to measure child communication outcomes and caregiver learning outcomes following a caregiver-mediated intervention. Child communication outcome measures include caregiver report of language (e.g., MacArthur–Bates Communicative Development Inventories; Fenson, 2007), standardized assessments (e.g., The Preschool Language Scales–Fifth Edition; Zimmerman et al., 2011), and observational methods (e.g., language sample; Barokova & Tager-Flusberg, 2020; Heidlage et al., 2020). Additionally, caregiver fidelity outcomes are frequently unreported and, when they are, vary considerably in the settings, routines, and materials (Barton & Fettig, 2013; Roberts et al., 2019). Such underreporting of caregiver fidelity has prompted the development of fidelity scales such as the Naturalistic Developmental Behavioral Intervention Fidelity Rating Scale (NDBI-Fi) and the Measure of NDBI Strategy Implementation–Caregiver Change, which assess the fidelity of caregivers' use of Naturalistic Developmental Behavioral Intervention (NDBI) strategies on a common scale and can be used across interactional contexts (Frost et al., 2020; Sone, Kaat, & Roberts, 2021; Vibert et al., 2020). Consistency of measurement is particularly imperative given that studies assessing the efficacy of caregiver-mediated interventions have shown mixed results for both child and caregiver outcomes, perhaps due to inconsistent outcome measures (Roberts et al., 2019). In addition to variability in what caregiver and child outcome measures are used, how these constructs are measured is also notably incongruous in the caregiver-mediated intervention literature. One of the most marked differences is in the standardization of the child–caregiver interactional context, with some studies measuring outcomes in laboratory settings and/or with standardized toy sets (e.g., Kasari et al., 2010; Roberts et al., 2022) and other studies measuring outcomes in home settings during child–caregiver play or daily routines (e.g., Bradshaw et al., 2017; Brian et al., 2017).

Importance of Considering Measurement Context

Despite a paucity of research comparing the effect of standardizing interactional contexts on intervention outcomes, it is well established that characteristics of caregiver–child communication are context dependent. Setting (e.g., lab or home), task type (e.g., structured or unstructured), and toy type (e.g., electronic toys, book reading, traditional toys) have been shown to impact caregiver and child language and behavior (Hoff-Ginsberg, 1991; Holme et al., 2022; Kwon et al., 2013; Sosa, 2016; Tamis-LeMonda et al., 2017). For instance, caregivers tend to use more directive interactional styles in novel or difficult tasks (González, 1996). Furthermore, toddlers have been found to communicate more frequently during free-play compared to structured tasks (Kwon et al., 2013). Given the reciprocal nature of caregiver–child communication, even small changes in the dyadic interactional context can consequentially impact both caregiver and child intervention outcomes (Hudry et al., 2023). The combined direct effects (e.g., child or caregiver altering their communication as a result of changes in the interactional context) and indirect effects (e.g., child or caregiver altering their communication as a result of their partner's altered communication) of standardizing the contexts for outcome measures warrants concurrently comparing child and caregiver outcomes between different observational contexts following intervention.

Measurement of caregiver-mediated intervention outcomes should also account for the similarity, or lack thereof, between the interactional context of the intervention and the interactional context of outcome measures. The Individuals with Disabilities Education Improvement Act prioritizes family-centered therapy delivered via interventions for toddlers that support caregiver–child interaction in contexts that are typical for the client and their family (Individuals with Disabilities Education Act, 2004). Family-centered practice also involves prioritizing the use of materials already present in the child's natural environment during therapy (Lee et al., 2022). Increasing the relevance and ecological validity of interventions for young children and their families maximizes the likelihood of intervention acceptability and improvement in children's skills (Schreibman et al., 2015). The standardized toy sets frequently used to measure intervention outcomes may not reflect the materials and activities prioritized by or available to all families participating in interventions, such as families living in poverty (Nwokah et al., 2013). While standardized activities reduce extraneous variability between participants, they may capture more distal outcomes (i.e., generalization of skills to less familiar contexts) that are less sensitive to proximal intervention outcomes detectable immediately following the intervention. This is supported by evidence that effect sizes are often larger for context-bound outcome measurements (i.e., intervention and outcome context match) compared to generalized outcome measurements showing change across contexts (i.e., intervention and outcome context mismatch; Sandbank et al., 2020). Furthermore, outcomes measured during standardized activities may be even more distal for families who do not typically use materials such as those in standardized toy sets, potentially contributing to inequities in these research methods. Additionally, a mismatch between the context of the intervention and assessment prompts a critical and unaddressed methodological question as to whether interactions captured while using standardized toys are indeed measuring outcomes that are representative of the typical caregiver–child interaction.

Objectives

This exploratory study involving secondary data analysis addresses the potential impact of standardizing caregiver–child interactional contexts on the measurement of intervention outcomes commonly used in early childhood intervention research. The purpose of this study is to:

1. Examine the extent to which caregiver intervention outcomes (caregiver fidelity of using of communication facilitation strategies) differ between family-selected and standardized interactional contexts.

2. Examine the extent to which child communication outcomes (child spontaneous directed communicative acts) differ between family-selected and standardized interactional contexts.

3. Describe the activities, materials, and routines that caregivers report prioritizing in daily interactions with their child and chose during the family-selected interactional context.

4. Describe how the activities and materials chosen by caregiver–child dyads during the family-selected interactional context compare to those provided to families in a standardized toy set.

Method

Participants

This within-subject study involved exploratory secondary analyses of 22 autistic toddlers and their mothers who participated in a larger randomized clinical trial intended to compare the effects of in-person and telehealth caregiver-mediated interventions (NCT04501588, https://clinicaltrials.gov). Data collection occurred between April 2020 and August 2021, a period characterized by stay-at-home orders among other COVID restrictions in Illinois. The Early Intervention Research Group recruited mother–child dyads in the Chicagoland area through Illinois Early Intervention providers, pediatricians, and autism diagnostic clinics. Dyads were eligible for this study if they had (a) participated in the telehealth arm of the clinical trial (see https://reporter.nih.gov/project-details/8943128 for a detailed description of eligibility criteria, which was confirmed via parent report) and (b) completed both a standardized and family-selected interaction post-intervention. Toddlers participating in this study (M = 35.98 months, SD = 5.5, range: 27–45) were 77.3% male participants (n = 17). They represented diverse racial and ethnic backgrounds, with 81.8% (n = 18) of mothers identifying their child as Hispanic and/or non-White. Less than half of mothers in this study were employed either part time or full time (n = 10, 45.4%), and mothers represented a broad range of educational backgrounds, with 13 mothers (59%) receiving college or graduate degrees. Half of the families (n = 11) reported an income of less than $75,000 (Chicago's median household income is around $72,000). Table 1 displays participating mother and child demographic characteristics.

Table 1.

Family demographics.

Characteristic	Child	Mother
	n = 22	n = 22
Age (mo), M (SD)	35.98 (5.5)	—
Gender, n (%)
Male	17 (77.3)	0 (0)
Female	5 (22.7)	22 (100)
White non-Hispanic, n (%)	4 (18.2)	5 (22.7)
Non-White and/or Hispanic, n (%)	18 (81.8)	17 (77.3)
Black	6 (27.3)	6 (27.3)
Native American/Alaska Native	1 (4.5)	1 (4.5)
Asian	2 (9.0)	3 (13.6)
Multiple	3 (13.6)	1 (4.5)
No racial category specified	3 (13.6)	4 (18.2)
MCDI total number of words produced,a M (SD)	77.45 (98.25)	—
RBS-R overall score,b M (SD)	22.85 (17.16)	—
Employment status, n (%)
Employed full-time	—	9 (40.9)
Employed part-time	—	1 (4.5)
Stay-at-home parent	—	9 (40.9)
Not employed	—	3 (13.6)
Education, n (%)
Some high school	—	2 (9.0)
High school graduate	—	3 (13.6)
Special training	—	2 (9.0)
Some college	—	2 (9.0)
College graduate	—	6 (27.3)
Graduate degree	—	7 (31.8)
Income, n (%)
< $30,000	—	3 (13.6)
$30,000–$74,999	—	8 (36.4)
> $75,000	—	10 (45.5)
No response	—	1 (4.5)
Government assistance, n (%)
Receiving assistance	—	6 (27.3)
Not receiving assistance	—	16 (72.7)

Note. mo = months.

^aMCDI = MacArthur–Bates Communicative Development Inventories.

^bRBS-R = Repetitive Behaviors Scale–Revised; missing data from two participants.

Procedure

All mother–child dyads participated in an 8-week caregiver-mediated intervention delivered via telehealth, during which mothers received weekly hour-long coaching sessions to learn responsive communication facilitation strategies (e.g., responding contingently to their child's communication and following their child's lead). Responsive strategies are an evidence-based component of NDBIs, rooted in developmental theory, which improve young autistic children's social communication outcomes (Sandbank et al., 2020). See the work of Sone (2023) for a detailed description of the protocols for training, fidelity, and implementation of these intervention strategies.

After completing the full 8-week intervention program, all mother–child dyads were video-recorded at home over Zoom in two interactional contexts for 12 min each, using (a) family-selected activities (“family-selected interactional context”) and (b) a standardized toy set (“standardized interactional context”). All families were provided iPads and tripods, as well as audiovisual instructions about how to operate the equipment to support the quality of data collection. Assessors delivered instructions via a standardized script and followed a checklist protocol to ensure high-quality video capture. No protocol deviations regarding audiovisual quality or interruptions were reported by assessors. The standardized toy set included developmentally appropriate toys and sensory materials for toddlers commonly used to measure child communication spanning a variety of play levels (Condouris et al., 2003; Greenwood et al., 2010; Kasari et al., 2010; Roberts et al., 2014; see Table 3 for a comprehensive list of all items). Participants were instructed to “please interact just like you normally do at home” and asked to avoid electronics such as iPads or cellphones. Prior to the standardized context only, mothers were instructed to play with any or all of the toys provided to them and asked to avoid playing with any of the child's toys from home. Families were instructed not to use the standardized toy set during the family-selected interaction to control exposure to these specific toys across participants. Per protocol, 19 dyads completed the family-selected interactional context before the standardized interactional context to allow for smooth transitions between activities. Three families completed the conditions on different days due to scheduling challenges early in the pandemic. Child and caregiver outcomes, as described below, were scored using the middle 10 min of mother–child recorded interactions in order to provide sufficient time for the dyad to transition between activities. Mothers were not aware of this study's specific research questions but provided informed consent for all study procedures and the use of the data to answer research questions outside the primary aims of the clinical trial.

Table 3.

Materials/activity types chosen by mother–child dyads during the family-selected and standardized interactional contexts.

Materials/activity type	Family-selected interactional context	Standardized interactional context
Materials/activity types available in standardized contexta	20 (90.9)	22 (100)
Transportation toys	11 (50.0)	20 (90.9)
Physical or social play	10 (45.5)	5 (22.7)
Puzzle toys	9 (40.9)	10 (45.5)
Stacking toys	8 (36.4)	14 (63.6)
Dolls/toy animals	6 (27.3)	7 (31.8)
Pretend food play	2 (9.1)	6 (27.3)
Miniature lifelike toys (e.g., doll house, farm, school)	2 (9.1)	9 (40.9)
Singing	2 (9.1)	3 (13.6)
Ball	1 (4.5)	5 (22.7)
Sensory materials (e.g., popsicle sticks)	1 (4.5)	1 (4.5)
Materials/activity types unavailable in standardized context	17 (77.3)	—
Musical toys	7 (31.8)	—
Other daily routines (e.g., snack, handwashing, tooth-brushing)	4 (18.2)	—
Sensory play (e.g., water, Play-Doh)	4 (18.2)	—
Stringing beads/small objects	3 (13.6)	—
Book reading/identifying picture cards	2 (9.1)	—
Art activity	1 (4.5)	—

Note. Em dashes indicate that those fields are not applicable, because it is referencing how many families used materials or activities unavailable in the standardized interactional context (i.e., only possible to count in the family-selected interactional context).

^aThe standardized context materials include Lego set, doll house, school bus, puzzle, ball, stacking cups, shape sorter, dolls, tea party set, plastic food set, dump truck, and assorted sensory materials (popsicle sticks, pompoms, sponges).

Measures

Both caregiver and child outcomes were measured during the middle 10 min of the recorded mother–child interactions. Because materials were in view during all recordings of mother–child interactions, it was not possible for coders of caregiver and child outcomes to be naive to interactional context.

Caregiver outcomes. Caregiver outcomes (fidelity of using responsive communication facilitation strategies) were measured using the responsive composite score of the NDBI-Fi observational rating scheme (Frost et al., 2020), which has demonstrated strong convergent validity (r = .58, p < .001) and interrater reliability (Krippendorff's α = .774; Sone, Kaat, & Roberts, 2021). This measure was chosen as it was designed by expert consensus to measure caregivers' use of NDBI strategies across NDBI programs and was validated across multiple NDBI programs (Frost et al., 2020, Sone, Kaat, & Roberts, 2021). The NDBI-Fi responsive composite score reflects a global rating of a mother's competency (i.e., quality and quantity) using responsive NDBI strategies overall, for each 10-min interaction. Four responsive communication facilitation strategies factor into this composite score, which include (a) following their child's lead, (b) adjusting their language to meet the child's developmental level, (c) responding to their child's communication attempts, and (d) appropriately pacing their communication to provide the child opportunities to communicate. Each of these criteria were scored on a 5-point ordinal scale and then averaged to compute the responsive composite score, with higher scores indicating overall higher competence in using the taught responsive strategies. See Supplemental Material S1 for the full rating scheme and scoring.

Raters were doctoral students who were licensed SLPs or master's students in speech-language pathology. Raters were trained to reliability on the full NDBI-Fi measure, composed of nine items. Reliability was achieved once ratings on three consecutive standard videos met the criteria described in the work of Frost et al. (2020): (a) seven items within 1 point, (b) no items greater than 2 points apart, and (c) composite scores within 0.5 points. Separate raters were used to score caregiver outcomes for a given dyad's family-selected and standardized interactional contexts to reduce the bias inherent in global rating schemes. Interrater reliability was completed at random across conditions for 34% of the sample (n = 15) and measured to be α = .73, indicating acceptable interrater reliability (Krippendorff, 2004). Coders met with the lead coder to review discrepancies and develop consensus whenever their reliability on an individual video was less than .70.

Child outcomes. Child outcomes (number of spontaneous directed communicative acts) were extracted from behavioral microcodes of the 10-min interactions using Mangold INTERACT Software (Mangold, 2020). Each child communicative act was coded as being spontaneous if it was not imitated, elicited, or prompted by the child's mother. Directed communication was defined as (a) child communication (vocalizations, words, signs, or gestures) paired with clear eye contact; (b) simultaneous use of a gesture and vocalization; (c) use of a give, point, or show gesture to request or comment; or (d) use of the mother's name or “you” during a communicative turn. For example, if a child independently pointed to a toy to label it, this would be coded as a spontaneous directed communicative act, but if the child was responding to the caregiver asking for the label of the object, it would not be coded because it was prompted. Coders included two doctoral students and master's level SLPs who were trained to 80% reliability on the behavioral code by achieving 80% interobserver agreement across all codes on three consecutive videos. Interrater reliability was calculated on 20% of the videos (n = 9) and reached α = .73, indicating acceptable reliability (Krippendorff, 2004).

Play setting and routines. The materials and activities the mother–child dyads chose during each interactional context were coded and grouped into categories such as toy play, physical play, social play, singing, and daily routines. Only activities or materials that the child and/or mother engaged with for more than 15 s were considered in this code. Lastly, prior to the start of intervention, all families completed a daily routines survey to report their value and frequency of common routines such as play, art, music and dance activities, chores, grooming and dressing, mealtime, book reading, and bath time (see list of surveyed routines in Table 2). Mothers endorsed whether each of these routines were important to them, a frequent activity for their family (i.e., completed at least three times per week), enjoyable to their child, and/or difficult for them and their child.

Table 2.

Participants' experience with common routines.

Routine, n (%)	“It is important to me”	“We do it at least three times per week”	“It is enjoyable to my child”	“It is difficult for me and my child”	Completed during family-selected context
Play	21 (95.5)	17 (77.3)	22 (100)	10 (45.5)	21 (95.5)c
Toy play	19 (86.4)	17 (77.3)	19 (86.4)	8 (36.4)	20 (90.9)
Sensory play	15 (68.2)	10 (45.5)	18 (81.8)	4 (18.2)	5 (22.7)
Outdoor play	15 (68.2)	11 (50.0)	22 (100)	1 (4.5)	0 (0)
Art, music, and dance	18 (81.8)	18 (81.8)	22 (100)	14 (63.6)	5 (22.7)
Singing songs	14 (63.6)	14 (63.6)	20 (90.9)	4 (18.2)	2 (9.1)
Dancing	12 (54.5)	13 (59.1)	15 (68.2)	6 (27.3)	0 (0)
Playing musical instruments	7 (31.8)	8 (36.4)	13 (59.1)	6 (27.3)	2 (9.1)
Drawing	10 (45.5)	6 (27.3)	10 (45.5)	9 (40.9)	1 (4.5)
Choresa	10 (45.5)	11 (50.0)	14 (63.6)	14 (63.6)	0 (0)
Grooming and dressingb	17 (77.3)	17 (77.3)	15 (68.2)	11 (50.0)	2 (9.1)
Eating meals	17 (77.3)	16 (72.7)	14 (63.6)	5 (22.7)	3 (13.6)
Book reading	17 (77.3)	13 (59.1)	14 (63.6)	8 (36.4)	2 (9.1)
Bath time	16 (72.7)	16 (72.7)	17 (77.3)	4 (18.2)	0 (0)

^aThis category includes feeding pets, doing laundry, cooking/preparing meals, setting the table, cleaning dishes, sweeping/vacuuming, and picking up the mail.

^bThis category includes brushing hair, brushing teeth, putting shoes on/off, and dressing/undressing.

^cThis total includes physical and social play, which were not surveyed in the routines checklist.

Analysis

Wilcoxon paired signed-ranks tests were used to separately compare caregiver intervention outcomes (caregiver fidelity of using of communication facilitation strategies; RQ1) and child communication outcomes (child spontaneous directed communicative acts; RQ2) between the family-selected and standardized interactional contexts. Nonparametric tests were chosen due to the small sample size and nonnormally distributed dependent variables. A priori significance levels for these comparative analyses were set at p < .05. Findings for RQ1 and RQ2 were summarized using descriptive statistics including medians (Mdns) and interquartile ranges (IQRs), given the small sample size and nonparametric statistical tests. Effect sizes (r) were calculated using r = abs(Z)/√N (Rosenthal et al., 1994). Percentages were used to descriptively summarize families' perceived value of toy play and other daily routines (RQ3), to describe common activities and materials dyads chose during the two interactional contexts, and to assess dyads' patterns of toy play between contexts (RQ4).

Results

Caregiver and child outcomes between contexts (RQ1 and RQ2). Results of the Wilcoxon paired signed-ranks tests for this study indicated that caregiver outcomes (fidelity of using communication facilitation strategies) did not significantly differ between the family-selected and standardized interactional contexts (V = 40.5, Z = −1.70, p = .089, r = .36). Mothers' fidelity of taught communication facilitation strategies was commensurate across conditions (Mdn_{family-selected} = 3.50, IQR_{family-selected} = .69, Mdn_standardized = 3.63, IQR_standardized = .50). While not statistically significant, 54.5% (n = 12) of mothers showed lower fidelity of communication facilitation strategies in the family-selected context. Child outcomes (number of spontaneous directed communicative acts) similarly did not significantly differ between the two interactional contexts (V = 156, Z = −1.89, p = .059). However, it is of note that differences were approaching significance with children communicating more in the family-selected context (Mdn = 3, IQR = 12.00) compared to the standardized context (Mdn = 2, IQR = 6.75). Results for child outcomes yielded a medium effect size (r = .40), favoring the family-selected context. Descriptively, 59.1% (n = 13) of children showed more spontaneous directed communication in the family-selected context compared to the standardized context. It is important to note that this analysis was not able to account for order effects because conditions were not counterbalanced. See Figures 1 and 2 for box plots displaying individual caregiver and child intervention outcome scores by condition. Individual dyads' difference scores (family-selected minus standard context) for the complete participant sample are presented in Supplemental Material S2.

Figure 1.

Box plot of caregiver outcomes (fidelity of using communication facilitation strategies) by interactional context. NDBI-Fi = Naturalistic Developmental Behavioral Intervention Fidelity Rating Scale.

Figure 2.

Box plot of child outcomes (count of spontaneous directed communicative acts) by interactional context.

Reported and observed routines/activities (RQ3). On the routines survey that all mothers completed, play was the routines category most frequently rated as being important to them (95.5%, n = 21) and enjoyable to their child (100%, n = 22; see Table 2). Mother–child dyads of 77.3% (n = 17) reported that they frequently engaged in toy play, and 86.4% (n = 19) find toy play important and enjoyable. In contrast, 36.4% of mothers (n = 8) reported toy play as being difficult for them and their child. Sensory play and outdoor play were considered important to 68.2% of mothers in the study (n = 15). In comparison to mothers' report of these daily routines, all but one mother–child dyad (95.5%, n = 21) engaged in play during the family-selected interactional context, with 90.9% (n = 20) playing with toys. Sensory play was also frequently rated by many mothers as an activity enjoyable to their child (81.8%, n = 18), and during the family-selected interactional context, 22.7% (n = 5) played with sensory materials, such as water, Play-Doh, and marbles.

While all 22 mothers reported that outdoor play and art, music, or dance activities were enjoyable to their child, these were not common activities chosen by mother–child dyads during the family-selected interaction. Three mother–child dyads chose to eat meals or snacks during the family-selected interactional context (13.6%), an activity that the majority of mothers value (77.3%, n = 17), children enjoy (63.6%, n = 14), and dyads complete frequently (72.7%, n = 16). Five mothers (22.7%) reported that mealtime is difficult for them and their child. During the family-selected interactional context, two mother–child dyads (9.1%) engaged in book reading, two mothers (9.1%) facilitated personal-hygiene routines (i.e., washing hands, brushing teeth), and no participants chose to complete a chore, routines endorsed by over one third of mothers as being difficult for them and their child.

Family-chosen versus standardized materials and activities (RQ4). The most notable similarities between the materials and activities chosen by mother–child dyads in the family-selected interactional context and the toys provided to families for the standardized interactional context include preference for transportation toys, puzzle toys, and physical or social play (e.g., chase, jumping, cuddling; see Table 3). Physical and social play were more common in the family-selected context (45.5%, n = 10) compared to during the standardized context (22.7%, n = 5), whereas the opposite trend was noted for pretend food play, miniature lifelike toys, and transportation toys. During the family-selected context, 90.9% (n = 20) of mother–child dyads chose to play with types of materials (e.g., puzzle toys) or complete activities (e.g., physical or social play) that were available to them during the standardized context. However, 77.3% (n = 17) of participants chose to also play with types of materials or complete activities that were not available to them during the standardized context. Some of these activities included playing with musical toys (31.8%, n = 7); completing a daily routine, such as snack or a personal hygiene activity (18.2%, n = 4); engaging in sensory play with water or Play-Doh (18.2%, n = 4); and book reading (9.1%, n = 2).

Discussion

Methodological decisions regarding outcome measures in clinical trials are of critical importance. The purpose of this exploratory within-subject study was to compare caregiver and child intervention outcomes measured across two interactional contexts following a caregiver-mediated communication intervention: one interactional context using family-selected activities and materials, and one interactional context using a standardized set of toys. While standardized toy sets are commonly used as a context for measuring child and caregiver intervention outcomes, this is the first study to consider the potential impact of interactional context on outcome measures in a sample of autistic children and their caregivers. Of particular clinical and research importance, these results represent a sample of racially and ethnically diverse families and low socioeconomic status households, participants who are disproportionately underrepresented in extant research on parent–child interactions across contexts (Holme et al., 2022). Furthermore, the use of outcome measures specific to NDBI programs is important given that this type of intervention program is the most frequently researched type of intervention for young autistic children (Sandbank et al., 2023); thus, there are important considerations for the appropriateness of measurement contexts used in this field.

Major Findings

Findings indicate that both caregiver intervention outcomes (use of communication facilitation strategies) and child intervention outcomes (spontaneous directed communication) were comparable across interactional contexts. The similarity in outcomes regardless of interactional context may be explained by the fact that during the family-selected interactional context, all but two mother–child dyads chose to play with toys commensurate with those in the standardized toy set. In addition, most mothers in the present study reported that they frequently engaged in toy play and found it enjoyable and important. Taken together, this suggests that the types of toys commonly included in standardized toy sets are representative of the materials many families chose for play, which is a routine highly familiar to and preferred by children and their caregivers. While extant research has found elements of child–caregiver interactions (e.g., caregiver responsiveness, child communication, or dyadic joint engagement) to differ based on context (e.g., toy play compared to caregiving routines or gross motor play; Binns et al., 2022; Holme et al., 2022; Lindsey et al., 2010), the similarity of the two interactional contexts in this study may explain the comparable measurement of intervention outcomes between the family-chosen and standardized conditions.

Although children's communication did not significantly differ in the interactional context using family-selected materials compared to play with a standardized toy set, this difference was approaching statistical significance with 13 of the 22 children communicating more in the family-selected interactional context compared to the standardized context. While dyadic interactions vary across contexts (e.g., play, book reading, grooming, feeding), within neurotypical children's play, outcomes are unlikely to vary across specific traditional toys (Barokova & Tager-Flusberg, 2020), such as those used by many of the families in this study. However, for an autistic sample of toddlers, the novelty of the standardized materials provides one possible explanation for trending differences in child communication between contexts. While the types of toys in the standardized toy set were familiar to many of the children, they had no previous exposure to the specific toys included in the standardized toy set. The presence of these novel items may have attenuated autistic toddlers' social communication because they are often more focused on objects during play, less jointly engaged with play partners, and exhibit concomitant specific interests and repetitive behaviors (restricted and repetitive behaviors [RRBs]; Adamson et al., 2009). This emerging difference in child communication may also be due to the similarity between the interactional context with family-selected materials and the context in which caregivers learned to implement communication strategies; the standardized context may represent more distal intervention outcomes for caregivers and their children (Barokova & Tager-Flusberg, 2020).

Despite marked similarity between the materials of the standardized interactional context and those chosen by dyads in the family-selected context, there were several differences. These included more physical and social play, sensory play, and other daily routines observed in the family-selected context. The types of activities caregiver–child dyads chose during the family-selected interactional context (e.g., play, book reading, feeding, and grooming) are consistent with those identified in previous research comparing caregiver–infant interactions between naturalistic and structured contexts (Holme et al., 2022; Tamis-LeMonda et al., 2017). This indicates that measuring intervention outcomes during interactions with standardized toy kits alone may fail to capture outcomes during common contexts of caregiver–child interaction, activities that serve as key opportunities for increased engagement and facilitation of social communication skills. While standardization reduces extraneous variability in child communication, it overlooks dyadic interactions during many activities salient to families in their everyday routines, contexts in which caregivers are often coached to implement strategies to support their child's communication.

Limitations

The present study has two primary limitations: the potential impact of temporal effects and the small sample size. First, the order of interactional contexts was not counterbalanced in an effort to reduce child frustration during transitions. As such, temporal effects cannot be ruled out as an explanation for this emerging trend of differences in child communication between contexts because most families completed the standardized play context immediately following the family-selected interactional context. Therefore, this sample of autistic toddlers may have communicated somewhat less frequently in the standardized context given prolonged social demands of consecutive dyadic interactions and their existing social communication difficulties. Taken together, while the small median differences in child communication between interactional contexts was neither statistically nor clinically significant, more research is needed as this study's small sample size and noncounterbalanced conditions may limit our ability to detect an effect. In future research, 52 children and 63 caregivers would be needed in the sample to replicate the medium effects found for the differences in child and caregiver outcomes between contexts. We hope that findings from this study will guide additional methodologically rigorous research designed to investigate caregiver–child intervention outcomes across interactional contexts.

Second, the findings of the present study are limited by a small sample size with just two 10-min samples per dyad at a single timepoint. This precluded exploring potential moderating factors to determine for whom standardization of intervention outcome measures may most significantly impact. Future research should explore whether the novelty of contexts with standardized toy sets more significantly attenuates the communication of autistic toddlers exhibiting frequent and persistent RRBs compared to those with few, less intense RRBs. Furthermore, it is crucial to consider that, while play with traditional toys is common for many families, play contexts differ across families and cultures, including play partners, play materials, and the role of play materials as the focus of an interaction or a means of facilitating interaction (Nwokah et al., 2013; Rochanavibhata & Marian, 2022; Roopnarine, 2010; Tamis-LeMonda et al., 1992). The cultural appropriateness of outcome measures must also be considered. While the NDBI-Fi was chosen for this study as it is an outcome measure that assesses implementation of common NDBI strategies across interventions (Frost et al., 2020), NDBIs, and in turn NDBI outcome measures, are limited in representing the range of caregiver interactional styles, particularly in a culturally diverse sample (Guiberson & Ferris, 2019). The same is true of how child outcomes are commonly measured in intervention research, such as consideration of eye contact among other forms of communication, which vary across cultures (Golson et al., 2022). Future research is warranted to investigate the impact of these cross-cultural differences in play and communication on the design of interventions and measurement of intervention outcomes to ensure that these outcomes are not under- or misrepresenting the interactions of child–caregiver dyads, such as those who less frequently interact during toy play with materials resembling those in standardized toy sets. Additionally, if using a standardized toy set, researchers should seek feedback from families on the representativeness of the provided materials given the context of their day-to-day interactions. Equitable measurement of intervention outcomes is one small yet consequential step toward developing and implementing equitable evidence-based clinical practice.

Implications

Based on this exploratory research, researchers should consider measuring intervention outcomes beyond play contexts with standardized materials, given that toy play with materials such as those in standardized toy sets captures outcomes in just one, highly specific activity among numerous others that similarly facilitate high-quality, salient child–caregiver interaction. Play, regardless of whether the toys are standardized, is a context of peak social interaction between many children and their caregivers (Tamis-LeMonda et al., 2017). This study demonstrates that traditional toy play is valued by caregivers, enjoyable for dyads, and frequently chosen by caregivers as a context for interacting with their child. On the one hand, standardization of materials during caregiver–child play can be appropriate and useful for measuring outcomes for several clinical and research purposes, such as reducing variability across individuals and characterizing more distal intervention outcomes. On the other hand, it appears that many caregivers and children are already engaging in play that is highly similar to that of standardized play contexts, possibly influenced by their perception of what providers or researchers commonly expect for caregiver–child interaction. Using family-selected materials would be more cost-effective, naturalistic, and equitable. Additionally, this practice would align with the family-centered model of early intervention that emphasizes the importance of using materials already present in the child's natural environment (Lee et al., 2022). As such, clinicians and researchers evaluating the efficacy of caregiver-mediated communication interventions should consider measuring outcomes during play with families' chosen toys and materials, in addition to measuring outcomes during families' other daily routines that they value as opportunities for caregiver–child interaction.

Conclusions

In conclusion, investigations of intervention outcomes across interactional contexts have significant research and clinical implications as these efficacy trials directly inform the implementation of evidence-based practice. Additional research is necessary given this study's limitations in experimental design, particularly as families were not randomized to counterbalanced conditions. These preliminary findings contribute to the measurement of intervention efficacy to suggest that (a) measures should represent contexts that are meaningful to each child's participation in activities of daily living and that (b) outcome measurements should equitably detect change for all families participating in research. It is necessary to carefully consider a more expansive approach to standardization in which intervention outcomes are measured in ecologically valid contexts, which meaningfully, accurately, and equitably capture caregiver and child functional outcomes, and the translation of interventions to families' everyday routines.

Data Availability Statement

Data contributing to the findings of this study are available by the corresponding author upon request.

Funding Statement

This study was funded by a National Institute on Deafness and Other Communication Disorders Grant 1R01DC014709 (awarded to Megan Y. Roberts at Northwestern University).