Caregiver Behavioral Changes Mediate the Effects of NDBIs: Combining Evidence from Three RCTs

Abstract

Naturalistic Developmental Behavioral Interventions (NDBIs) target developmentally appropriate skills in young children with autism spectrum disorders (ASD) using behavioral techniques in naturalistic interactions. NDBIs demonstrate strong empirical support and frequently utilize caregiver training of intervention strategies. However, our understanding of the mechanisms of change linked to NDBIs remains limited. Based on retrospective, secondary data analyses from three previously conducted randomized controlled trials (RCTs) of caregiver-mediated NDBIs, the current study examined the direct effect of intervention on caregiver NDBI strategy use and how caregiver changes mediate intervention effects on children’s social communication. A total of 419 videos from 229 dyads consisting of autistic children (M=32.5 months/ages 1–5 years) and caregivers were included in analyses using the masked ratings of the Measure of NDBI Strategy Implementation-Caregiver Changes (MONSI-CC) and Brief Observation of Social Communication Change (BOSCC). Using longitudinal mediation analyses, we examined the mediation effect of caregiver behavioral changes on children’s outcomes. Results demonstrated a significant intervention effect on MONSI-CC scores and of MONSI-CC scores on BOSCC scores, but no overall total effect of intervention on BOSCC scores. This study demonstrates the mediating role of caregiver behavior on the child intervention response, a first step toward better understanding underlying mechanisms in NDBIs.

Lay Abstract

Naturalistic Developmental Behavioral Interventions (NDBIs) use behavioral techniques in naturalistic interactions and settings. NDBIs also teach caregivers to use strategies to enhance social communication in their children with ASD. NDBIs are effective in supporting young autistic children but how NDBIs work (“underlying mechanisms”) is not known yet. Therefore, by applying new instruments to videos already collected from the three previously conducted randomized controlled trials (RCTs) of caregiver-mediated NDBIs, we examined how caregivers change in their strategy use over the course of intervention and how that relates to changes in social communication in their autistic children. A total of 419 videos from 229 dyads consisting of autistic children (M=32.5 months/ages 1–5 years) and caregivers were included in analyses using two measures, one for the caregiver behavioral changes (the Measure of NDBI Strategy Implementation-Caregiver Changes [MONSI-CC]), and another for the children’s social communication (the Brief Observation of Social Communication Change [BOSCC]). We found that NDBIs were effective in leading to increased mastery of caregiver NDBI strategy use, which then affected the changes in child’s social communication skills. However, we found no direct effect of treatment for child social communication skills. This study highlights the mediating role of caregiver behavior on the child intervention response, supporting the importance of involving caregivers as partners in intervention for young autistic children.

Naturalistic Developmental Behavioral Interventions (NDBIs) include a series of interventions based on similar principles that teach behavioral strategies to target developmentally appropriate skills in young children with autism spectrum disorders (ASD) in naturalistic settings (Schreibman et al., 2015). Recent meta-analyses for NDBIs demonstrate promising effects for child expressive language, play skills, social communication, and cognitive development (Crank et al., 2021; Sandbank et al., 2023). Although NDBIs, such as Early Start Denver Model (ESDM) and Joint Attention Symbolic Play Engagement and Regulation (JASPER), have been independently created and validated; they share common teaching elements, such as natural reinforcement, child-initiated teaching episodes, modeling, prompting, and imitating, as well as intervention targets, such as social communication, engagement, and play skills). Combining evidence from different studies and comparing intervention effects across those studies have not been feasible due to lack of a standardized outcome measures that can be applied uniformly across trials.

Another core component of many NDBIs is caregiver training or coaching. Interventions that focus on these approaches, rather than intervention directly delivered by a professional, are commonly referred to as caregiver-mediated. Caregiver involvement in ASD intervention, especially with very young children, has long been recommended by experts and federal policy because it significantly enhances generalization and maintenance of skills attained (“Individuals with Disabilities Education Act [IDEA]” 2004; National Research Council, 2001; Pickles et al., 2016; Schreibman et al., 2015; Wetherby et al., 2014). Caregiver-mediated interventions build upon a transactional model of development, in which the behaviors of the caregiver reciprocally shape the behaviors of the child (Sone et al., 2021). In caregiver-mediated NDBIs, providers teach caregiver strategies to increase child social communication abilities which can lead to positive downstream developmental gains. Despite strong theoretical justification and support for teaching caregivers specific intervention techniques, understanding of mechanisms or mediators of child change in caregiver-mediated NDBIs remains relatively limited partly due to small sample sizes in previously conducted RCTs. Furthermore, no studies have examined changes in caregiver gains by combining evidence across various NDBIs.

Recently, Trembath and colleagues (2019) conducted a systematic review of factors that influence the outcomes of caregiver-mediated interventions for young autistic children. Approximately 36% of included studies explored potential mediating or moderating factors, with significant variability noted regarding identified caregiver and child factors. One of the common caregiver factors in NDBI studies was caregiver implementation of NDBI strategies, mostly measured by study-specific fidelity ratings. However, this approach limits generalizability across studies because it focuses on adherence to the full intervention protocol versus common or overlapping elements, a necessary step required to understand active ingredients or mediators of change across various NDBIs (Frost et al., 2020). Given that a few studies have shown promising results on the potential mediating role of caregiver NDBI strategies (e.g., “mirrored pacing,” “synchrony”) on child social communication outcomes (Gulsrud et al., 2016; Pickles et al., 2015), combining evidence from multiple studies to examine the effects of caregiver behavioral changes as a mediator of child intervention outcomes is essential to improve our understanding of intervention mechanisms.

Measurement tools that allow for the examination of common strategies across studies of different NDBIs can help answer these imperative questions regarding overarching intervention mechanisms. The Measure of NDBI Strategy Implementation-Caregiver Changes (MONSI-CC) was recently developed and tested across various early autism intervention models (Vibert et al., 2020), showing promise to examine caregiver changes in three distinct yet similar NDBIs. Similarly, the Brief Observation of Social Communication Change (BOSCC) has been used to examine changes in broad social communication skills over the course of intervention in children with ASD across various NDBIs (Grzadzinski et al., 2016, Swain et al., in press). The current study quantitatively examines the impact of caregiver NDBI strategy use (via MONSI-CC) on child social communication abilities (via BOSCC) across a sample of young autistic children who participated in one of three randomized control trials (RCTs) of caregiver-mediated NDBIs. These data were not meant to compare effects of different models since the RCTs varied in targets, length, and setting. Rather, through secondary analyses of data obtained from previously published studies, we aimed to examine (1) whether caregivers receiving active intervention would show significantly larger increases in their NDBI caregiver strategy use compared to controls (both active and intervention as usual) and (2) whether changes in child behaviors would be mediated by the changes in caregiver behaviors.

Method

Participants

Participants consisted of caregiver and child dyads who were enrolled in one of the following studies: ESDM (original sample n=118; Rogers et al., 2019), ESI (original sample n=82; Wetherby et al., 2014), and JASPER (original sample n=112; Kasari et al., 2014). Participants came from multiple sites, including University of Michigan Autism and Communication Disorders Center (ESI, JASPER, ESDM), University of California, Davis (ESDM), University of Washington (ESDM), University of California Los Angeles (JASPER), Kennedy Kreiger (JASPER), and Florida State University (JASPER). The current study and data sharing were approved (IRB#1707018417). The comparison groups in each study were receiving group-based (compared to individual) playgroups and education sessions (JASPER and ESI), or community based, “as usual” interventions (ESDM; See Supplemental materials for more details). Informed consent was obtained from all families included in this study at each institute, and all data were shared after data sharing agreements were executed among institutes.

Out of participants from the above studies (n=312), 229 dyads were selected for the current study that met inclusion and exclusion criteria (See procedures below). Children were between the ages of 1 and 5 years, predominantly White (n=117, 51.1%) and demonstrated variability in regard to baseline language and nonverbal abilities. Caregivers ranged in age from 20.6 to 56.6 years, were predominantly female and college educated (50.2%). See Table 1 for additional details.

Table 1.

Baseline Demographic Information

	Intervention	Control	Total
	n (%)	n (%)	n (%)
Child Sex: Female	26 (21.8)	21 (19.1%)	47 (20.5%)
Child Race/Ethnicity**
Asian	6 (5.0%)	5 (4.5%)	11 (4.8%)
Biracial	8 (6.7%)	6 (5.5%)	14 (6.1%)
Black/African American	13 (10.9%)	21 (19.1%)	34 (14.8%)
Latinx	16 (13.4%)	13 (11.8%)	29 (12.7%)
Other	11 (9.2%)	8 (7.3%)	19(8.3%)
White/Caucasian	61 (51.3%)	56 (50.9%)	117 (51.1%)
Missing	4 (3.4%)	1 (0.9%)	5 (2.2%)
Language level
No words	42 (35.3%)	37 (33.6%)	80 (34.9%)
1–5 words	36 (30.3%)	38 (34.5%)	74 (32.3%)
Single words	19 (16.0%)	11 (10.0%)	30 (13.1%)
Occasional phrases	5 (4.2%)	11 (10.0%)	16 (7.0%)
2+ word utterances	12(10.0%)	8 (7.3%)	19 (8.3)
3+ word utterances	3 (2.5%)	3 (2.7%)	6 (2.6%)
Complex speech	2 (1.6%)	2 (1.8%)	4 (1.8%)
Interactant/Caregiver Sex: Female	104 (87.4%)	97 (88.2%)	201 (87.8%)
Interactant/Caregiver Education
< Bachelor’s degree	62 (52.1%)	53 (48.2%)	115 (50.2%)
Missing	7 (5.9%)	6 (5.4%)	13 (5.7%)
	M (SD)	M (SD)	M (SD)
Child Age (months)**	32.68 (12.14)	32.33 (12.40)	32.51 (12.24)
NVIQ**	76.94 (23.84)	76.74 (20.38)	76.84 (22.20)
CSS-SA	7.14 (1.98)	7.51 (1.96)	7.19 (1.86)
CSS-RRB*	7.54 (1.77)	7.51 (1.87)	7.32 (1.97)
BOSCC-SC	27.37 (7.64)	28.03 (7.66)	27.68 (7.63)
BOSCC-RRB	6.41 (3.04)	5.92 (2.54)	6.17 (2.81)
Interactant/Caregiver Age (years) a	35.23 (6.53)	34.18 (6.73)	34.73 (6.63)

Note. Significant differences

^**<0.01

^*<0.05 emerged from Chi-Square or ANOVA tests.

^aCaregiver age was missing for 18 caregivers in the intervention group and 19 in the control group.

Measures

Measure of NDBI Strategy Implementation – Caregiver Change (MONSI-CC; Vibert et al., 2020).

The MONSI-CC was recently developed and validated as a measure of changes in caregiver implementation of common NDBI strategies over the course of early autism intervention. The MONSI-CC is coded by assigning scores to each of two 5-minute video segments and averaging the codes from the two segments, parallel to the model of Brief Observation of Social Communication (BOSCC; Grzadzinski et al., 2016) procedures described below. Item scores on each of the strategies are then summed within five Domain Scores (i.e., Environmental Set-up, Child-Guided Interactions, Active Teaching and Learning, Opportunities for Engagement, and Natural Reinforcement and Scaffolding; see Vibert et al., 2020 for details about the items included in each domain). The Domain Scores are then summed to produce a single MONSI-CC Total Score. The MONSI-CC was validated based on a subset of the sample included in the current study, and showed promising psychometric properties, including high inter-rater reliability and validity (Vibert et al., 2020). Higher scores indicate more frequent and effective implementation of the strategies. The current study used MONSI-CC Total Scores for all analyses.

Brief Observation of Social Communication; Minimally Verbal version (BOSCC-MV; Grzadzinski et al., 2016).

The BOSCC consists of two domains (social communication [BOSCC-SC] and Restricted and Repetitive Behaviors [BOSCC-RRB] as well as an overall ASD total (BOSCC-ASD). Lower scores across domains indicate better social communication skills and fewer RRBs. Different versions of the BOSCC have been developed to account for various language levels (e.g., Minimally Verbal, Phrase Speech, Fluent; Byrne et al., 2022). The BOSCC-MV demonstrates high inter-rater reliability and validity and has been used internationally in several behavioral and pharmacological intervention studies (Byrne et al., 2022; Green et al., 2022; Grzadzinski et al., 2016; Kitzerow et al., 2016; Pijl et al., 2018). However, variability has been found in changes captured by the BOSCC, with one intervention study reporting no significant changes in BOSCC scores (Green et al,. 2022) although the same intervention model resulted in modest changes in social communication measured by the BOSCC in another population (Divan et al., 2018). Thus, the magnitude of changes in BOSCC scores may vary by individuals as well as intervention dosage, intensity and duration.

In addition, studies have found pronounced social communication symptom changes using the BOSCC but less change in RRBs (Grzadzinski et al., 2016; Kim et al., 2019). Thus, the current study focuses on BOSCC-SC domain from the Minimally Verbal version (i.e., eye contact, facial expressions, gestures, vocalizations, integration of vocal and non-vocal, social overtures, social responses, engagement). While it is possible that NDBIs may result in an indirect effect of repetitive behaviors (e.g., increasing flexibility in play routines, partners, and materials may decrease a child’s tendency for repetitive play), we focused on the primary (and overlapping) intervention outcomes and skills targeted across all three NDBIs analyzed in the current study (e.g., social communication, play skills, engagement), which is best captured by the social communication domain of the BOSCC.

Baseline Clinical Features

Baseline nonverbal IQ (NVIQ) was computed based on the Mullen Scales of Early Learning (MSEL) fine motor and visual reception domains (Bishop et al., 2011). Baseline autism symptom severity was measured by ADOS-2 calibrated severity scores (CSS) (Esler et al., 2015; Gotham et al., 2009). Additionally, baseline language levels were computed based on the observed spontaneous expressive language scores from the ADOS-2 using a standardized and validated metric (Mazurek et al., 2019). Values ranged from 1 (i.e., no words or word approximations) to 8 (uses sentences in a largely correct fashion, including some complex speech), with higher scores reflecting increasingly mature verbal language abilities. Caregivers reported the child’s biological sex, race/ethnicity, and age as well as their own sex, age, and education level at intervention entry.

Procedures

A total of 1166 videos for 285 caregiver and child dyads were made available for this study across the RCTs for all sites, excluding one of two sites from the ESI study which did not contribute videos (Figure 1). Of note, the ESI study did not show significant differences in outcome measures across sites (Wetherby et al., 2014). Because this paper focused on caregiver-mediated NDBIs and the RCT studies of ESDM, ESI and JASPER varied in their designs (e.g., caregiver- versus therapist-mediated), intensity, and length of enrollment, a subset of videos and dyads (i.e., 419 videos from 229 dyads) were selected for current study that met the following specific inclusion and exclusion criteria (Figure 1 Adapted CONSORT Diagram). First, videos recorded within approximately 3 months from the beginning of the caregiver-mediated intervention or active control period were included as this was the intervention duration for JASPER and ESDM. Although the ESI trial was 9-months in length, only videos from the first 3 months were included. Similarly, after 3 months of caregiver-mediated ESDM, families were provided 2 years of therapist implemented ESDM; these additional videos during the therapist-implemented ESDM sessions were excluded. For each dyad, a total of 4 videos maximum were included to account for baseline, month 1, month 2, and month 3. For any dyad with more than one observation within a time period, the first video was selected to reduce the extent of imbalance in the data (mean number of videos per dyad=1.8, SD=0.6, range=1–4; See Supplemental materials for additional details). Individual videos were excluded if any of the following occurred: child cried or had a pacifier in their mouth for more than half of the video segment; the main adult interactant in the video was not identified as a primary caregiver; the caregiver was not visible for more than 70% of the interaction. For various reasons (e.g., intervention duration extended to make up for missed sessions due to family and clinician availability, vacation, and illness to ensure the full dosage), the timing of the month 3 video varied with a mean of 15.29 weeks and standard deviation of 2.60 months from the intervention entry. All videos included in the month 3 timepoint were labeled as exit visits; none were from follow-up after treatment was completed. As the main focus was to examine the mediation effect of caregiver changes on the children who received intervention using the secondary data analytic approach, we excluded videos of children who dropped out of the intervention from the original studies. All interactions coded for each child were based on one main caregiver.

Figure 1. CONSORT for Sample Selection from the Original RCTs.

Note. ESDM = Early Start Devern model; RCT = Randomized Control Trial; JASPER = Joint Attention, Symbolic Play, Engagement, and Regulation; ESI = Early Social Interaction; CMM = Caregiver-Mediated Module; CEM = Caregiver Education Module.

*Only videos for one site in the ESI RCT were provided and included in the final data analysis.

**Participant videos provided to the study team included those who were randomized as well as those who were excluded during the assessment for eligibility process.

The BOSCC and MONSI-CC coders included 21 post-baccalaureate research assistants and post-doctoral research associates. All coders were supervised by a licensed clinical psychologist. The coders underwent 4–6 weeks of training and joint coding (about 10–20 videos) and established scoring reliability above 80% agreement with previously coded videos for BOSCC and MONSI-CC codes across three consecutive videos. The criteria for reliability were the following: (1) at least 80% reliability on all items within each segment (no more than 4 items could be different by more than 1 point); (2) 80% accuracy on each domain score within each segment; and (3) 90% agreement for total scores. Consensus codes were used for analysis when multiple codes were available for each video, but not for inter-rater reliability. To maintain and monitor fidelity, bi-weekly consensus coding sessions were conducted. Two-way random absolute intraclass correlation coefficient (ICCs) was computed to calculate inter-rater reliability. Inter-rater ICC was good for MONSI-CC Total at r=.79, 95% Confidence Interval=[.47-.92] and high for BOSCC-SC domain at r=.96, 95% Confidence Interval=[.88-.98].

Videos were de-identified and randomized to ensure coders remained masked to study, site, intervention allocation, and timepoint (i.e., “de-identification” did NOT mean video was blurred but the coders did not have any intervention related information). A single coder was not allowed to code both MONSI and BOSCC for the same child to minimize the bias; the information from the other measure was masked. Video length was standardized at 10 minutes; for those over 10 minutes in length, a 10-minute segment was extracted from the point when the examiner or study personnel left the room for the caregiver to start interacting with the child independently. The same segments were used for both the MONSI-CC and BOSCC coding. The videos were gathered during participation in the intervention trials, and most interactions were collected in a clinic setting (n=401, 96.4%) while the remaining observations were collected in participants’ homes. Although the exact set of toys used in the interactions varied across the three studies, they were all age-appropriate, engaging, and very similar, allowing for play levels ranging from cause-and-effect (less mature) to construction and pretend (more mature). All caregivers were given similar instructions (i.e., play and engage with your child as you normally would). Videos were rated for audio and video quality to maximize the validity of the coding on the scale from 1 (Poor) to 5 (Very Good). The mean quality for audio and visual codes were 3.68 (SD=.88) and 3.23 (SD=.86) respectively. Community members were not involved in the study.

Statistical Analyses

Our mediation analysis was intended to determine whether change in caregiver behavior (as measured by the MONSI-CC) as a result of the intervention resulted in observed changes in the child behavior (as measured by the BOSCC-SC). Intervention proceeded over an extended period and the number of assessments of caregiver and child behavior varied between families. We examined two structural equation models, one simpler, that used only the baseline and final eligible assessments (Figure 2, n=121), and another, more complex model for longitudinal mediation analyses of trials by Goldsmith et al (2018, Figure 3), that exploited all available assessments (n=229; 149 baseline, 48 month 1, 31 month 2, 189 month 3). In the latter model, we examined the progressive relationship of experience of intervention accumulating over time to progressive change in caregiver behavior (solid red lines) and the consequent incremental change in child behavior (solid black lines) and any other effects of intervention on the child (dashed red lines). Thus, the random intercept latent growth curve model examined intervention effects on the caregiver which could be mediated by cross-lagged effects. At baseline, each interaction of caregivers and children was allowed to vary with the study site, child’s age at first assessment, their ADOS CSS total score, their non-verbal IQ (NVIQ) and their level of language, which were entered into the model as fixed effects with random intercepts for each caregiver and child. Error variances for the repeated measurements of the MONSI-CC were set as equal and similarly for the BOSCC-SC scores. We first estimated models separately for the child BOSCC-SC and the caregivers’ MONSI-CC videos. Potential heterogeneity of intervention effects by study (ESDM, ESI, vs. JASPER studies) was assessed by Wald tests of added interaction terms. We then estimated a joint caregiver and child model in which the direct effect of change in caregiver behavior on change in the child behavior could be assessed. We fitted three models. In the first model, intervention effects on the caregiver were assumed to increase linearly with time as therapy progressed and all intervention effects on the child were forced to be mediated through changes in the care behavior. In the second model, a direct intervention effect on the child was allowed, an effect that also was assumed to increase linearly with time since start of intervention. A penultimate model removed the linearity assumption for the intervention effects. A final sensitivity analysis examined how robust the results were to correlated measurement error between contemporaneous MONSI-CC and BOSCC-SC scores.

Figure 2. Simple Mediation Model.

Notes. BOSCC-SC: Brief Observation of Social Communication- Social Communication; MONSI-CC: Measure of NDBI Strategy Implementation – Caregiver Change; a path (solid red line): effect of intervention on caregiver NDBI strategy implementation; b path (solid black line): the consequent change in child behavior; c’ path (dashed red line): any other effects of intervention on child.

Figure 3. Full Longitudinal Mediation Model.

Notes. ADOS CSS: Autism Diagnostic Observation Schedule calibrated severity scores; BOSCC-SC: Brief Observation of Social Communication- Social Communication; MONSI-CC: Measure of NDBI Strategy Implementation – Caregiver Change; NVIQ: nonverbal IQ. Lines represent the following; a1–3 paths (solid red line): progressive relationship of experience of intervention accumulating over time to progressive change in caregiver behavior; b1–3 paths (solid black lines): the consequent incremental change in child behavior; c’1–3 paths (dashed red lines): any other effects of intervention on the child.

All models were estimated in Stata v16.0 (StataCorp, 2019) using the gsem command (Rabe-Hesketh et al., 2004). Most of the imbalance in data availability arose from missing data, where different studies had different timepoints of assessment (see Supplemental materials for more details). Including dummy variables for study among the covariates and using maximum likelihood estimation allowed account to be taken of both the imbalance by design and attrition related missingness associated with baseline and any later observed interaction assessments and covariates. Direct and indirect effects were estimated using the product of coefficients method with confidence interval estimated by empirical bootstrap for the simple model and the delta-method for the more complex model (bootstrap was not practical due to constraints of computation and model convergence reliability).

Results

Mediation based on the Subset of Sample with Pre and Post Intervention Data

Table 2 shows descriptive statistics for the dyads with baseline (pre-intervention) and 3-month endpoint (post-intervention) data (n=121) by intervention group. There was no significant intervention group difference in the baseline MONSI-CC and BOSCC-SC scores (t=0.7, p=.332 and t=−1.66, p=.100 respectively). There was a clear increase in MONSI-CC scores (increased mastery in caregiver NDBI strategy implementation) from pre- to post-intervention in the intervention group, although the decrease in the BOSCC-SC scores (reduction in child social communication symptoms) was more modest. Simple regression analysis of the complete baseline-endpoint data pairs for the intervention effect covarying for baseline and study (i.e. ESDM, ESI vs. JASPER) gave estimates of 6.42 (95% CI [2.08, 10.76]; p=.004) for the MONSI-CC scores and −0.61 (95% CI [−2.93, 1.71]; p=.605) for the BOSCC-SC. The simple mediation model of Figure 1 (paths using traditional a, b and c’ labels) fitted by maximum likelihood SEM estimated a significant (p=.017) indirect effect of intervention through the MONSI-CC (a x b) on the endpoint BOSCC-SC of −1.39 (95% CI [−2.53, −0.24]). The residual direct effect of intervention on the BOSCC-SC though not significant (c’ of 0.870, 95%CI [1.34, 3.08], p=.440) was in the opposite direction. At baseline, MONSI-CC and BOSCC-SC scores were correlated −0.39 (p = .001) in the controls but only 0.05 (p = .680) among the intervention group. At endpoint, the correlation was −0.438 (p < .001) in the controls and −0.317 (p=.001) for the intervention group. As both BOSCC-SC and MONSI-CC scores were changing over time, and the correlation became stronger for the intervention group but not for the comparison group, these results suggest that there might have been an increase in the strength of the synchrony of caregiver and child.

Table 2.

MONSI-CC and BOSCC Scores at Pre and Post Intervention Data Points and Change Scores (Post – Pre Intervention)

	n Observations	Mean (SD)	Min	Max
Intervention Group
MONSI-CC Pre	80	59.44 (11.92)	34	87
MONSI-CC Post	100	71.61 (11.77)	41	90
MONSI-CC Change	66	13.00 (15.14)	−26	49
BOSCC-SC Pre	80	30.23 (6.81)	10	40
BOSCC-SC Post	100	26.62 (7.88)	9	39
BOSCC-SC Change	66	−3.30 (6.47)	−18	18
Control Group
MONSI-CC Pre	69	62.74 (12.35)	27	91
MONSI-CC Post	90	65.43 (12.61)	26	88
MONSI-CC Change	55	4.59 (13.72)	−29	35
BOSCC-SC Pre	69	29.05 (8.00)	4	40
BOSCC-SC Post	90	27.95 (7.90)	8	40
BOSCC-SC Change	55	−2.66 (7.28)	−19	14

Note. MONSI-CC = Measure of NDBI Strategy Implementation-Caregiver Change; BOSCC = Brief Observation of Social Communication Change.

Mediation based on All Available Data (Across the 4 Observed Assessment Points)

We next conducted analyses using all available repeated observations. Table 3 shows descriptive statistics for the whole study sample (n=229) by intervention models included in the analysis over the four assessment points. There was variability in the number of observations available due to differing assessment timepoints by the study (ESDM, ESI vs. JASPER).

Table 3.

MONSI-CC and BOSCC Scores at All Available Data Points

	n Observations	Mean (SD)	Min	Max
Intervention Group
MONSI-CC at Time 0	80	59.44(11.92)	34	87
MONSI-CC at Time 1	30	62.67(14.28)	37	88
MONSI-CC at Time 2	14	61.87(14.27)	40	83
MONSI-CC at Time 3	100	71.59(12.07)	37	90
BOSCC-SC at Time 0	80	30.23(6.81)	10	40
BOSCC-SC at Time 1	30	27.50(7.88)	12	39
BOSCC-SC at Time 2	14	24.61(8.78)	6	39
BOSCC-SC at Time 3	100	26.50(7.86)	9	39
Control Group
MONSI-CC at Time 0	69	62.74(12.35)	27	91
MONSI-CC at Time 1	18	60.78(12.66)	34	15
MONSI-CC at Time 2	18	64.36(11.15)	40	81
MONSI-CC at Time 3	90	65.43(12.61)	26	88
BOSCC-SC at Time 0	69	29.05(8.00)	4	40
BOSCC-SC at Time 1	18	30.22(6.87)	15	39
BOSCC-SC at Time 2	18	26.67(7.77)	12	39
BOSCC-SC at Time 3	90	27.88(8.00)	8	40

Note. MONSI-CC = Measure of NDBI Strategy Implementation-Caregiver Change; BOSCC = Brief Observation of Social Communication Change

Figure 4 shows spaghetti plots for the raw caregiver and child assessment data for the whole sample. The graphs suggest an increase in MONSI-CC caregiver scores over the course of intervention, and a decrease in the BOSCC-SC scores, with these changes being larger in the group receiving intervention. The model -predicted time-paths were obtained from growth curve models with non-linear time trend but linear time by intervention interactions. Fitted separately to each caregiver (MONSI-CC) and child score (BOSCC-SC), the caregiver MONSI-CC scores for the intervention group increased significantly more than controls (p=.047), while the greater decrease under intervention of the BOSCC-SC scores was non-significant (p=.194). Interactions testing intervention effect differences by intervention model (ESDM vs. ESI vs. JASPER) were not significant for the MONSI-CC or BOSCC-SC (2df p=.630 and .969 respectively). Subsequent analyses assumed intervention effects common across study.

Figure 4. Raw and Model-based Caregiver MONSI-CC and Child BOSCC-SC Scores.

Notes. BOSCC: Brief Observation of Social Communication- Social Communication; MONSI: Measure of NDBI Strategy Implementation – Caregiver Change

Figure 3 shows the mediation model fitted to make use of all the MONSI-CC and BOSCC-SC scores available. The model allowed caregiver and child random intercepts to be correlated at baseline (r=−0.49), for intervention to influence caregiver behavior (solid red paths) and for the change in caregiver interaction to effect change in child interaction (solid black lines). The intervention effect on the MONSI-CC was assumed to increase linearly with the duration of the intervention (i.e., a2=2 times a1 and a3=3 times a1), and each incremental change in the MONSI-CC was assumed to have the same effect on the BOSCC-SC (b1=b2=b3). Though caregivers were the proximal target of caregiver-mediated intervention components, some direct effect of intervention on the child was also allowed (red dashed lines), and this was also assumed to increase linearly with duration of intervention. Shown in Table 4, and as when analyzed separately, the direct effect of intervention on caregiver MONSI-CC (a1) was significant (p<.001) and the direct effect of intervention on child BOSCC (c’1) was not (p=.561). However, the mediation effect of caregiver MONSI-CC scores on child BOSCC-SC scores was highly significant (p<.001). Together, these estimates gave an indirect effect of each month of intervention on the child of 1.99 × −.12 = −0.23 (95%CI [−0.41,- 0.06], p=.010) and a total effect from the sum −0.23 + −0.15 = −0.38 (95% CI [−0.90, 0.14], p=.148). Additionally, the effects of the child’s initial autism symptom and language levels as well as NVIQ on the BOSCC-SC scores were significant. Specifically, lower autism symptom severity (ADOS CSS), more mature language skills (higher scores), and higher NVIQs were associated with lower BOSCC-SC scores (higher social communication skills), and those who received the ESDM and JASPER (compared to ESI) showed higher MONSI-CC (higher NDBI skill implementation) and higher BOSCC-SC scores (higher social communication symptom levels ).

Table 4.

Bivariate Growth Curve Model with Direct Intervention Effects on Change from Baseline (slope) for the MONSI-CC and the BOSCC and Indirect Intervention Effect on the BOSCC via the MONSI-CC (parameter labels in square parentheses correspond to Figure 3)

	Caregiver MONSI-CC Scores			Child BOSCC Social Communication Scores
	Estimate	95% CI	p-value	Estimate	95% CI	p-value
Coefficient
Intercept factor variance	1.0			1.0
Intercept loading	5.67	4.18 to7.16	<.001	3.41	2.71 to 4.12	<.001
Error variance	93.0			19.3
Intervention effect per month	1.99 [a1]	0.97 to 3.01	<.001	−0.15 [c’1]	−0.66 to 0.36	0.561
Caregiver on child				−0.12 [b1]	−0.18 to −0.05	<.001
Baseline/Intercept covariates
ADOS CSS	−0.12	−0.25 to 0.00	.056	0.13	0.02 to 0.24	.021
Language level	0.18	−0.05 to 0.41	0.127	−0.66	−0.90 to −0.42	<.001
Nonverbal IQ	0.27	−0.06 to 0.60	0.114	−0.50	−0.80 to −0.21	.001
Age	1.69	−2.33 to 5.71	0.410	−1.66	−5.09 to 1.77	0.343
ESDM Study	2.10	12.5 to 2.94	<.001	1.22	0.59 to 1.84	<.001
JASPER Study	1.38	0.36 to 2.40	.008	1.03	0.17 to 1.88	.019

Note. MONSI-CC = Measure of NDBI Strategy Implementation-Caregiver Change; BOSCC = Brief Observation of Social Communication Change.

^aMONSI-CC and BOSCC Baseline/Intercept correlation −0.49; [a1][b1][c’1] are parameters shown on Figure 3.

Robustness of Results to Possible Correlated Measurement Error in Caregiver and Child Ratings

Goldsmith et al (2018) point out that though randomization helps considerably in recovering an unbiased estimate of the intervention on the mediator, there remains the risk of the path from mediator to outcome being biased by the presence of omitted variables. In this instance, the fact that both caregiver and child behavior are obtained from the same segments of interaction means that ephemeral aspects of behavior of the dyadic partners that are correlated (and potentially the result of such omitted variables) could contribute to the correlation from which we are trying to recover the causal effect of caregiver on child. To account for this possibility, rarely considered in standard mediation models, we re-estimated the model of Figure 3, this time allowing for correlated errors for the four pairs of contemporaneous caregiver and child behavior measures. Although this did not give rise to a significant improvement in model fit (LR c2 p=.064) nor materially change any of the coefficient estimates for the covariates or factor loadings, the inclusion of the error correlation resulted in the estimated causal effect (b1) of MONSI-CC on BOSCC-SC reducing from −0.12 to −0.07 (p=.096), and the indirect effect of intervention on child BOSCC-SC (a1 × b1) being reduced by 40% to a non-significant −0.14 (p=.117). The total effect remained little changed at −0.36 per month (p=.179).

Additional model specification tests were conducted to allow the intervention (vs. control) to have a non-linear effect over time. This gave entirely non-significant improvements in model fit. Models of the form of Pickles et al (2015) that adjusted for measurement error in the mediator failed to converge, likely due to the extent of data imbalance. However, given the high inter-rater reliability for the BOSCC-SC and MONSI-CC reported in the past (Grzadzinski et al., 2016; Kim et al., 2019; Vibert et al., 2020), the models might not have been expected to alter estimates much.

Discussion

This study serves as a first step toward better understanding underlying mechanisms in caregiver-mediated NDBIs based on secondary analyses of data obtained from previously conducted RCTs. By combining evidence from studies of three different NDBIs, we found a significant effect of intervention on caregivers, demonstrating that caregivers assigned to the intervention condition showed increased mastery in NDBI strategy implementation than those in the active control group. Second, we found a significant indirect effect of intervention on child outcomes mediated by caregiver NDBI strategy use. Third, we did not find any overall total effect of intervention on child social communication outcomes. The results demonstrate that changes in caregiver implementation of NDBI strategy led to changes in child social communication, with no other significant paths that resulted in intervention-related change in child social communication. Only when models were re-estimated to test extreme scenarios with caregiver-child rating error correlations to account for all possible confounders, rarely considered in standard mediation models, did this become non-significant.

The residual direct effect of intervention on the BOSCC-SC indicated an unexpected increase in scores (less mature social communication) although the effect was nonsignificant. A caregiver-mediated intervention is built on the assumption that caregiver changes are key to changes in child behaviors, and thus, examining the changes in NDBI implementation in response to the caregiver-mediated NDBIs as a mediator of child outcomes was an essential step, even in the absence of expected, direct effect of intervention on child outcomes. Put another way, child improvement in social communication symptoms was not merely a result of being randomized to the intervention condition, but rather through the mechanism of caregiver mastery of NDBI skills. The results indeed support that targeting caregiver behaviors positively impacts child outcomes and can be accomplished with interventions varying in intensity and dosage, starting with a low-intensity (i.e., one to three hours per week) to high-intensity (i.e., ten to fifteen hours per week) model. Identifying high-impact, low-intensity models may be particularly important for caregivers supporting intervention programing for their autistic children who frequently report high levels of stress (Shepherd et al., 2018). In addition, increased correlations between the caregiver and child scores over time may suggest an increase in the strength of the dyadic synchrony for the intervention group as well. As such, caregivers successfully learning core NDBI strategies taught through each specific NDBI intervention package may be a key mechanism of child reduction in social communication symptoms over the course of caregiver-mediated NDBIs.

There are many strengths and limitations of the study. To our knowledge, this is one of the largest examinations of NDBI mechanisms as it relates to caregiver influences on child outcome. This study would not be possible without the willingness of trialist authors to pool data across studies. Although we focused on post-hoc harmonization of data, we hope the current study serves as a call for future studies to standardize data elements a priori in an effort to build more extensive datasets to better explore mechanisms of caregiver-mediated NDBIs. We used a masked objective coding system targeting core features of ASD, a broad range of social communication behaviors that were common across trials. This study was limited by use of a single outcome measure (BOSCC) due to no other overlapping outcome measures being used across the three trials. Future studies need to examine whether specific caregiver NDBI strategies can also impact more specific child outcomes, such as caregiver-child engagement and joint attention. Intervention effects may be also moderated by family and child characteristics. Furthermore, because the current study only evaluated three different NDBI packages, it will be important to understand whether outcomes extend to other specific NDBIs; although a recent study has revealed similar results with a pilot RCT (Settanni et al., 2023). For better data harmonization among RCTs of varying lengths and dosage, we selected the data from the first three months of caregiver-mediated intervention, the extent of time believed to be required to enact such change. Thus, we do not know whether the effects observed in the current study extend beyond the first three months.

Imbalance and timing variation in the assessments was inevitable because we retrospectively combined a subset of data to apply a uniform coding system across studies with variations in protocols, selection criteria and the relatively low priority given to quality of video of the original studies for coding. This resulted in a large sample size reduction from those originally randomized. Thus, this report can only be considered as an analysis of the intervention mechanism within families where a higher level of study participation was achieved. These challenges were somewhat mitigated by the analyses in the current study which used all available data under the ignorable missing data properties of maximum-likelihood SEM. When the mediation effects were tested while controlling for correlated errors for the four pairs of contemporaneous caregiver and child behavior measures, the evidence was not robust. On the assumption that a causal effect on the child arises from exposure to an improved caregiver interaction over an extended period, we should exclude occasion-specific synchrony between mediator and outcome when calculating the mediated effect. On one hand, permitting correlated errors accounts for both measurement errors and transient influences on the caregiver child interaction videos, such as temporary mood or health states. On the other hand, using masked behavioral coding techniques should have minimized these effects. This cautious approach comes at the cost of statistical power, leading to broader confidence intervals in any mediated effect estimate. For comparison with other studies, the results from the simpler model, where a significant mediated effect is observed, are most appropriate for interpretation. Meanwhile, the results from the model accounting for correlated errors serve as a cautionary note, underscoring the need for larger consortia to achieve greater confidence in our findings. Furthermore, although almost every child was minimally verbal (or preverbal), 11 to 14% of each study sample consisted of children with phrase or fluent speech, which may have limited the ability of the BOSCC to detect meaningful change in this small subset of children. At the time of behavioral coding, the BOSCC version for phrase speech and verbally fluent young children (PSYF) was still under development. The BOSCC-MV was also used for all cases as scores across different versions of the BOSCC are not standardized. However, future studies will benefit from implementing the different versions of the BOSCC when sample sizes are large enough for different language levels for separate analyses. In addition, although clinician fidelity for all data included in the current study was high (all above 75%), we did not have direct access to the clinician fidelity data. Thus, future studies can also examine the association between clinician fidelity and caregiver skill changes and their impact on child outcomes. Finally, the BOSCC and MONSI-CC have shown high reliability and validity in past studies (Grzadzinski et al., 2016, Kim et al., 2019), including convergent validity with other clinical outcome measures, and thus changes we observed in the past and current studies have important clinical implications as mentioned above. However, we also acknowledge that there are individual differences in the response rates that are clinically meaningful and the family’s perception of these changes. Further studies could examine individual variability in behavioral changes and solicit direct feedback from families about their impression of change, and how these relate to the information gained from more objective measures such as the BOSCC and MONSI-CC.

Conclusion

This is one of the largest secondary data analyses based on multiple previously conducted RCTs examining the mechanism of NDBIs for young autistic children. Specifically, the results support the mediating role of caregiver mastery in strategy implementation over the course of NDBIs, shedding light on how caregiver-mediated interventions can empower social communication skills in autistic children. Supporting caregivers to increase the use of intervention strategies in NDBI packages is thus an essential component to improve a broad range of social communication, the core need of young autistic children.