Does Treatment Fidelity of the Early Start Denver Model Impact Skill Acquisition in Young Children with Autism?

Abstract

There is increasing evidence supporting the effectiveness of the Early Start Denver Model (ESDM) for children on the autism spectrum. However, substantial variability in response to the ESDM has been reported across participants. We examined the plausible yet untested hypothesis that variations in the fidelity level of therapists delivering the intervention contribute to variability in children’s response to the ESDM. Videotaped sessions (n = 40) of toddlers on the autism spectrum who received the ESDM from trained therapists were coded to obtain measures of therapist fidelity and children’s learning in response to the therapists’ instruction. Variations in overall fidelity, along with variations in most items included in the ESDM fidelity checklist, contributed to the children’s learning response during the sessions.

Early interventions for children on the autism spectrum¹ can result in substantial improvements across multiple domains (Landa, 2018; Vivanti et al., 2020). For example, the Early Start Denver Model (ESDM; Rogers & Dawson, 2010), a naturalistic developmental behavioral intervention for children on the autism spectrum ages 12–48 months, has resulted in improvements in the areas of language/communication and cognitive functioning across several intervention trials (Fuller et al., 2020).

However, substantial variability in intervention response across trials, participants and implementation contexts has been reported in response to the ESDM (Fuller et al., 2020; Vivanti et al., 2016, 2019; Waddington et al., 2016) as well as other evidence-based interventions (Howlin et al., 2009; Vivanti et al., 2020). Previous research has documented a range of factors that might contribute to variability in intervention outcomes, including child pre-treatment cognitive, social, and language skills (Schreibman et al., 2009; Shine & Perry, 2010; Vivanti et al., 2014). More recently, research has started to examine the role of intervention fidelity in relation to variability in intervention response. The construct of fidelity refers to the degree to which prescribed elements of an intervention are delivered as intended (Schoenwald et al., 2011). A growing body of literature has demonstrated that the outcomes of various healthcare and educational programs are impacted by implementation fidelity (Durlak & DuPre, 2008; Miller & Rollnick, 2014; Reed & Codding, 2014; Tucker & Blythe, 2008). In the Autism Spectrum Disorder (ASD) early intervention field, a small but revealing body of literature has similarly started to document that variations in treatment fidelity substantially affect intervention outcomes. Långh et al. (2020) found that treatment fidelity predicted clinical outcomes in response to Early Intensive Behavioral Intervention. Another study which focused on a teacher-implemented program called Learning Experiences Alternative Program for Preschoolers and Parents (LEAP) reported that children whose teachers had higher implementation fidelity made greater improvements in response to the program (Strain & Bovey, 2011). Additionally, Pellecchia et al. (2015) evaluated the association of fidelity and treatment outcomes in response to the Strategies for Teaching based on Autism Research (STAR) program, a comprehensive treatment model that includes several evidence-based practices. Interestingly, it was reported that in classrooms where program fidelity was either high or low, participants experienced larger intervention gains—suggesting a non-linear link between fidelity and outcomes.

An association between treatment fidelity and child outcomes has also been documented in three studies evaluating parent-mediated programs based on the ESDM. Rogers et al. (2018) reported a positive relationship between degree of improvement in parent fidelity and increases in skills that were targeted in the intervention. Similarly, Vismara et al. (2013) reported that children whose parents showed higher levels of treatment fidelity experienced more language gains. Finally, Waddington et al. (2020) reported that the correct use of parent-implemented ESDM (P-ESDM) techniques by caregivers was associated with both child engagement and expressive language.

Although this body of literature provides preliminary evidence on the role of fidelity in the treatment response to the ESDM and other interventions, there are several gaps and limitations requiring further research effort. First, previous research in this area has mostly focused on distal measures of intervention outcomes, such as changes in standardized tests of cognitive or adaptive functioning following intervention delivery. However, more proximal measures of response to instruction (e.g., the child’s success or failure to acquire a skill being directly targeted), might provide a more actionable insight on the impact of fidelity on intervention response. For example, examining whether children imitate in response to a teaching technique delivered with varying degrees of fidelity across different teaching episodes provides a more direct indication of the importance of delivering that technique at fidelity compared to changes in IQ following 1 year of intervention. Consistent with this notion, the above-mentioned Rogers et al. study (2018) showed that fidelity of intervention delivery was linked to proximal but not distal measures of intervention outcomes.

Secondly, although most interventions for ASD include multiple components which might contribute differently to intervention response (Stahmer et al., 2019), existing research has focused on overall fidelity across the components of intervention packages. An exception to this trend is the recent Waddington et al. (2020) study, which documented a different contribution from different fidelity items to the intervention response of children receiving P-ESDM. Therefore, it is plausible that variations in the degree of fidelity to which the different components of an intervention are delivered contribute differently to child’s response. Examining the distinct contribution of fidelity for each component of an intervention on proximal response to instruction might illuminate the role and relevance of the different components (“ingredients”) included in intervention packages. This knowledge, in turn, can critically inform therapist training procedures (e.g., focus on specific techniques whose integrity is critical to facilitate child’s learning), mechanisms of intervention action, and adaptations of interventions (Vivanti & Stahmer, 2018).

Third, most research to date has focused on the impact of intervention fidelity on child outcomes in the context of parent-mediated early intervention approaches. In particular, the impact of fidelity on responsivity to the ESDM, to our knowledge, has been examined exclusively in the context of caregiver-delivered programs. Extending this body of knowledge to the examination of the impact of intervention fidelity in therapist-implemented interventions is of critical importance, as many community programs rely on the delivery of intervention practices by community practitioners whose fidelity of implementation is likely to be variable (Suhrheinrich et al., 2013).

To address these gaps in the literature, the current study examined whether variations in treatment fidelity have an impact on proximal measures of learning response for children receiving therapist-delivered ESDM. As the ESDM includes multiple components, which are operationalized through a 13-item fidelity checklist (Rogers & Dawson, 2010), we examined the impact of overall fidelity as well as the contribution of each fidelity item in the ESDM to children’s learning response to the intervention, with the goal of identifying components of the intervention that might play a more relevant role in producing benefits for the intervention recipients. Accordingly, two research questions were tested in the study;

1. Does overall therapist intervention fidelity contribute to child’ proximal intervention response to ESDM?

2. What is the specific contribution of each individual fidelity items to child’s intervention response?

We predicted that variations in fidelity scores of trained therapists delivering the ESDM would be associated to variations in child’s response to the intervention (i.e., the degree to which they engage in the targeted behavior in response to the instructional cues delivered by the therapists during the session). In other words, we hypothesized that children whose therapists were implementing the ESDM more rigorously, as reflected in higher fidelity scores, would be more likely to engage in the target behaviors. No specific hypothesis was advanced with regards to the contribution of individual fidelity items to child’s intervention response as previous literature does not offer specific predictions on the relative weight of different ESDM techniques in promoting learning for young children on the autism spectrum.

Methods

Overall Procedure

The two research questions in the current study were tested using 40 video-recorded ESDM intervention sessions. The sessions were conducted as part of two larger studies involving the implementation of the ESDM in a therapist-delivered 1:1 format. Both projects were approved by the Drexel University Institutional Review Board (approval IDs 1512004088A018 and 1607004653). All parents consented to the treatment study and to recording of the treatment sessions. The sessions involved children on the autism spectrum receiving the ESDM individually from trained therapists (see ‘Participant’ section) without additional specialists involved. Children completed approximately 2 weekly 2-h sessions of ESDM over the course of 1 year either in the child’s home or at the A.J. Drexel Autism Institute clinic.

The ESDM intervention was conducted according to the manualized procedures detailed in Rogers and Dawson (2010), which include a set of treatment practices and a comprehensive curriculum covering multiple developmental domains. Key concepts emphasized in the ESDM include the use of “joint activity routines”, i.e., play activities and daily routines that build upon the child’s initiative and preferences, during which clinicians scaffold the child’s acquisition of new behaviors using a variety of behavioral and developmental strategies (Rogers et al., 2017). Treatment fidelity in the ESDM is operationalized and monitored using the ESDM fidelity checklist (see ‘Measures’ section).

Videorecorded ESDM sessions were coded to obtain measures of (a) therapist fidelity (using the ESDM fidelity checklist); and (b) the child’s response to the therapists’ instructional cues during the session (for example, showing a pointing gesture in response to the instructional cue “which one do you want?”, or saying “go!” in response to the therapist saying “ready, set” and then pausing).

Participants

Participants included; (a) 16 children on the autism spectrum ages 20–39 months, who had a confirmed ASD diagnosis and who received ESDM as part of two larger studies (see ‘Overall Procedure’ section) and; (b) 15 therapists at various stages of ESDM experience.

Children

Children included in the study had either been previously diagnosed by community providers or were referred to the A.J. Drexel Autism Institute for an ASD diagnostic evaluation by their primary care providers. Regardless of age of diagnosis, the Autism Diagnostic Observation Schedule—2nd Edition (ADOS-2; Lord et al., 2012) was administered for diagnostic confirmation. Additionally, all children were administered the Mullen Scale of Early Learning (MSEL; Mullen, 1995), and Vineland Adaptive Behavior Scale—3rd Edition (VABS-3; Sparrow et al., 2016) to characterize the sample in terms of cognitive and adaptive functioning. See Table 1 for demographics and baseline scores of children participating in the study.

Table 1.

Characteristics of toddlers in the sample

Demographic variable	Mean (SD)
Child age in video (months)	28.93 (4.70)
ADOS-2
Overall CSS	7.19 (1.83)
Social affect CSS	7.06 (2.29)
Restricted/repetitive behaviors CSS	7.44 (1.59)
MSEL ELC	70.15 (16.74)
VABS-3 ABC	71.31 (13.58)
	n (%)
Child sex
Male	11 (68.75)
Female	5 (31.25)
Child race
White	10 (62.5)
Asian	1 (6.25)
Black/African American	4 (25)
Multiracial	1 (6.25)
Child ethnicity
Hispanic/Latinx	3 (18.75)
Non-Hispanic or Latinx	13 (81.25)
Maternal education
High school diploma/GED	1 (6.25)
Some college	1 (6.25)
College degree	11 (68.75)
Post-college degree	3 (18.75)

SD standard deviation; ADOS-2 Autism Diagnostic Observation Schedule, Second Edition; ADOS-2 CSS Calibrated Severity Score; MSEL ELC Mullen Scales of Early Learning Composite Standard Score; VABS-3 ABC Vineland Adaptive Behavior Scale—3rd Edition Adaptive Behavior Composite Standard Score

Therapists

Therapists were trained by a certified ESDM trainer according to standard ESDM training and fidelity procedures (Rogers & Dawson, 2010). Consistent with the ESDM generalist model of intervention delivery, the training covered interdisciplinary knowledge required to implement a comprehensive curriculum, with the same therapist covering multiple developmental domains. After reaching fidelity, the therapists regularly implemented ESDM with participating children. All of the participating therapists received training on the ESDM for the first time as part of the project. All videos of therapists delivering the ESDM were taken after therapists completed the training process. As part of the study, therapists received ongoing individualized supervision from a senior clinician who held a BCBA. Additionally, all the therapists and senior clinicians involved in the study met as a team twice per week with a certified ESDM trainer to discuss and address ESDM implementation issues. Fidelity of implementation was monitored approximately every month. In response to fidelity scores below 80%, feedback was provided on the specific areas in need for improvement (corresponding to the items in the fidelity checklist with scores of 1, 2 or 3; see ‘Measures’ section). Fourteen of the fifteen participating therapists were female. Therapists’ highest educational attainment ranged from some college (i.e., BA or BS degree in progress) to PhD degrees. The amount of experience working with children on the autism spectrum in the therapist team ranged from less than 1 year to more than 10 years.

Measures

Child Participant Characteristics

Mullen Scale of Early Learning

The MSEL is a well normed measure of cognitive development for infants and young children ages 0–68 months. Children are assessed across several developmental domains including: fine motor, visual reception, expressive language, and receptive language. Scores from the visual reception, fine motor, receptive language, and expressive language domains are combined to create the Early Learning Composite (ELC) score which provides an estimate of the child’s overall cognitive level. The MSEL were administered by trained clinicians supervised by licensed psychologist. Pretreatment ELC scores were used to characterize participants’ cognitive level.

Vineland Adaptive Behavior Scale—3rd Edition

The VABS-3 is a semi-structured parent interview that measures adaptive functioning across four domains including: communication, daily living skills, socialization, and motor functioning. Scores from each domain are combined to create the overall Adaptive Behavior Composite score. The VABS-3 were administered by trained clinicians supervised by licensed psychologist. Pretreatment composite scores were analyzed to characterize participants’ adaptive functioning.

Autism Diagnostic Observation Schedule, 2nd Edition

The ADOS-2 is a semi-structured, standardized assessment used for the diagnosis of ASD in individuals ages 12 months and up. The ADOS-2 assesses ASD symptomatology across multiple domains of functioning including communication, reciprocal social interaction abilities, play, and restricted and repetitive behaviors and yields diagnostic classifications or concern classification scores. The toddler module or Module 1 of the ADOS-2 was administered by a research reliable clinician to confirm the ASD diagnosis and to characterize participants in terms of symptom severity using calibrated severity scores (CSS).

Therapist Fidelity of ESDM Implementation

Fidelity of ESDM implementation was assessed from videos of intervention sessions using the ESDM fidelity checklist. Following the fidelity procedures indicated in the ESDM manual (Rogers & Dawson, 2010), fidelity of ESDM implementation was measured using a Likert scale rating system that covers 13 distinct intervention techniques. These include management of child attention, quality of behavioral teaching (i.e., the ability to organize teaching episodes with clear antecedent-behavior-consequence sequences and provide frequent teaching episodes), the accurate use of instructional techniques such as fading, shaping and prompting, adult ability to modulate child affect and arousal, management of unwanted behaviors using positive approaches, use of turn-taking, quality of dyadic engagement, optimizing child motivation for participation in activity, use of positive affect, sensitivity and responsivity to child communications, targeting multiple and varied communicative functions (e.g. requesting, commenting, protesting, labelling, greeting), appropriateness of adult language for child’s language level, use of joint activity routines and smooth transitions between activities that maximize child interest and engagement.

Although the original Likert scale involves a 1–5 score range (reflecting poor to proficient use of each technique), in the present study we used an adapted scoring system with a 1–4 range, whereby the scores of 4 and 5 (representing “competent display of teaching behavior with no major flaws,” and “best possible display of teaching behavior,” respectively) were collapsed into a single category reflecting proficient display of the treatment technique. This adaptation was introduced based on pilot work, which indicated improved inter-rater agreement using a 1–4 range. Additionally, an overall fidelity score was derived from the sum of the 13 individual item scores.

Child Learning Response

Child learning response was operationalized as the child’s emission of the targeted behaviors in response to the first four consecutive teaching episodes delivered by the therapist in each video. A teaching episode was operationalized as the therapist’s delivery of an instructional cue. For example, in an activity where the child and therapist were playing with blocks, the therapist asking “which color block do you want?” was operationalized as a teaching episode during which the therapist was targeting “indicating a choice”. Targeted behaviors were informed by an individualized assessment of the child’s strengths and needs conducted at baseline.

A score of 1 was given for each episode in which the child emitted the targeted behavior in response to the therapist’s instruction within approximately 5 s. For example, if a therapist instructed a child to clap their hands and the child responded by clapping their hands independently, the learning response was coded as “1”. Conversely, a score of “0” was given if the child did not emit the targeted behavior or did not do so independently (i.e., received a partial or full physical prompt by the therapist). Scores were summed to derive a total learning response score for each video.

Coding Procedures

Forty videotaped sessions of 16 children receiving ESDM intervention sessions from 15 trained interventionists were randomly selected from all available video-recorded ESDM sessions (100+). Online folders that included randomly ordered video-files of all sessions received by each participant were examined by the first coder to determine whether they met inclusion criteria. Inclusion criteria included a clear transition in and out of the activity, and the camera field of view capturing both the therapist and child. The first 40 consecutive videos that met these criteria were included in the study. For each video, the first activity that included at least four teaching episodes involving four distinct treatment targets was selected. The segments selected from the videotaped sessions ranged from 4 to 19 min (M = 9.16, SD = 4.22) in duration. Each child and each therapist were featured in one to seven videos. The same child-therapist pair was featured in no more than three videos.

Coding of the selected videos was preceded by a training stage, during which three experimenters were trained to code a set of “practice” videos of ESDM sessions until at least 80% agreement was achieved. Subsequently, raters were randomly assigned batches of videos to code independently. Thirty percent (n = 12) of the videos were randomly selected and then double coded for reliability by one of the raters who was blind to the study aims and hypotheses. All raters were unaware of which videos were being coded for reliability and were blind to the scores on other measures used in the study. Interrater agreement was calculated using intraclass correlation (ICC) which is considered to be the most appropriate approach for data that include multiple coders and observations (Koo & Li, 2016). ICC across raters and measures using absolute agreement revealed good average reliability (mean ICC = .78).

Data Analysis

We first conducted a preliminary examination of the possible influence of the settings in which sessions were conducted (home versus clinic settings) and the amount of intervention received prior to the involvement in the ESDM program on the variables of interest. Subsequently, our research questions were tested using correlational and regression analyses. Normality was assessed for the main study variables using z-skewness and kurtosis indices, with a critical value set at ± 3.29 (Tabachnick et al., 2007). To test our research questions using the most parsimonious regression model while still accounting for potentially confounding variables, we first estimated bivariate associations using Pearson Product Moment Correlation coefficients between child learning response, overall fidelity, individual fidelity items, and other factors potentially associated with child learning response. These included VABS-3 Adaptive Behavior Composite (ABC), MSEL ELC, ADOS-2 Overall CSS, ADOS-2 Restricted and Repetitive Behavior CSS, and ADOS-2 Social Affect CSS, maternal education, child age, and child time in treatment. Additionally, as all therapists received training on ESDM or similar naturalistic developmental behavioral interventions for the first time as part of the project, therapist time in the study was included as a variable reflecting their amount of experience and practice with ESDM. Next, variables with a bivariate association with child learning response that showed a statistical significance level of p ≤ .20 were entered together into multiple linear regression models predicting child learning response. This approach is based on indication that variables with a statistical significance of p ≤ .20 can confound the association between independent and dependent variables. Additionally, as most of the 16 children and 15 therapists in the study were featured in more than one video, child and therapist were included as factors in the regression analyses. Subsequently, we tested whether overall fidelity as well as each fidelity item contributed to child learning response using separate regression analyses. Benjamini–Hochberg’s false discovery rate (FDR; 1995) procedure was used to account for the possibility of Type I error resulting from the use of multiple statistical tests. The FDR was set at 5% for these analyses. A power analysis conducted using gpower indicated that our sample size of n = 40 was sufficient to detect medium to large effect sizes at 80% power and probability level of 0.05.

Results

Preliminary analyses to examine the possible confounding factor of setting (clinic versus home) showed no differences between sessions conducted at home and those conducted in clinic settings in terms of child learning response, t (40) = − .37, p = .70, or therapist fidelity, t (40) = − .58, p = .56.

Correlational analyses to test the hypothesized association between fidelity and child learning response showed that, as illustrated in Table 2, overall fidelity was significantly and positively correlated with child learning response, r = .66, p < .01. Additional correlational analyses at the individual fidelity item level revealed that ten of the thirteen fidelity items were significantly correlated with child learning response, r range = .35–.72, p range = < .001–.03. Ability to modulate child affect and arousal, r = .20, p = .21, management of unwanted behaviors, r = .08, p = .63, and provision of multiple and varied communicative opportunities, r = .18, p = .28, were the only fidelity items not significantly associated with child learning response. Child learning response was also significantly correlated with child’s age, r = .35, p = .03, but not with child’s time in treatment, maternal education, VABS-3 ABC, MSEL ELC, ADOS-2 scores, or therapist time in the project (Table 3).

Table 2.

Summary of descriptive statistics (n = 40) and intercorrelations between child learning response, overall fidelity, and the 13 individual fidelity items

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
1. Child learning response	–
2. Overall fidelity	.66**	–
3. Management of child attention	.72**	.78**	–
4. Quality of behavioral teaching	.53**	.78**	.81**	–
5. Instructional technique application	.55**	79**	.63**	79**	–
6. Modulating child affect and arousal	.2	.28	.40*	.16	.05	–
7. Management of unwanted behaviors	.08	.22	.18	.34*	.21	−.04	–
8. Quality of dyadic engagement	.47**	.59**	.42**	.32*	.25	.16	.09	–
9. Optimization of child motivation	.60**	.80**	.68**	.62**	.59**	.14	.13	.51**	–
10. Adult use of positive affect	.38*	.52**	.34*	.2	.34*	.23	−.16	.40*	.46**	–
11. Sensitivity and responsivity to child cues	.40*	.53**	.37*	.2	.22	.28	−.08	.12	.35*	.39*	–
12. Multiple varied communicative opp.	.17	.49**	.22	.29	.32*	.06	−.02	.38*	.3	.09	.25	–
13. Appropriate adult language	.45**	.76**	.43**	.50**	.63**	.05	−.09	.36*	.53**	.42**	.49**	.37*	–
14. Joint activity and elaboration structure	.44**	.80**	.53**	.53**	.57**	.19	.36*	.52**	.59**	.47**	.36*	.42**	.50**	–
15. Transition between activities	.35*	.75**	.42**	.45**	.56**	.04		.39*	.61**	.33*	.48**	.32*	.73**	.54**	–
Mean	3.55	43.23	3.30	2.92	3.22	3.80	3.83	2.75	2.97	3.83	3.43	2.82	3.35	3.43	3.57
Standard deviation	1.20	5.73	0.61	0.83	0.89	0.46	0.50	0.71	0.70	0.39	0.68	0.59	0.86	0.71	0.81
Range	1–5	29–51	2–4	1–4	1–4	2–4	2–4	1–4	1–4	3–4	2–4	2–4	1–4	2–4	1–4

^*p<.05

^**p<.01

Table 3.

Summary of descriptive statistics and intercorrelations between child learning response and factors potentially associated with child learning response

	1	2	3	4	5	6	7	8	9	10
1. Child learning response	–
2. Child age (months)	.35*	–
3. Child time in treatment (days)	.15	.63**	–
4. Maternal education	−.16	−.28	−.14	–
5. Therapist time in training (days)	.17	.14	.38*	.18	–
6. VABS-3 ABC	.06	−.39*	−.34*	−.002	70	–
7. MSEL ELC	.06	−.18	−.43**	.15	.09	.61**	–
8. ADOS-2 overall CSS	.07	−.06	.32*	−.36*	.03	−.22	−.14	–
9. ADOS-2 RRB CSS	.14	.14	.07	−.28	−.30	.07	.04	.13	–
10. ADOS-2 social affect CSS	−.06	−.09	.36*	−.18	.10	−.39*	−.32*	.90**	−.25	–
Mean	3.55	28.93	143.03	3.78	173.53	72.33	70.42	7.38	7.35	7.38
Standard deviation	1.20	4.70	107.08	1.05	99.66	12.52	16.64	1.146	1.46	2.10
Range	1–5	20–39	8–354	1–5	17–345	45–90	49–102	3–10	5–9	4–10

ABC VABS-3 Adaptive Behavior Composite Score; CSS Calibrated Severity Score; ELC MSEL Early Learning Composite Standard Score; RRB restrictive and repetitive behaviors

^*p < .05

^**p < .01

Subsequently, a series of separate multiple hierarchical linear regression models were conducted to test the degree to which overall fidelity and the ten fidelity items that were significantly associated with child learning response contributed to variance on child learning response (see Table 4). The first regression analysis included child learning response as the dependent variable, and overall fidelity as the independent variable. In order to account for other variables potentially contributing to the dependent variable, child age was entered in Step 1 due to its significant correlation with child learning. Child and therapist were also entered as factors in Step 1, given that most children and therapists were present across multiple videos that were included for analyses. As illustrated in Table 4, overall fidelity significantly contributed to child learning response above and beyond the variance explained by the factors included in Step 1, F(4,35) = 7.54, p = < .001, R² = .40, explaining an additional 30% of the variance.

Table 4.

Summary of hierarchical regression analyses for fidelity variables predicting child learning response

Predictor variables	B	SE B	β	95% CI
Overall fidelity	.14	.03	.65**	0.07–0.20
Management of child attention	1.31	.22	.66**	0.86–1.76
ABC format (quality of behavioral teaching)	.67	.21	.46**	0.24–1.09
Instructional techniques application	.65	.22	.49**	0.20–1.10
Quality of dyadic engagement	.72	.24	.43**	0.25–1.20
Optimize child motivation to participate in activity	.94	.25	.55**	0.43–1.44
Use of positive affect	.89	.50	.29	−0.12–0.70
Sensitivity and responsivity to child comm. cue	.56	.27	.31*	.002–1.11
Appropriate adult language for child language level	.49	.24	.35*	−0.01–0.98
Joint activity structure and elaboration	.56	.28	.34*	0.004–1.12
Transition between activities	.34	.26	.23†	−0.17–0.85

Each row represents separate regression analyses. All regression analyses adjusted for chronological age, as well as specific child and therapist featured in the teaching episode. The complete regression model for each fidelity item is reported in the supplementary materials. All significant results remained significant after false discovery rate analyses

ABC antecedent-behavior-consequence

^*p < .05

^**p < .01

^†p = .05

We then conducted similar regression analyses for each fidelity item. Following Saville (1990), we conducted a separate analysis for each item rather than adopting a family-wise approach, given the exploratory nature of the analyses, and given that each item in the fidelity checklist is designed to capture a distinct construct. Most fidelity items significantly contributed to variance in child learning response above and beyond the variance explained by the factors in Step 1, including management of child attention, F(4,35) = 12.28, p < .001, R² = .58, quality of behavioral teaching, F(4,35) = 4.76, p = .004, R² = .35, instructional techniques application, F(4,35) = 4.38, p = .006, R² = .33, quality of dyadic engagement, F(4,35) = 4.59, p = .004, R² = .34, optimization of child motivation to participate in the activity, F(4,35) = 6.02, p = .001, R² = .41, sensitivity and responsivity to child communicative cues, F(4,35) = 3.01, p = .03, R² = .17, appropriate adult language for child language level, F(4,35) = 2.97, p = .03, R² = .25, and joint activity structure and elaboration, F(4,35) = 3.02, p = .03, R² = .26. There was a non-significant trend for adult use of positive affect, F(4,35) = 2.70, p = .05, R² = .24, and transition between activities, F(4,35) = 2.32, p = .08, R² = .21. All significant results remained significant after the false discovery rate analyses were performed.

Discussion

In this study we examined the impact of treatment fidelity on learning response for children on the autism spectrum receiving the ESDM. We found that the degree to which therapists implemented the intervention as prescribed affected children’s skill acquisition in response to the ESDM. These results align with previous literature suggesting that children whose interventionists implement interventions more rigorously are more likely to learn the skills being targeted (Långh et al., 2020; Strain & Bovey, 2011). Additionally, our results align with and extend previous research showing a link between treatment fidelity and intervention response for children receiving ESDM in the context of parent-mediated programs (Rogers et al., 2018; Vismara et al., 2013; Waddington et al., 2020).

We also found that the variability in the use of most ESDM techniques, as indexed by the items in the ESDM fidelity checklist, contributed to variability in child’s learning response, suggesting that most components of the ESDM are important for successful response. The finding that both techniques originating from behavioral literature (e.g., use of the ABC format) and developmental literature (e.g., use of joint activity routines) were associated with children’s acquisition of the targeted skills suggest that behavioral and developmental components of the ESDM provide complementary contributions to learning response, rather than being incompatible or redundant (Vivanti & Stahmer, 2020).

Three techniques covered in the ESDM fidelity checklist, namely the ability to modulate child affect and arousal, the management of unwanted behaviors, and the provision of multiple and varied communicative opportunities, did not appear to contribute to child’s learning response based on zero-order correlational analyses. Two additional techniques—the use of positive affect and the ability to facilitate a smooth transition between activities—were significantly associated with child’s learning response in the correlational analyses but narrowly missed the significance threshold in the regression analyses that controlled for the contribution of other factors to children’s learning response (p = .05 and p = .08 respectively). Future research should clarify whether these techniques are truly not related to the acquisition of targeted behaviors or whether the absence of associations between these aspects of ESDM and children’s response reflects methodological factors, such as a low frequency of these techniques in the segments of the videos that were coded. Notably, however, these results are substantially consistent with the findings from a recent study by Waddington et al. (2020). In the study, which focused on the impact of fidelity in a P-ESDM program, the techniques that were least related to child’s intervention response included the ability to modulate child affect and arousal, the management of unwanted behaviors, the provision of multiple and varied communicative opportunities and the facilitation of transitions between activities. The almost complete overlap in ESDM techniques found to be less related to intervention response across independent samples and formats (therapist-implemented format in the current study versus parent-implemented format in Waddington et al., 2020) points to the importance of further research on the relevance of these specific techniques for ESDM intervention effectiveness.

Despite the alignment of our results with existing literature, several limitations in the current study should be noted. The video clips used for coding in this study were brief (ranging from 4 to 19 min) as only the first four teaching episodes were used. This might have resulted in not capturing the full picture of the therapists’ ability to administer all the intervention components at fidelity. For example, accurately assessing the therapists’ ability to manage unwanted behaviors can be difficult to capture within a short period of time given the relatively low frequency of unwanted behaviors in children receiving intervention in this age range. Furthermore, limited variability in fidelity scores for treatment techniques that are easy to master (i.e., use of positive affect) complicates the evaluation of their individual contributions to fidelity. Additionally, it is possible that we were underpowered to detect the associations between these fidelity items and child’s learning response and, had we had a larger sample and thus greater power to detect small-medium effect sizes (i.e., of the magnitude of contribution fidelity items made to child’s learning response), the contribution of all fidelity items may have been found to be significant in our analyses. However, the relatively small sample size in our study yields a conservative bias, if any, and most predicted associations were indeed detected, and they were in the direction that was hypothesized based on our predictions. Further research should test the degree to which the ability to modulate child affect and arousal, the management of unwanted behaviors, and the provision of multiple and varied communicative opportunities contribute to child’s learning in response to intervention using a fully powered research design.

Additional limitations included the possibility of factors affecting child learning besides fidelity which were not taken into account for the current analyses, such as child sleep or temperament, or learning experiences that might have occurred outside of the ESDM sessions. It is also possible that child behavior during the session affected the results of this study. For instance, if a child is non-compliant, both therapist fidelity and learning responses might be negatively impacted. However, this potential issue was mitigated by including the child as a factor in the regression analyses. Additionally, none of the videos had scores that would indicate the presence of significant unwanted behaviors.

Furthermore, fidelity items were highly correlated with one another, likely because therapists involved in the study learned all of the ESDM techniques at the same time, so therapists who were low in fidelity in certain items tended to be low in fidelity across other items, making it more difficult to disentangle the distinct weight of each component of the intervention on child learning response.

Additionally, the present study focused on children’s response to the instructional cues received during intervention sessions. Although this is arguably a strength, as discussed in the introduction, future research should complement this information with data on distal outcomes such as changes in standardized assessment scores from pre to post treatment.

Finally, it would be valuable for future research to examine whether specific behavioral targets are related to more variability in treatment fidelity or child learning response. In light of the limitations discussed above, our results provide a preliminary indication in support of our prediction, rather than conclusive evidence.

In conclusion, we documented for the first time that therapists’ fidelity plays an important role in children’s ability to learn new skills in response to the ESDM. The alignment of these preliminary results with literature involving other treatments and other formats of ESDM delivery highlights that efforts for training therapists and monitoring fidelity in ASD early interventions are critical, especially in community settings where resources for training are more limited to compared to research settings.