Mapping the active ingredients and mechanisms of change of a naturalistic developmental behavioral intervention using mixed methods

Abstract

Limited research has examined the active ingredients and mechanisms of change of naturalistic developmental behavioral interventions (NDBIs). The present study used an exploratory sequential mixed-methods design to develop a comprehensive Theory of Change of Project ImPACT, an empirically supported NDBI. We used qualitative data from interviews with intervention experts (n=10), community providers (n=22), and caregivers (n=12) to develop a comprehensive causal model of the intervention process. We then tested select paths of the causal model using path analyses with an archival dataset (n=92). The causal model described how developmental techniques aimed at supporting children’s attention and engagement lay the foundation for more adult-directed learning opportunities and subsequent child skill growth. However, hypothesized causal relationships were not supported by our quantitative analyses. In the future, this research can be used to develop and prioritize nuanced research questions related to the timing, optimization, and mechanistic process underlying NDBIs.

Interventions focused on supporting autistic children’s social communication development have emerged from both developmental and behavioral theoretical orientations. Current best practices have converged on a combination of strategies from both perspectives (Schreibman et al., 2015). This class of interventions, called naturalistic developmental behavioral interventions (NDBIs), has growing empirical support for improving social communication for young autistic children (Sandbank et al., 2020; Tiede & Walton, 2019). Despite their promise, little research has examined the active ingredients and mechanisms of change underlying NDBIs. Because NDBIs are complex interventions, there are a variety of potential mechanistic processes at play, with some components having empirical support in the literature. For example, evidence suggests that both adult responsiveness and growth in ‘pivotal skills’ mediate treatment outcomes in early interventions for social communication delays. Increased “mirrored pacing” and “parental synchrony,” both related to adult responsiveness, have been associated with treatment outcomes (Aldred et al., 2012; Gulsrud et al., 2016; Pickles et al., 2015). In addition, NDBIs have been shown to support expressive language gains via growth in child imitation and intentional communication (Yoder et al., 2021a), providing support for the idea that directly targeting early social communication skills has downstream effects on other developmental outcomes. Indeed, supporting social engagement may enable children to learn optimally from their environment; recent work found that caregiver language modeling was more strongly associated with children’s sentence diversity when caregivers’ NDBI strategy use increased (Clark-Whitney et al., 2022).

There are numerous barriers to understanding active ingredients and mechanisms underlying NDBIs. First, an emphasis on studying packaged programs as a whole largely neglects the complexity of multi-component interventions, obscuring key active ingredients as well as potential redundant, ineffective, or harmful intervention elements which should be trimmed from an intervention to optimize its effectiveness (Collins et al., 2005; Dimidjian & Hollon, 2010; Scriven, 1994). This approach also obscures similarities and differences among packaged NDBIs. Careful examination of component parts of NDBIs may help us build cumulative knowledge that applies across packaged NDBIs which include those components (Frost et al., 2020). In addition, NDBIs target a variety of developmental outcomes (e.g., social engagement, joint attention, expressive language) which are expected to occur on different timelines, such that early, context-dependent changes in social communication are thought to facilitate later development on more distal, generalized outcomes. Typical RCT designs may not be optimally timed to capture these developmental processes as they unfold. In sum, the complexity of NDBIs, which includes many intervention elements and outcomes, makes it difficult to understand the intervention process.

Mixed methods have numerous strengths for advancing complex interventions research, including their ability to examine multifaceted and complex phenomena, and the ability to integrate stakeholder perspectives which support practical, real-world benefits (Borglin, 2015; Craig et al., 2008; Curry & Nunez-Smith, 2015). As such, mixed methods are increasingly used in clinical, health services, and implementation research (Curry & Nunez-Smith, 2015), however, to our knowledge, have yet to be used to develop and test causal explanatory models of the component parts of complex interventions. Here, we illustrate how mixed methods can be used to take a theory-based perspective on evaluation, focusing on exploring intervention processes: how and why complex intervention components bring about change in outcomes.

The present study

We used the Theory of Change (ToC) framework, a pragmatic, theory-based evaluation framework (Weiss, 1995, 1997), to explore the active ingredients, mechanisms of change and associated short-, mid-, and long-term outcomes of Project ImPACT (Ingersoll & Dvortcsak, 2019). ToC is a pragmatic framework for theory-based evaluation which incorporates intervention components, short-term, mid-term, and long-term outcomes, as well as the rationale or mechanistic links that connect them. A key tenet of the framework is that interventions are “based on explicit or implicit theories about how and why the program will work” (Weiss, 1995, p. 66). Theory-based evaluations are meant to identify and test those theories and assumptions. Project ImPACT is an evidence-based NDBI for young children with autism or social communication delays (e.g. Ingersoll et al., 2016, 2017; Yoder et al., 2021b). Of note, Project ImPACT is a parent-mediated intervention which include multiple levels of intervention delivery, including a) clinician instruction and feedback on intervention strategies delivered to the caregiver, and b) caregiver implementation of intervention strategies with the child. Here, we present results focusing on caregiver implementation of the intervention with the child. Results from the broader study which include the clinician-parent coaching process will be presented elsewhere for the purpose of clarity and article length constraints. The aims of this study were to: 1) Develop a comprehensive Theory of Change of Project ImPACT using stakeholder perspectives on potential active ingredients and mechanisms of change, and 2) Provide proof-of-concept of the Theory of Change model using archival data from treatment trials of Project ImPACT.

Method

This project used an exploratory sequential mixed methods design (Curry & Nunez-Smith, 2015) with multiple phases, in which results from a qualitative analysis are expanded upon with quantitative analyses (Figure 1). Phase 1 (qualitative) involved conducting and analyzing semi-structured interviews from key stakeholders. Phase 2 (qualitative) focused on integrating the qualitative data to develop a plausible causal model. Phase 3 (quantitative) used archival data from clinical trials to test select hypothesized relationships modeled in Phase 2. This study was approved by the Human Research Protection Program at Michigan State University.

Figure 1.

Exploratory sequential mixed methods design overview.

Phases 1 and 2: Qualitative data collection and analysis

Recruitment and study sample

Our purposive sampling strategy targeted stakeholders with a range of experiences with the intervention, from developers/trainers to end-users and consumers. Interview participants were compensated $30 for their time. Recruitment focused on three key stakeholder groups: intervention experts (i.e., “trainer-consultants,” qualified to lead workshops and certify community providers; n=10), community providers (i.e., coaches certified to deliver Project ImPACT in their practice; n=21), and caregivers who participated in the intervention with certified providers (n=12). Demographics are available in Table 1. We emphasized recruitment from certified providers and their families to increase the likelihood that our qualitative data described a high-fidelity implementation of Project ImPACT (i.e. representative of how the program is meant to be delivered, and likely to include all of the intended treatment elements). The coach certification process involves reading the manual, completing an online tutorial, attending a Project ImPACT Introductory Training workshop, receiving group or individual consultation from a certified trainer consultant, and meeting criteria for fidelity of implementation from taped sessions. Community providers were purposively sampled to include a variety of disciplines (Applied behavior analysis: 23%; Psychology: 23%; Speech-language pathology: 32%; General education: 14%, Special education: 41%, Early childhood education: 27%; Other: 14%). Providers reported working a variety of service settings, including early intervention programs (59%), hospital/medical centers (14%), public schools (14%), university-based clinics (14%), university academic appointments (14%), private practice (5%), and specialty clinics (5%)¹. They had a range of 1–15 years of experience with Project ImPACT (mean = 3.5), with 45% reporting they had used Project ImPACT with 1–5 clients, 14% reporting 6–10 clients, and 41% reporting 11+ clients.

Table 1.

Qualitative sample demographics

	Expert		Provider		Caregiver		Child
	n	%	n	%	n	%	n	%
Gender
Male	0	0%	1	5%	2	17%	8	73%
Female	10	100%	21	95%	10	83%	3	27%
Race
White or Caucasian, non-Hispanic/Latinx	8	80%	16	73%	9	75%	6	55%
Black or African American	0	0%	2	9%	0	0%	0	0%
Asian	0	0%	2	9%	1	8%	1	9%
Multiracial/multiethnic	1	10%	1	5%	0	0%	2	18%
Hispanic or Latinx	1	10%	1	5%	2	17%	2	18%
Level of education
Some college/specialized training	0	0%	0	0%	3	25%
Associate’s degree	0	0%	0	0%	2	17%
Bachelor’s degree	0	0%	2	9%	3	25%
Master’s degree	6	60%	17	77%	4	33%
Doctoral degree	4	40%	3	14%	0	0%

Caregivers were eligible to participate if they received Project ImPACT from a certified coach within 1 year of the time of the interview. Caregivers were provided with information about the study by certified community providers and were invited to contact the research team directly to participate. Twelve caregivers (including 1 couple) participated in interviews. Four caregivers received in-person services using the individual model (i.e., one-on-one with a coach); 7 caregivers participated in the individual model via telehealth, and 1 caregiver participated in a group model, which began in person but transitioned to telehealth.

Interview procedure

We developed a semi-structured qualitative interview guide based on knowledge of Project ImPACT and the ToC framework (Kallio et al., 2016). While the complete interview protocol included questions on all facets of the ToC framework, here, we focus on presenting results related to parent-child level active ingredients, mechanisms, mid-term outcomes and long-term outcomes. All interviews were conducted and audio-recorded via Zoom.

Phase 1 analysis approach

We used the Framework Method, a codebook-based thematic analysis approach, to analyze the data (Gale et al., 2013; Ritchie & Lewis, 2003). Interviews were transcribed by trained undergraduate research assistants and subsequently verified by the lead author. The qualitative coding team, comprised of two research assistants and an advanced graduate student, familiarized themselves with the data through reading and re-reading of transcripts and began to identify codes (Gale et al., 2013). Coding was conducted with computer-assisted qualitative data analysis software (MAXQDA2020, 2020).

We used causation coding to identify sequences of causes, outcomes, and mediating variables (or mechanisms) that link causes to outcomes (Saldaña, 2013). Most ‘causes’ (i.e., Project ImPACT treatment elements) were developed deductively from the intervention manual and applied as structural codes, while mechanisms and outcomes were identified inductively from the content of the interviews. Because of the complexity of the data, causes, mechanisms, and outcomes were coded separately in the first coding cycle. A second cycle of coding was conducted during the final audits of transcripts at which point comments were created to describe links between cause(s), mechanism(s), and outcome(s) according to each participant. Examples of each code type can be found in Table 2.

Table 2.

Sample codes to illustrate types of codes and hierarchical coding system.

Codes	Code types
Child-directed intervention elements	Category used to organize codebook
Imitate the child*	Deductive structural code from manual
Playful obstruction*	Deductive structural code from manual
Child mechanisms and outcomes	Category used to organize codebook
Social engagement*	Inductive code
Novelty/surprise*	Inductive code
Downstream effects	Category used to organize codebook
Developmental cascade	Inductive code
Quality of life	Inductive code

Note.

^*Examples of codes used for causation coding, which include treatment elements, mechanisms, and outcomes.

We used a collaborative, team-based approach to coding (Hemmler et al., 2020),which began with consensus coding and frequent discussion to refine the codebook (including collapsing, splitting, and clarifying definitions of codes). Next, all coders independently coded 3 transcripts, and inter-coder consistency was evaluated (exact agreement of at least Kappa = .60), suggesting moderate working agreement of the codebook. The remaining transcripts were coded by consensus. Both undergraduate coders independently coded all transcripts, their codes were merged with each other, and then the lead author audited all transcripts and finalized the codes. The first 12 interviews coded were audited near the end of the coding process to account for the final codebook and coding conventions.

Trustworthiness and sample adequacy

Several strategies were used to increase the rigor of this work, including an audit trail and biweekly coding meetings . Our purposive sampling strategyincluded individuals with extensive expertise in the topic as well as the intended end-users of the intervention program, indicative of incorporating a range of perspectives about the intervention process (Mays & Pope, 2000). Our semi-structured interview guide supported attainment of a thick, rich description of intervention processes. Taken together, these contribute to the credibility, dependability, and transferability of the findings (Morse, 2015).

Several characteristics of this study increased our information power to address our research questions (Malterud et al., 2016), including our purposive sampling strategy, use of an established theoretical framework as a guide (i.e., Theory of Change) and the ability to draw on established theory to analyze and integrate results (e.g., behavioral or learning theory, social interactionist theory). The adequacy of our sample was reflected in the coding process; codes were consistently being added to the codebook early in the coding process and new codes were no longer being added for the last several interviews.

Phase 2 analysis approach

Phase 2 of the research was characterized by connected integration or “building” of a causal model to be examined using quantitative data in Phase 3. Qualitative data, including coded segments and researcher comments, were charted using spreadsheets in order to summarize the data by category and respondent type (Gale et al., 2013). This allowed the research team to examine potentially-meaningful variation in descriptions across stakeholder groups via triangulation (Farmer et al., 2006). The first author used the charted data to develop causal models for each intervention element, visualized in joint displays (see supplemental tables). The last author, a developer of the intervention with a deep level of knowledge and clinical experience, reviewed the data charts and interpretation to further refine the analysis.

Following the development of the causal models for individual intervention elements, we considered the relations among them by examining the different change processes side by side. We specifically looked for shared mechanisms and outcomes and intervention elements thought to work in similar ways in order to create an integrated, cohesive, and parsimonious ToC. In addition, we considered how the change processes related to relevant established psychological theories, which would provide further support for the model.

Phase 3: Quantitative data collection and analysis

Recruitment and study sample

We used the integrated theory of change model to identify select paths that could be tested quantitatively using available archival data from two studies (a pilot RCT and RCT) examining the efficacy of Project ImPACT delivered using an online telehealth platform (Ingersoll et al., 2016). The online platform included various materials such as a digital treatment manual, narrated slideshows with video examples, and a video library. In the RCT, participants were randomized to one of three groups: therapist-assisted (receiving twice-weekly telehealth coaching comprised of content review and practice with feedback), self-directed (receiving access to the Project ImPACT online platform without coaching), and resource support (receiving access to a general online resource library and monthly support phone calls). The pilot study included only the therapist-assisted and self-directed groups.

Participants included in this study were 92 children between the ages of 18 to 93 months (M=46.6, SD=17.0) and a primary caregiver who received either therapist-assisted intervention (n=36), self-directed intervention (n=37), or resource support (n=19). Children were included in the studies if they had a community diagnosis of autism spectrum disorder or suspected autism, confirmed by administration of the Autism Diagnostic Observation Schedule (ADOS-G or ADOS-2; Lord et al., 2000, 2012), and limited language skills (i.e., expressive language age equivalent of less than 4 years at study entry). Caregivers had to be proficient in English but could speak other languages in the home. Family demographics can be found in Table 3.

Table 3.

Demographics of quantitative study sample.

Primary Caregiver Demographics	n	%	Child Demographics	n	%
Gender			Caregiver-reported sex
Male	12	13	Male	68	26
Female	80	87	Female	24	74
Race			Race
White	52	57	White	67	73
Black or African American	4	4	Black or African American	7	8
Asian	6	7	Asian	6	7
More than one race	2	2	More than one race	11	12
American Indian/Alaska Native	1	1	Other race	1	1
Missing	27	29
Ethnicity			Ethnicity
Hispanic or Latino	5	5	Hispanic or Latino	7	8
Not Hispanic or Latino	60	65	Not Hispanic or Latino	58	63
Missing	27	29	Missing	27	29
Education Level
Some high school	1	1
High school graduate	6	7
Some college/specialized training	35	38
4-year college	24	26
Graduate degree	26	28
Marital Status
Married; living with partner	62	67
Single; divorced or separated	9	10
Single; living with partner	5	5
Single; never married	12	13

Phase 3 analysis approach

Following the completion of Phases 1 and 2, we selected specific active ingredients (predictors), mechanisms (mediators), and outcomes that could be examined using available quantitative data. Active ingredients were operationalized as the caregiver’s fidelity of implementation of individual intervention elements. We developed two mediation models which mapped onto select paths from joint displays illustrating the change process for individual intervention elements and the integrated ToC model of child change processes in Project ImPACT. Caregiver-child dyads in all groups were included in this analysis to ensure variability in our predictors. Analyses were conducted in Mplus version 8.6 (Muthén & Muthén, 2021) using full information maximum likelihood estimation and percentile bootstrap with 1,000 resamples. Detailed descriptions of each model along with the study measures used to quantify relevant constructs can be found in the Phase 3 section of the Results.

Results

Phases 1 and 2

Active ingredients, mechanisms of change, and outcomes

Respondents described the child-focused intervention process for Project ImPACT by describing the change process for each intervention element (see Table 4 for brief descriptions of each intervention element as described in the manual). A narrative summary of the change process for each intervention element is described below based on results from the qualitative analysis; joint displays which present causal diagrams alongside supporting quotations can be found in supplemental files (Table S1-S8). Respondent groups provided convergent or complementary descriptions of intervention processes, although multiple pathways or variability in children’s responses were sometimes reported (and are indicated in our diagrams and descriptions below).

Table 4.

Project ImPACT Intervention Elements.

Intervention Unit
Intervention element	Brief description
Focus on your child
Follow your child’s lead	Stay face to face with your child, join in a child-led activity, avoid directions, and respond to your child’s actions
Imitate your child	Imitate your child’s gestures, facial expressions, body movements, vocalizations, and play with toys and objects
Adjust your communication
Use animation	Be excited about the activity, exaggerate gestures, facial expressions, and vocal quality, adjust your animation to help your child stay regulated
Model and expand communication	Comment on what your child is seeing, hearing, and doing using simple language as well as gestures and visual cues, and expand on the child’s communication
Create opportunities
Playful obstruction	Use an anticipatory cue or phrase then playfully block the child’s activity, then respond to the child’s communication
Balanced turns	Use an anticipatory cue or phrase then take a turn and model a play action, then respond to the child’s communication and/or give the child a turn
Communicative temptations	Put items in sight but out of reach, control access or give small portions, do something silly, or use items requiring adult assistance, then respond to the child’s communication
Teach new skills
Prompts & Rewards  Using communication  Understanding  communication  Imitation  Play	When the child is motivated, use prompts to support the child in demonstrating a more complex social communication skills; Once the child demonstrates the skill, provide a natural, positive reward
Shape the interaction	Use Project ImPACT techniques together to keep your child engaged and learning; select strategies to emphasize based on your child’s motivation, mood, and the activity

Note. Intervention element descriptions adapted from the Project ImPACT Manual for Parents (Ingersoll & Dvortcsak, 2019).

Following the child’s lead increases child social attention and engagement within an interaction in a few ways. At first the novelty might get their attention, by giving children agency and validating their interests, which increases their motivation and engagement with an adult. This engaged interaction sets the stage for supporting a positive interaction and relationship, as well as the development of new social communication skills over time.

Imitating the child initially supports child social attention and engagement through novelty. Imitation also provides a sense of agency or validation for children which is highly motivating, and supports their continued attention to and engagement with the adult. Over time, this engaged interaction supports a positive interaction and relationship, as well as the development of new social communication skills, particularly in imitation and social engagement.

Animation, when adjusted to children’s sensory needs, supports social attention and engagement by creating a playful, novel, and fun interaction. Increased social attention and engagement supports a positive interaction quality and also helps children attend to linguistic input which can support receptive language learning.

Modeling and expanding communication supports expressive communication growth through two mechanistic pathways. Repetitive exposure to words leads to receptive communication growth via associative learning, while simplified speech increases child vocal imitation. Imitation and increased receptive vocabulary are both then thought to support expressive communication growth over time. Decreased question-asking was also described as supporting social engagement in interactions, which then impacts parent-child relationship quality over time.

Playful obstruction can gain a child’s attention. When motivated, children often initiate, giving the adult an opportunity to contingently reinforce their communication. Over time, children initiate more frequently. For some children, however, unclear expectations in the interaction can lead to frustration or confusion and disengagement from the interaction. When not perceived as effective, parents may stop using this technique.

At first, many children do not understand the back-and-forth nature of balanced turns and may become upset or disengaged. Eventually, adult expectations become clear through repeated exposure, and children become motivated to initiate communication or continue to engage in a back-and-forth interaction and begin to develop skills in turn-taking. Parents may stop using this technique with children if children become upset/frustrated in response.

Communicative temptations create a motivating situation for the child to initiate communication. Over time, these initiations are reinforced, and the child is more likely to initiate spontaneously.

When teaching new skills, prompts provide consistent, scaffolded cues that clarify adult expectations and help children demonstrate a new skill. Knowing that a reward is coming motivates children to perform a skill, and over time, contingent reinforcement supports increased spontaneous use of those skills.

Shape the interaction

“Shape the interaction,” the final lesson of Project ImPACT, emphasizes how to skillfully use and balance various individual intervention techniques with each other. Participants described how the ImPACT techniques work better in combination, such that their effects are difficult to disentangle: “So I think, you know, for them to be effective, it is using all of them, being able to follow the sequence of them together” [expert]. In particular, respondents described how later-taught techniques rely on earlier-taught, foundational techniques to be most effective, in alignment with the hierarchical organization of intervention techniques into a pyramid with three levels as described in the Project ImPACT manual (Figure 2). Although there was consensus that Project ImPACT techniques are most effective when used in synchrony, several experts and providers talked about how difficult it is for caregivers to learn so many techniques and balance them appropriately: “there’s a lot of information in Project ImPACT and even though everything builds on each other, […] I think sometimes parents get caught up in the specific strategy that you’re focusing on for that week” [provider].

Figure 2.

Adaptation of the Project ImPACT ‘pyramid’ visual showing how earlier-taught techniques support implementation of later-taught techniques.

Respondents also described how caregivers implement and balance the different ImPACT techniques varies across families and situations. Although a few providers described it as a passive process in which “certain concepts become more habitual” [provider], most respondents described it as a deliberate and intentional process, tailored to the child’s day-to-day needs: “sometimes when he was having a hard time regulating himself for the week, we would then change our strategy and go down the pyramid to sort of help him regulate himself” [caregiver].

Finally, although there were many common themes and consistency with which respondents described intervention process, the fact that different children respond in different ways such that certain techniques work better for certain children was a common thread across responses: “That there’s not one technique that works better for all children, but that there are some techniques that work better for this child and other techniques that work better for another child” [expert].

Downstream outcomes

We identified three primary themes pertaining to downstream outcomes for children. First, across intervention techniques, respondents alluded to social communication skill growth in various domains which unfold over time. Second, respondents in all groups described the intervention’s effect on the parent-child relationship and interaction quality: “just understanding the impact of responding to him any time that he makes a bid of attention towards us has really changed our relationship, and in really powerful ways” [caregiver]. Some respondents attributed this to caregivers’ “knowing how to communicate with their child […and] see the way they view the world a little bit” [provider], which decreases frustration in day-to-day interactions. Others attributed this to caregivers’ “ability to feel like they can meet their kids’ needs” [expert] which makes caregivers “more likely to interact with their child more [… and] more likely to have positive reciprocal interactions” [expert]. Both of these downstream outcomes contributed to child quality of life, which we conceptualized across three subthemes: social connections, confidence and independence, and inclusion.

Beyond facilitating more positive interactions within the parent-child dyad, caregivers also talked about facilitating social connections with others by sharing Project ImPACT techniques: “it’s helping me to teach others how to interact with her, like I’ve spent time talking [to] like my parents […] they can at least do that to build a connection with their granddaughter.”

Caregivers also noted changes to their child’s confidence and independence as they moved through Project ImPACT. One caregiver described it as, “giving her independence, and getting her to understand that she knows how to, that she can move her body and she can do things independently,” and another felt increased confidence emerged from an “understanding that we’re noticing him and he’s not like on his own.”

Finally, caregivers and providers also described increased inclusion in activities in school and the community. For example, one caregiver described that, before Project ImPACT, their child would not participate in circle time at daycare, whereas after the program, “now he sits in the circle, he participates.” Other caregivers whose children were not yet in preschool were hopeful that skills gained would support their child’s inclusion in school settings in the future.

Integrated model of child change processes

An integrated theory of change model of processes at the parent-child level of Project ImPACT can be found in Figure 3. The model is consistent with key theoretical foundations of NDBIs, including the related concepts of “pivotal skills” and “developmental cascades,” and the central role of adult responsiveness to children’s communication (see Discussion).

Figure 3.

Integrated model of child change processes in Project ImPACT.

Phase 3

Two quantitative models were identified pragmatically based on available data to test hypothesized paths developed in phases 1 and 2 of the present study. In other words, we selected a cause, mechanism, and outcome represented in the theory of change model for which we had an appropriate measure, taken at an appropriate time point. Respondents described that following the child’s lead and imitating the child both increased children’s social attention and engagement in the moment, which lead to increases in communication development over time (Supplemental Tables S1-S2). Model 1 examined whether caregiver fidelity for the Focus on your Child domain (Follow the child’s lead, Imitate the child) supported generalized child communication skills via increases in context-dependent social engagement and social attention. We expected that caregiver fidelity for the Focus on your Child domain would be associated with Time 3 child communication skills, existing as an indirect effect through Time 2 child social attention/engagement (covarying for baseline child communication and social attention/engagement). Respondents also described how Playful Obstruction and Communicative Temptations increased (and provided reinforcement for) child initiations in the moment, which supported communication skill growth over time (Supplemental Tables S5 and S7). Thus, Model 2 examined whether caregiver fidelity for Creating Opportunities (a combined rating for Playful Obstruction, Communicative Temptations, and Balanced Turns) supported generalized child communication skills via increases child initiations. We expected caregiver fidelity for Creating Opportunities would be associated with Time 3 child communication skills, existing as an indirect effect through Time 2 child initiating communication (covarying for baseline child communication and initiating communication).

Measures

Caregiver fidelity of implementation was measured using the Project ImPACT Fidelity Checklist. Caregivers are rated from 1–5 on a series of indicators reflecting their use of different components of the Project ImPACT intervention, with 5 being excellent implementation. Here, we used data from two indicators (Focus on your child, Create opportunities). Fidelity was coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and post-intervention in two contexts: free play, and a snack routine. Ratings were averaged across the two contexts to form an overall score for each of the two indicator scores. Reliability was calculated for 25% of the observations across time point and conditions using intra-class correlation (ICC=.87).

Child social attention/engagement was measured using a composite score created from a subset of items from the Brief Observation of Social Communication Change which best mapped on to our qualitative construct as described by participants (BOSCC; Grzadzinski et al., 2016). The following items were summed: (1) eye contact, (4) vocalizations directed to others, (6) frequency and function of social overtures, (7) frequency and quality of responses, and (8) engagement in play activities/interaction. The BOSCC was coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and follow-up in two contexts: free play, and a snack routine (Frost et al., 2019). Previous analyses using this dataset indicated good interrater reliability for social communication items in both play (ICC=.87) and snack routines (ICC=.93; Frost et al., 2019). Ratings were averaged across the two contexts to form an overall social attention/engagement score. It should be noted that a higher score on the BOSCC is indicative of social communication impairment; for ease of interpretation, we have changed the sign of our coefficients in the statistical models reported below so that path models can be read intuitively.

Generalized child communication was measured using the Vineland Adaptive Behavior Scales (Sparrow et al., 2005), a widely used standardized assessment of adaptive functioning. Specifically, we used the Communication domain standard scores to capture communication skills.

Child initiations were coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and follow-up in two contexts: free play, and a snack routine. Weighted coding of all child intentional communication was completed using computer software (Yoder et al., 2021b). Each instance of child communication was recorded and weighted by complexity (gestures and contingent vocalizations = 1 point; single words = 2 points; simple phrases = 3 points; complex phrases/sentences = 4 points) and classified as imitated or non-imitated. The total non-imitated weighted score was used here. Scores were averaged across the two contexts to form an overall child initiations score. Reliability was calculated for 30% of the observations across time point and conditions using intra-class correlation (ICC=.97).

Model 1

A path diagram with standardized parameter estimates can be found in Figure 4. The direct effect of caregiver fidelity for the Focus on your Child domain at time 2 on child communication at time 3 was not significant (95% CI: [−1.482, 2.640]), nor was the indirect effects through child social attention/engagement at time 2 (95% CI: [−.110, .895]). The model provided the following fit to the data: χ2 (2) = 3.01, p = 0.22, CFI = 0.99, TLI = 0.96, RMSEA = 0.074.

Figure 4.

Model 1 path diagram and parameter estimates. *p < 0.05.

Model 2

A path diagram with standardized parameter estimates can be found in Figure 5. The direct effect of caregiver fidelity for the Create Opportunities domain at time 2 on child communication at time 3 was not significant (95% CI: [−1.095, 1.474]), nor was the indirect effect through child initiations at Time 2 (95% CI: [−.031, .766]). The model provided the following fit to the data: χ2 (2) = 2.07, p = 0.36, CFI = 0.99, TLI = 0.99, RMSEA = 0.019. Unstandardized parameter estimates and bivariate correlations for both models can be found in supplemental materials (Tables S9-S10).

Figure 5.

Model 2 path diagram and parameter estimates. *p < 0.05.

Discussion

This study used an exploratory sequential mixed methods design to explore potential active ingredients and mechanisms of change of Project ImPACT, an empirically supported NDBI for young children with autism or social communication delays. We developed a detailed causal model of the intervention process using qualitative methods, and subsequently tested select pathways using quantitative models. A strength of the study was the engagement of stakeholders in the intervention of interest, including individuals with a high level of expertise and end-users in the community. In contrast with other recent work for developing program theory prospectively using researcher input (e.g. Edmunds et al., 2022; Kirk et al., 2019), our retrospective approach emphasized practice-based knowledge from community clinicians as well as caregiver perspectives.

Pivotal skills and the developmental cascade.

Our qualitative results demonstrate that Project ImPACT targets a variety of interrelated developmental outcomes which naturally occur together during engaged social interactions. Consistent with the goal setting process laid out in the intervention manual, our analysis identified key social communication outcomes including initiating social interactions, expressive and receptive communication, and social engagement. These social communication outcomes were sequenced, such that child motivation and social engagement (targeted by the first strategies learned in Project ImPACT) support receptive understanding, which in turn supports expressive communication (see Figure 3). Together, these early changes were described as contributing to skill development over time as communication was naturally reinforced. As such, child motivation and social engagement can be considered ‘pivotal skills’ which support children in developing other social communication skills (such as language and communication skills) via increased participation and interest in the social environment (Koegel et al., 2001). In other words, these skills support the development of other skills in a cascading effect which unfolds over time, consistent with conceptualizing development as a dynamic process shaped by several interacting inputs, in which later stages build on previous stages (Griffiths & Tabery, 2013).

Adult responsiveness.

Our results also speak to the central role of adult responsiveness in increasing children’s participation in reciprocal social interaction and use of spontaneous communication. For example, following the child’s lead and imitating the child are ways that adults can be responsive to children’s interests and thus support their motivation and engagement in an interaction. Similarly, responsiveness to children’s attempts to communicate (i.e., contingent reinforcement), either spontaneously or in response to adult cues, supports children in increasing initiations and subsequent broader social communication growth.

These processes can be conceptualized from different theoretical lenses, consistent with the dual theoretical foundations of NDBIs. For example, behavioral theory and principles of operant conditioning were often evoked in participants’ descriptions of how communicative temptations or prompting strategies support children’s communication development; attempts to communicate are rewarded by adult responsiveness (i.e., contingent reinforcement), and thus more likely to occur in the future (Skinner, 1953). Results also correspond to previous research which has demonstrated associations between parent verbal responsiveness and child communication from a transactional theoretical perspective, which goes further in describing the bidirectional, mutual influence between communication partners (Edmunds et al., 2019). For example, respondents talked about how engagement in a reciprocal social interaction, supported by ‘bottom of the pyramid’ intervention techniques, facilitates adult implementation of more explicit teaching opportunities. Moreover, respondents talked about how children’s response to the techniques, particularly for playful obstruction and balanced turns, affected adults’ propensity to continue to use these techniques in the future. Taken together, our results emphasize that parent mediated NDBIs are not unidirectional (parent → child) but rather a bi-directional and co-constructed process.

Merged integration

Previous research provides some evidence in support of our causal model. For example, Yoder and colleagues found that the effect of Project ImPACT on child social communication was serially mediated by parent fidelity and then child intentional communication (2021b), supporting the hypothesis that child initiations support generalized communication growth.

Yet, our hypotheses were not supported by the quantitative data, as indicated by non-significant direct and indirect effects of our putative predictors on outcomes. Select individual pathways, however, were consistent with aspects of our theory of change model. For example, child social attention/engagement and initiations at time 2 was significantly associated with generalized communication skills at time 3, when accounting for child social attention/engagement and initiations at time 1. It is possible that the bi-directional and co-constructed nature of parent mediated interventions can account for our findings. For example, regarding Model 1, it is possible that some children are generally attentive and socially engaged within caregiver-child interactions even when a caregiver is not using the ‘Focus on your child’ techniques, allowing caregivers to spend more time implementing other techniques without adversely impacting social engagement. Similarly, for Model 2, caregivers whose children are infrequent initiators may benefit from frequent use of Communicative Temptations in order to elicit child initiations (and enable opportunities for natural reinforcement). On the other hand, for children who are frequent initiators, these techniques may not confer a meaningful change in the rate of child initiations. In fact, caregivers of children who initiate frequently may actually use these techniques less often, because there are fewer natural opportunities for them to do so (i.e., times when their child is not initiating). Other moderators – such as child chronological age or developmental level – may have also impacted our results, as it is possible that these relationships are present in specific points in development. Although following up on these possible moderators was outside of the scope and purpose of the current study, they are an interesting avenue for future research. In addition, because of how fidelity of implementation was measured, we were not able to differentiate between adult use of communicative temptations, playful obstruction, and balanced turns. Our qualitative findings suggest that adults may implement these techniques to different extents based on children’s response to them, therefore future research may consider examining these techniques separately.

Limitations and future directions

There were a number of methodological limitations to the present study. Although we were able to recruit a relatively large sample for our original data collection for Phase 1, the respondents were not necessarily representative of all consumers of Project ImPACT. For example, by necessity, we spoke with caregivers who completed most or all of the Project ImPACT intervention, the vast majority of whom reported a positive experience with the program. It is possible that selection bias played a role in who opted to participate in the study to share their experiences, or providers may have chosen to only share study information with select families. In addition, although we were able to elaborate on change processes for each intervention component, we were not able to ascertain which strategies seemed to be the most important in effecting change in children’s development. In fact, experts and providers often described how strategies work differently for different children and families, depending on child characteristics and caregivers’ typical interaction style. Although our integrated model represents the modal response as described by our participants, considering baseline child and family characteristics as potential moderators of treatment response is essential. Participating caregivers experienced the intervention via different modes of delivery (i.e., in person/telehealth, group/individual), which could have affected their perception of the intervention. Finally, our qualitative sample was lacking in racial-ethnic diversity and single-parent families. Future research emphasizing the intervention experience for families from a variety of marginalized backgrounds (e.g., socioeconomically disadvantaged families, linguistic minorities, immigrant families, minoritized racial/ethnic groups) is essential for understanding the fit of Project ImPACT for diverse families and to avoid reproducing systemic inequities in service access (e.g., Shenouda et al., 2022) in the context of the intervention itself.

In terms of our quantitative phase, our use of archival data meant that we did not have access to quantitative measures that could assess each hypothesized mediator and outcome in an ideal sequence or at the appropriate times for an ideal longitudinal mediation analysis (Little, 2013). Similarly, because the data were not collected with this type of analysis in mind, the analyses were underpowered. Together, these limitations prevented us from examining many of the hypothesized pathways in our Theory of Change model and limited our ability to understand causal relationships. It is also possible that some key variables (e.g., moderators, covariates) were omitted from our models, and that omitted variables biased our results.

Engaging end-users of the intervention allowed us to identify change processes which seem important or meaningful to consumers of the intervention and to consider potential change processes which have been overlooked in research to date. This approach allowed us to explore different elements of the intervention as they unfold over time and consider a broader scope of intervention processes and outcomes than is typically measured in experimental studies. Although our causal model was not supported by quantitative analyses of our archival dataset, we believe that aspects of our integrated change model are worth investigating prospectively in the future. For example, child motivation and emotion regulation both emerged as important constructs in understanding the child learning process. Operationalizing and measuring these constructs may be difficult but is perhaps needed to capture the child learning process quantitatively. In addition, other factors outside the scope of the present study warrant investigation, including key moderators of intervention response.

Our respondents also described several long-term treatment outcomes pertaining to children’s quality of life (e.g., social connections, inclusion, confidence and independence). Consistent with calls from neurodiversity advocates to focus more on outcomes relating to quality of life, social support, and wellbeing (Kapp, 2018), we believe that engaging the autistic community in the development and validation of measures of these outcomes is an important next step for the field. In the long-term, our results may be used to design prospective experimental studies of the change processes underlying this complex intervention, including factorial designs which can estimate the effects of individual component parts of an intervention. Indeed, we hope to use the data from this generative process to develop new research ideas and continue to refine the intervention under study. In the future, the theory of change models developed in the present study can be leveraged to design prospective studies with the intent of capturing causal processes. Further, qualitative interviews using the ToC framework could be used during the development phase of a new intervention to ensure that it addresses stakeholders’ service needs and targets long-term outcomes that are most important to autistic individuals and their families.