Understanding the Effects of Naturalistic Developmental Behavioral Interventions: A Project AIM Meta-analysis

Abstract

We examined the quality of evidence supporting the effects of Naturalistic Developmental Behavioral Interventions (NBDIs) for facilitating change in young children with autism. We also investigated whether effects varied as a function of specific features of the intervention, samples, and outcomes measured. Twenty-seven studies testing the effects of NDBIs were extracted from data collected for the Autism Intervention Meta-analysis (Project AIM), a comprehensive meta-analysis of group design, non-pharmacological intervention studies for children with autism aged 0-8 years. We extracted effect sizes for 454 outcomes from these studies for use in meta-regression analyses testing associations between intervention effects and mean participant chronological age, language age, autism symptomatology, percentage of sample reported as male, cumulative intervention intensity, interventionist, outcome boundedness, outcome proximity, and risk of parent/teacher training correlated measurement error. The extant literature on NDBIs documents effects on social communication, language, play, and cognitive outcomes. However, our confidence in the positive and significant summary effects for these domains is somewhat limited by methodological concerns. Intervention effects were larger for context-bound outcomes (relative to generalized), and for proximal outcomes (relative to distal). Our results indicate that NDBIs have promise as an approach for supporting development for some, but not all of the core and related features of autism in early childhood. Confidence in summary effect estimates is limited by study quality concerns, particularly an overreliance on measures subject to high detection bias. The results of this review support the use of proximity and boundedness as indicators of the limits of intervention effects.

Lay Summary

Naturalistic Developmental Behavioral Interventions may increase language, social communication, play skills, and cognition in young children with autism, but these increases are largest for skills directly targeted by the intervention, and in contexts that are similar to that of the intervention. These conclusions are tempered by some concerns regarding research design across the studies that have been conducted to date.

In 2015, the developers of several prominent ‘named’ interventions (i.e., established, often manualized interventions associated with specific names and developers) designed for young children with autism¹ authored a consensus statement designating that those approaches comprised a new intervention category called Naturalistic Developmental Behavioral Interventions (NDBIs; Schriebman et al., 2015). These interventions blend strategies from behavioral and developmental learning theories to target developmental milestones. Strategies shared among NDBIs include the use of loose teaching to target multiple exemplars across a variety of contexts throughout a child’s daily routines, following the child’s lead and engaging with the object or activity that draws the child’s focus of attention, and creating a naturally reinforcing context for learning where the adult and the child share control of the interaction. Some examples of prevalent NDBIs are Enhanced Milieu Teaching (Hemmeter & Kaiser, 1994; Kaiser, 1994), Pivotal Response Training (Koegel, Koegel, Harrower & Carter, 1999; Schreibman & Koegel, 1996), and the Early Start Denver Model (Dawson et al., 2010; Rogers & Dawson, 2010).

The primary developmental achievements targeted by NDBIs are those that have been identified as predictors of later social communication, language, and cognition, particularly in this population. These include imitation, shared affect, joint attention, turn-taking, attending to others, functional and symbolic play, prelinguistic communication (e.g., gestures, vocalizations, eye gaze, etc.), and emerging word use (Schreibman et al., 2015). The behavioral teaching techniques used in NDBIs include modeling, shaping, chaining, prompting, and differential reinforcement (Schreibman et al., 2015). Natural antecedents and contingencies are leveraged to occasion and reinforce targeted behaviors. For example, a spoken request would be met with the delivery of the requested item or activity, which is a natural consequence for this behavior. These naturalistic teaching strategies are embedded into the daily routines of the child to ensure that activities take place within meaningful and generalized social contexts.

Empirical Support for NDBIs

Group design studies of NDBIs suggest that this type of intervention can facilitate improvements in a number of domains, including social communication (Brian, Smith, Zwaigenbaum, & Bryson, 2017; Dawson et al, 2010; Kasari, Paparella, Freeman, & Jahromi, 2008; Lawton & Kasari, 2012; Shire et al., 2017), language (Chang, Shire, Shih, Gelfand, & Kasari, 2016; Dawson et al., 2010; Drew et al., 2002; Hardan et al., 2015), adaptive behavior (Dawson et al., 2010; Estes et al., 2015; Ingersoll, Wainer, Berger, & Walton, 2017; Wetherby & Woods, 2006), play (Chang et al., 2016; Kasari et al., 2008; Shire et al., 2017), and cognition in children with autism (Drew et al., 2002; Kasari et al., 2008; Wetherby & Woods, 2006). Recent meta-analytic work has suggested that NDBIs effect positive and significant improvements in language, play, cognition, autism symptomatology, and social communication (e.g., joint attention and joint engagement; Tiede & Walton, 2019); however, this work failed to include all available outcomes in summary effect estimation or to describe study quality beyond providing overall categorical ratings (i.e., ‘weak’, ‘adequate’, or ‘strong’). In our own recent meta-analysis of interventions for young children with autism (Sandbank et al., 2020a), we estimated summary effects across all available outcomes from studies of NDBIs in young children with autism. Among the eight outcome domains of interest to the quantitative synthesis, we found positive and significant effects on social communication, language, play, and cognition of children with autism between birth and 8 years of age. We additionally summarized the quality of studies in terms of risk of selection bias, attrition bias, detection bias, and reliance on parent/teacher reports (which constitute a specific subset of measures that are subject to high risk of detection bias, or a ‘placebo-by-proxy’ effect; Grelotti & Katpchuk, 2011). Our systematic review showed that, overall, the quality of empirical support for NDBIs was relatively high compared to that of other intervention approaches. Furthermore, the current body of evidence supporting NDBIs was largely devoid of threats to internal validity such as selection and attrition bias, and was mostly derived from direct measures of child performance/development, rather than from caregiver reports. However, study results were threatened by high detection bias, which can be present when assessors are aware of participants’ group assignment. Though the summary effects of NDBIs and the overall quality of evidence has been reported, less is known about the quality of the evidence supporting the use of NDBIs to improve specific outcomes (e.g., divided by domain). Moreover, little is known about the extent to which NDBI effects vary based on specific aspects of the intervention, the characteristics of the study sample, and the nature of the outcomes measured.

Factors That May Influence Intervention Effects

Autism symptomatology.

Prior research has found that children with lower autism symptomatology are more likely to benefit from intervention (Itzchak & Zachor, 2011; Perry et al., 2011; Sallows & Graupner, 2005), though which characteristics of autism best predict intervention outcomes, and why, is less understood (Vivanti, Prior, Williams, & Dissanayake, 2014). It could be that higher rates of stereotypic behaviors and fewer foundational social communication skills may interfere with a child’s ability to socially engage with clinicians, potentially limiting their learning opportunities in an intervention that is largely situated within a dyadic social context. Previous studies have demonstrated that young children who exhibit lower levels of autism symptomatology at the start of intervention derive greater benefit from at least some treatments delivered during early childhood (Ben-Itzchak & Zachor, 2007; Gordon et al., 2011), and differentially benefit from NDBIs delivered at higher intensities (e.g., greater number of hours per week; Yoder et al., 2018, 2019).

Chronological age.

Researchers and clinicians frequently assert that intervention should begin as early as possible in order to maximize benefits for young children with autism (National Institute of Child Health and Human Development, 2017; Zwaigenbaum et al., 2015). This principle stems in part from developmental theory, which suggests that improvements in the earlier stages of development extend into later life by serving as a foundation for subsequent development. Key milestones are also most readily acquired during biologically predetermined periods (e.g., “developmental windows”), which occur early in a child’s life and are marked by high levels of neuronal plasticity, which heighten the brain’s responsiveness to learning experiences and trigger the development of new neural connections (Bornstein, Hahn, & Haynes, 2010; Halfon, Shulman, & Hochstein, 2001; Masten & Cicchetti, 2010; Mundkur, 2005; Ruben, 1997). Though the logical framework of these theories suggests that children with autism should receive intervention as early as possible to capitalize on periods during which we should expect optimal plasticity, intervention research has not definitively shown that children who receive intervention at younger ages benefit more than those who begin intervention later in life. In fact, several previous meta-analyses of early interventions for children with autism have found that chronological age at intervention was not a significant moderator of intervention effects on outcomes of interest (Makrygianni & Reed, 2010; Reichow & Wolery, 2009; Sandbank et al., 2020b; Virues-Ortega, 2010).

Parent-reported sex.²

Prior literature suggests that there are a number of sex-related differences in the diagnosis and presentation of autism. Autism is more prevalent in boys (, with a male-to-female ratio of prevalence of 4-5:1, though several recent epidemiological studies suggest the ratio may be lower, at 2-4:1 (e.g., Fombonne, 2009; Hinkka-Yli-Salomäki et al., 2014; Kim et al., 2011; Saemundsen, Magnússon, Georgsdóttir, Egilsson, & Rafnsson, 2013; Surén et al., 2012). Girls are additionally often diagnosed at a later age compared to boys (Begeer et al., 2013). Differences in prevalence and age at diagnosis may be explained by differences in the presentation of core symptoms in boys and girls (Kreiser & White, 2014; Lai, Lombardo, Auyeung, Chakrabarti, & Baron-Cohen, 2015). For example, girls may demonstrate fewer or different types of repetitive behaviors and restricted interests compared to boys with autism (Antezana et al., 2019). In addition, girls with autism tend to show greater attention to faces, produce more social imitation, and show greater desire for interaction compared to boys with autism, which may mask other social communication differences in this group (Green, Travers, Howe, & McDougle, 2019; Harrop et al., 2019). Given the differences in presentation associated with sex, it is logical to hypothesize that intervention effects may vary by parent-reported sex. For example, it is possible that compensatory social communication behaviors in girls may limit the amount of improvement achievable on NDBI targets; however, it is also possible that a greater desire for interaction and increased social attention may facilitate greater developmental progression through NDBIs for girls.

Language level.

Intervention effects may vary as a function of child language level at the start of intervention. Previous investigations have found that children with higher language level at intervention onset tend to demonstrate larger gains than their counterparts with lower language level (Bono, Daley, & Sigman, 2004; Itzchak & Zachor, 2011; Sandbank et al., 2020b). This may be because children with more advanced language levels have more tools to help them engage in a social dyad with the interventionist administering an NDBI, which is where much (or all) learning is expected to take place. In other words, it is possible that children with higher language ages have a developmental foundation that can facilitate subsequent developmental improvements affected by intervention. This developmental readiness may enable them to advance more quickly than children who enter intervention with lower language ages.

Interventionist type.

Whether an NDBI is implemented by an educator, clinician, caregiver, or caregiver-clinician team may influence the strength of intervention effects. Prior meta-analytic work suggests that caregiver-implemented interventions can have strong positive effects on language outcomes for children with disabilities (Roberts & Kaiser, 2011). However, a recent meta-analysis examining the effect of interventions on language outcomes of children with autism reported that effects of caregiver-implemented interventions were significantly smaller than those of clinician- and combination-led interventions (Sandbank et al., 2020b). NDBIs are designed to be implemented within a child’s natural environment and naturally-occurring daily routines. Caregivers have the most opportunities to utilize NDBI strategies to foster their child’s development in everyday settings, and are arguably able to provide the most meaningful social context for intervention, which theoretically positions them to be the most effective implementers of NDBIs. It is possible, though, that a clinician’s expertise and extensive experience with intervention may lead to more effective and efficient use of NDBI strategies than the typical caregiver can provide, which could lead to larger intervention effects in favor of clinician-implemented versus caregiver-implemented treatment. Many NDBI models recommend that intervention strategies be collaboratively implemented by caregivers and clinicians (Dawson et al., 2010; Hemmeter & Kaiser, 1994; Schreibman & Koegel, 1996; Schreibman et al., 2014). A previous meta-analysis found that effects on spoken language outcomes were largest for interventions implemented by caregivers and clinicians working together compared to those of interventions implemented by either clinicians or caregivers alone (Hampton & Kaiser, 2016). Similarly, prior studies have reported positive results of NDBIs implemented by educators in the preschool setting (Goods, Ishijima, Chang, & Kasari, 2013; Lawton & Kasari, 2012). Although educators may not be able to devote as much undivided attention to a single child as caregivers or clinicians, there are still many opportunities in the preschool setting to implement NDBI strategies during play, meal times, and instruction. Educators may also be particularly skilled at leveraging other resources, such as a child’s peers, to scaffold growth in the school setting. Thus, it is not clear which type of interventionist (or combination thereof) may yield the greatest magnitude of effects for NDBIs.

Cumulative intervention intensity.

Many practitioners assert the provision of intensive intervention (e.g., intervention provided for 20-40 hours per week) has greater potential to lead to improved outcomes for young children with autism as compared to moderately administered interventions (National Research Council, 2001). However, the evidence to support this rather common recommendation is mixed. Though some primary studies have suggested that children who received more hours of intervention made greater gains (Lovaas, 1987), others have found that children who received different intervention intensities experienced non-significantly different levels of improvement (Fernell et al., 2011; Sallows & Graupner, 2005; Yoder et al., 2018; 2019). Two previous meta-analyses of early intensive behavioral intervention (EIBI) for children with autism found a positive relationship between intervention intensity and adaptive behavior outcomes (Virues-Ortega, 2010; Makrygianni & Reed, 2010), and one found that intervention intensity moderated effects of intervention on IQ outcomes (Makrygianni & Reed, 2010). However, other meta-analyses of interventions for children with autism have failed to find an association between cumulative intensity and intervention effects on IQ (Reichow & Wolery, 2009) or language outcomes (Hampton & Kaiser, 2016; Sandbank et al., 2020b).

Outcome proximity.

One potentially important aspect of intervention outcomes is their proximity to the domains directly targeted by the intervention. Outcome proximity characterizes whether an outcome reflects the exact targets that were taught in an intervention, untaught targets within the targeted domain, or untaught behaviors/skills in other non-targeted domains for which change would reflect ongoing development facilitated by the intervention. For example, in a naturalistic intervention geared towards improving initiations of joint attention by directly teaching participants to point, a measure of the number of times participants point would be considered a proximal outcome, as would other metrics indexing initiating joint attention (e.g., child gives or shows). Measures of broader social communication development would be considered distal outcomes, and measures of outcomes developmentally downstream from early social communication milestones, such as measures of language, would be considered very distal.

Prior systematic reviews and quantitative syntheses have found that investigations of interventions delivered to children with autism in early childhood often report outcomes that are overly proximal to intervention targets (Provenzani et al., 2019), that researchers are more likely to detect positive effects on proximal compared to distal outcomes (Yoder, Bottema-Beutel, Woynaroski, Chandrasekhar, & Sandbank, 2013), and that effect sizes for proximal measures are significantly larger than for distal measures across all intervention studies for young children with autism (Sandbank et al., 2020a). Thus, interventions geared towards young children with autism in general are more likely to improve performance in explicitly targeted skills, and less likely to facilitate broader development in targeted or untargeted domains. Given that the theoretical orientation of NDBIs suggests that targeting developmentally important proximal skills will bootstrap the achievement of downstream developmental milestones, it is likely that this trend will hold for NDBIs as well. However, it is important to assess whether these approaches vary in the degree to which they are actually able to impact proximal versus developmentally distal outcomes.

Outcome boundedness.

Similarly, the boundedness of outcomes to the intervention context likely influences the observed effects of interventions. Outcomes measured in a context that sufficiently differs from that of the intervention on a number of dimensions are more likely to reflect generalized changes, such that any improvement measured can be assumed to be independent from the intervention context. Documenting highly generalized effects not only requires that the assessment or measurement of outcomes take place in a different physical setting than the intervention, but also that the nature of the assessment differs from the nature of the intervention in several ways (Yoder et al., 2013). For example, results from standardized tests administered by unfamiliar assessors index generalized effects because they differ from the intervention context in terms of interaction partner, materials, and interaction style. Alternatively, measures of social behaviors collected within interactions that are similar to the context of intervention (e.g., a coded video of a play sample with the interventionist) are considered to be context-bound, as they may index intervention effects that manifest only when the child interacts with that particular individual, in response to that particular interaction style, or in the particular setting of the intervention.

Prior meta-analyses of autism intervention research have shown that researchers are more likely to detect positive effects on context-bound relative to generalized outcomes (Yoder et al., 2013) and that intervention summary effects for generalized outcomes are significantly smaller than those for context-bound outcomes (Sandbank et al., 2020a). This trend suggests that much of the overall positive effects reported in the early autism intervention literature may have low external and social validity, as the gains demonstrated in context-bound assessments are unlikely to generalize to other measures of the same constructs, or to environments and social situations that matter most for the child. Investigators relying on context-bound measures risk incorrectly interpreting intervention gains as indicative of generalized developmental change, even though the construct validity of these measures is likely poor.

Correlated measurement error (CME) related to parent and teacher training.

The focus on natural agents of intervention (e.g., caregivers and teachers) in NDBIs and the reliance on naturalistic assessment of skills within the context of interactions can lead to another threat to internal validity. Specifically, when caregivers or teachers act as interventionists (e.g., in parent- or teacher-mediated interventions) and then also serve as assessors, either as interaction partners in naturalistic assessments or as reporters on standardized caregiver reports, post-test scores may be positively biased in favor of the intervention group. That is, following intervention and training, the assessors (parents or teachers) in the treatment group are more informed about their children’s capabilities, and likely better able to elicit the behaviors of interest to the study, which may upwardly bias assessments that they complete, even in the absence of actual child gains due to the intervention. For example, in a study of a caregiver mediated social communication intervention, caregivers in the intervention group may be trained to imitate children and be more uniformly responsive as a means to increase their children’s social communicative behaviors. An observational measure of child social communication taken from a caregiver-child interaction might then reflect transiently enhanced social communication acts in the intervention group, because the measurement context featured a parent who was trained to better elicit social communication. This is a threat to internal validity that extends beyond that of detection bias, which is already present in outcomes derived from caregiver reports or observational measures of caregiver-child interactions, because in addition to being able to subtly influence interactions, assessors in the intervention group are specifically given strategies to influence these interactions over the course of the study. We refer to this threat to internal validity as parent/teacher training CME (Sandbank et al., 2020a). Prior evaluations of the NDBI literature have suggested that about half of reported outcomes are subject to this threat (Sandbank et al., 2020a), but no studies to date have examined whether effects for outcomes threatened by parent/teacher training CME are significantly greater than intervention effects on outcomes that are not subject to this threat.

Current Investigation

The purpose of the current study was to extend the results of previous work by Sandbank and colleagues (2020a) that reported summary effects of NDBIs on each of eight different outcome types by (a) examining the quality of the evidence supporting each specific summary effect according to previously established criteria, and (b) evaluating the extent to which NDBI intervention effects vary by other specified sample, intervention, and outcome characteristics hypothesized to influence the magnitude of summary effects. Our research questions were as follows:

1. What is the quality of the evidence supporting the use of NDBIs for facilitating gains in each of nine outcome domains (i.e., social communication; restricted, repetitive patterns of behavior, interests, or activities; sensory function, overall autism symptomatology; language; play; cognition; social-emotional skills/challenging behavior; and adaptive behavior) in terms of risk of selection bias, detection bias, attrition bias, reliance on parent/teacher report, risk of parent/teacher training CME, and the boundedness and proximity of outcomes?

2. Do the effects of NDBIs vary as a function of intervention characteristics including cumulative intervention intensity and interventionist type; sample characteristics including autism symptomatology, chronological age, parent-reported sex, and language age; and outcome features hypothesized to influence effect sizes, including outcome proximity, boundedness, and risk of parent/teacher training CME?

Method

Data for the current paper was collected as part of Project AIM (Autism Intervention Meta-analysis; Sandbank et al., 2020a), a scoping review and meta-analysis of all group design studies of non-pharmacological interventions for young children with autism. A brief overview of search and coding procedures is provided below, and additional details are provided in supplementary materials.

Search

A total of nine databases were searched to identify studies eligible for the larger Project AIM meta-analysis. This initial search yielded 12,933 results, which included group design studies of all intervention types on all outcome types for children with autism ages 0-8 years.In an effort to include data from unpublished studies, we contacted 90 researchers who had received federal funds to study autism to request unpublished data. No additional datasets were yielded by these requests.

In the parent meta-analysis, studies were screened for the following inclusion criteria: (a) published in English, (b) published from 1970 - 2018 (the year the initial search was conducted), (c) employed a group design that included both an intervention and control group, (d) included a simple majority of participants reported to have a diagnosis of an autism spectrum disorder (ASD), and (e) comprised a sample for which the average age of included participants was between 0 and 8 years. From these, studies of interventions categorized as NDBIs, based on the list proposed by Schreibman and colleagues (2015), were extracted. A total of 27 studies of NDBIs meeting the aforementioned criteria were identified and included in the current review and synthesis. The PRISMA diagram is presented in Figure 1.

Figure 1.

Prisma Flow Diagram of Systematic Search

Coding Procedures

All included studies were coded for sample, intervention, and outcome characteristics, as well as quality indicators by the second author and one additional member of a team of nine reliability coders. The team of coders comprised three PhD-level experts in non-pharmacological interventions for young children with autism and undergraduate and graduate students in their laboratories. Prior to coding, coders were trained to 80% agreement on a set of training studies. All studies were independently coded by two separate coders for the purposes of monitoring reliability. The coding manual is available upon request from the second author, and data has been deposited in a public repository (Open Science Framework, 2019).

Sample characteristics.

Included studies were coded for mean participant age in months at intervention onset, pre-intervention language age (age equivalency, in months), autism symptomatology, and percentage of males included in the sample.

Intervention characteristics.

When reported, cumulative intervention intensity in hours was extracted from the included studies by summing the total hours of intervention delivered to children across the duration of the study. Hours in which parents received coaching with children present were included in this total, but coaching hours during which children were not present were excluded. Individuals administering the intervention were coded as caregiver, clinician, educator, or combination.

Outcome characteristics and quality.

Outcomes were categorized by domain as either social communication; restricted, repetitive patterns of behavior, interests, or activities; sensory functioning; overall autism symptomatology; language; play; cognition; social-emotional skills/challenging behavior; and adaptive behavior. Outcomes were then coded for study- and outcome-level quality indicators, including their risk of selection bias, detection bias, and attrition bias, as well as their proximity to intervention targets, their boundedness to the context of intervention, and their risk of parent/teacher training CME.

Outcomes were categorized as either proximal or distal to intervention targets. An outcome was coded as proximal if it measured a skill that was directly taught, modeled, or prompted in the intervention, and was therefore over-aligned with the intervention targets. By contrast, an outcome was considered distal if it measured a skill that was not directly targeted in the intervention, and/or if the outcome was measured by a developmentally scaled assessment (e.g., the ADOS), following the assumption that such assessments tap broader learning within the targeted developmental domain. For example, if an intervention were to explicitly target, model, and prompt participants to say a set of five specific target words, an outcome which measured the use of those same five words would be coded as a proximal outcome. An outcome for the same intervention that measured expressive language using a standardized assessment would be coded as distal.

Outcomes were also categorized by their boundedness to the intervention context. Outcomes that were measured in a context identical to that of the intervention, or that differed from the intervention context on only one dimension (e.g., different assessor, materials, or activity) were coded as context-bound. Outcomes measured by a standardized instrument or that were measured in a context that differed from the intervention context across two or more dimensions were coded as generalized. In some cases, outcomes of parent-implemented interventions were measured by a standardized parent report assessment, and because it was not possible to know whether the parent completing the assessment was drawing on behaviors exhibited in a context sufficiently different from the intervention context, these outcomes were coded as potentially context-bound.

Parent/teacher training CME was considered high when caregivers or educators served as both the implementers of intervention and assessors of intervention outcomes (e.g., if a parent who was trained to implement the intervention also served as the interaction partner in an outcome measure for that intervention). Otherwise, parent/teacher training CME was considered low.

Effect size information.

Unadjusted post-intervention means, standard deviations, and ns were extracted for intervention and contrast groups. This information was used to calculate the standardized mean difference (d) between groups after intervention, which was then converted to Hedge’s g to correct for small sample sizes. Effect sizes were reflected as necessary so that higher effects consistently represented outcomes that favored the intervention group.

Reliability

Reliability was calculated for all studies in the primary meta-analysis using intraclass correlation coefficients (ICC) and unweighted kappa coefficients for continuous and categorical variables, respectively. Average kappa values across categorical variables included in the current paper ranged from 0.73-0.86, and ICCs across continuous variables ranged from 0.81-0.97. The aforementioned values reflect excellent reliability amongst coders for all variables of interest to the present report.

Preliminary Analyses

Variables were selected for moderator analyses if there were a sufficient number of studies reporting data on the putative moderator, and if there was theoretical and/or empirical justification for testing it. Prior to conducting moderator analyses, continuous predictors were plotted against effect sizes to determine whether linear meta-regression models were appropriate (e.g., rather than quadratic models), and categorical data were examined to ensure that multiple categories featured a minimum of five effect sizes per cell.

Primary Analyses

In order to account for the intercorrelation of multiple effect sizes extracted from single study samples, robust variance estimation (RVE) with small sample adjustments was used to conduct all summary effect estimation and meta-regression analyses using the R package Robumeta (Fisher, Tipton, & Zhipeng, 2017). Meta-regression analyses were conducted on all putative moderators that met preliminary analysis criteria for inclusion.

Results

Study Sample and Intervention Characteristics

A full list of included studies and coded sample and intervention characteristics is included in Supplementary Table S1, and a summary of sample and intervention characteristics across studies is presented in Table 1.

Table 1.

Features of Continuous and Categorical Moderators

	# of studies	M (SD)	MIN	MAX
Intervention Intensity	12	555.8 (1126.5)	8.3	3276
Chronological Age	27	38.7 (12.2)	18.2	75.4
Language Age	10	20 (4.8)	12	31.5
Interventionist
Educator	6
Caregiver	11
Clinician	5
Combination	5
Autism Symptomatology
Moderate	8
High	2
Unreported	17
Outcome Proximity
Proximal	17
Distal	22
Outcome Boundedness
Context-bound	110
Generalized	234
PCB	110
Outcome Risk of CME
High	218
Low	234

Note. SD = standard deviation, PCB = potentially context-bound, CME = correlated measurement error related to parent/teacher training

RQ1. Quality of Evidence by Outcome Domain

From the 27 included studies of NDBIs, 454 total outcomes were extracted for analysis. Of these, 10 were coded as measures of overall autism symptomatology, 234 were coded as social communication outcomes, 12 were coded as restricted, repetitive patterns of behavior, interests, or activities outcomes, 12 were coded as adaptive outcomes, 26 were coded as cognitive outcomes, 80 were coded as language outcomes, 53 were coded as play outcomes, and 12 were coded as social emotional/challenging behavior. In addition, 2 were coded as brain-imaging data, 9 were coded as motor outcomes, and 4 were coded as ‘other’. Because less than five studies were represented for each of these three (i.e., brain-imaging, motor, and other) outcome domains, these outcomes were excluded from subgroup examination of quality indicators, but included in moderator analyses. No studies included sensory outcomes. Individual forest plots reflecting primary and summary effects for each outcome type that met the five study threshold are included in Supplementary Figures S1-S8.

Figure 2 depicts quality indicators and outcome attributes for included studies separated by outcome domain, for all domains for which we were able to estimate summary effects. By far, the majority of the evidence supports NDBI effects on social communication outcomes, and to a lesser extent, language, play, and cognition. In this section, we restrict our quality reporting to domains for which summary effect estimates were significant and positive. We designated 50% of intervention effects within a given domain being subject to a given threat as a threshold for identifying serious methodological concerns that limit our confidence in conclusions regarding these specific summary effect estimates. For social communication outcomes, 64% were threatened by detection bias, and 54% were threatened by parent/teacher CME. More than half of social communication outcomes were considered overly proximal to intervention targets (62%) and bound or potentially bound (55%) to intervention contexts. For language outcomes, 59% were potentially or clearly threatened by detection bias. For play outcomes, 71% were threatened by detection bias, and a majority of outcomes were coded as context-bound (67%) and proximal (73%). For cognition, none of the quality indicators met the threshold for serious methodological concerns.

Figure 2.

Quality Indicators for Included Studies

Low risk of bias Unclear risk of bias High risk of bias

RQ2. Moderator Analyses

Descriptives of continuous and categorical moderators are presented in Table 1, and detailed results of moderator analyses are presented in Table 2.

Table 2.

Results of Meta-regression Analyses

Predictor	Studies	Outcomes		SE	t	df	p
Model 1: Intervention Intensity	12	265
Intercept			0.28	0.11	2.54	7.07	0.038**
Cumulative intensity in hours			0.09	0.12	0.75	1.33	0.563
Model 2: Interventionist	27	453
Intercept (Reference: Caregiver)			0.34	0.09	3.98	12.57	0.002***
Clinician			0.12	0.19	0.67	4.42	0.539
Combination			−0.26	0.19	−1.37	6.71	0.215
Educator			0.01	0.16	0.09	7.30	0.931
Model 3: Chronological Age	27	453
Intercept			0.31	0.06	4.93	24.04	< 0.001***
Age in Months			−0.06	0.05	−1.22	8.08	0.256
Model 4: Biological Sex	25	417
Intercept			0.33	0.06	5.19	18.50	< 0.001***
Percent Male			−0.06	0.05	−1.36	9.30	0.204
Model 5: Language Age	10	228
Intercept			0.34	0.09	3.84	7.44	0.006***
Language Age Equivalency			−0.02	0.05	−0.34	2.77	0.758
Model 6: Outcome Boundedness	27	453
Intercept (Reference: Context-bound)			0.55	0.09	6.10	11.30	< 0.001***
Generalized			−0.40	0.11	−3.53	15.00	0.003***
Potentially Context-bound			−0.31	0.13	−2.49	19.30	0.022**
Model 7: CME	27	453
Intercept (Reference: High Risk)			0.37	0.72	5.21	20.80	< 0.001***
Low Risk			−0.17	0.10	−1.66	22.50	0.112
Model 8: Outcome Proximity	27	453
Intercept (Reference: Distal)			0.20	0.08	2.40	17.20	0.028**
Proximal			0.25	0.11	2.17	23.80	0.041**

Note. CME = Correlated measurement error.

^***p < .01

^**p < .05

^*p < .10

Sample characteristics.

The results of meta-regression models indicated that neither the mean chronological age (B = −0.06, p = .256), nor the mean language age of samples at study entry (B = −0.02, p = .758), nor the reported percentage of males in each sample (B = −0.06, p = .204) moderated NDBI intervention effects. Because an insufficient number of participant samples with high autism symptomatology were represented (i.e., < 5), autism symptomatology categorization was excluded from moderation analyses.

Intervention characteristics.

Effect sizes did not significantly vary as a function of cumulative intensity of intervention in hours (B = 0.09, p = .563), or by the type of interventionist that implemented the intervention (clinician B = 0.12, p = .539; combination B = −0.26, p = .215; educator B = 0.01, p = .931; reference category = caregiver).

Outcome characteristics.

Effect sizes differed significantly as a function of both proximity and boundedness of the outcome to the intervention. Outcomes that were coded as proximal to the intervention had significantly larger effects (B = 0.25, p = 0.041) than those that were coded as distal. Compared to context-bound outcomes, effect sizes were significantly smaller for outcomes coded as generalized (B = −0.40, p = .003) or potentially context-bound (B = −0.31, p = .022). Figure 3 depicts summary effects for proximal and distal outcomes, and for context-bound, potentially context-bound, and generalized outcomes. Effect sizes for outcomes with a low risk of parent/teacher training CME were smaller on average than those with a high risk for CME, but this difference did not reach statistical significance (B = −0.17, p = .112).

Figure 3.

Forest Plot of Outcome Proximity and Boundedness

Discussion

NDBIs have recently amassed a substantial amount of evidence suggesting that they are a promising approach for supporting development in social communication, language, play, and cognition in young children with autism, and the quality of this evidence is relatively high in comparison with that of other intervention approaches commonly used to support development in this population (Sandbank et al., 2020a). However, the designation as promising comes with some caveats related to the nature of the outcomes and study quality concerns. Although a majority of the primary NDBI effects were not threatened by selection and attrition bias and were not derived from caregiver report measures, much of the evidence relies on outcomes that were threatened by high risk of detection bias and high risk of parent/teacher training CME. While we failed to find definitive evidence that parent/teacher training CME significantly inflates estimates of intervention effects, our summary estimates should be interpreted with these quality concerns in mind. In addition, nearly half of outcomes were proximal to intervention targets, and a majority were bound or potentially bound to the intervention context, which our findings suggest significantly influence intervention effect estimates. Thus, while NDBIs have certainly amassed enough evidence to warrant their use for supporting some core (i.e., social communication) and related (i.e., language, play, cognition) areas of need for children with autism, more high quality evidence is needed to ensure that the observed positive effects of NDBIs are not inflated as a result of study design and outcome measurement choices.

When considered without regard to quality, evidence suggests that NDBIs positively affect outcomes in the domains that are most frequently tracked in studies of these approaches. However, the magnitude of our summary estimates – particularly for social communication, language, and play – may have also been inflated by effects from outcomes that were context-bound, and/or threatened by detection bias and parent/teacher training CME. The substantial number of outcomes that were subject to all three of these potential risks of inflation reflects a common reliance on researcher-created observational measures collected during natural interactions with caregivers to index effects of parent-mediated interventions. In addition, while we did not find evidence to suggest that NDBIs are differentially effective according to chronological age, language age, and parent-reported sex composition of the study samples, or implementer or intensity of the intervention, discrepancies in reporting practices, limited power, and restricted range may have obscured true associations between these participant/intervention characteristics and intervention effectiveness. Future primary studies can use innovative research designs to further our understanding of the true impact of these participant and intervention characteristics on intervention effectiveness. Below, we further explore our findings in depth.

Understanding the Influence of Proximity, Boundedness, and Parent/Teacher Training CME

Outcome proximity.

We found that outcomes that measured skills that were highly proximal to intervention targets—that is, skills which were specifically and explicitly taught in the intervention—had significantly larger effects than outcomes coded as distal from intervention targets, although the summary effects for both were significantly different from zero (see Figure 3). This finding is unsurprising, given previous literature investigating the differential likelihood and size of effects for proximal versus distal dependent variables in early childhood autism interventions (Sandbank et al., 2020a; Yoder et al., 2013). Because NDBIs are intended to teach skills of developmental importance in early childhood with the intent of fostering broader development in downstream skills over time, it is important that investigations of this intervention approach include a robust representation of both proximal and distal outcomes, and ideally incorporate mediation analyses to link the two together in a developmental cascade. It is possible, though, that effects on distal outcomes for young children with autism may be present but undetectable for a substantial period of time following intervention. This possibility is supported by the developmental theory upon which NDBIs are, in part, based (Bornstein, Hahn, & Haynes, 2010; Masten & Cicchetti, 2010), as well as evidence from autism intervention research. A delayed effect on distal outcomes, for example, was observed in the landmark PACT study by Green and colleagues (2010; 2016), wherein a significant difference in autism symptom reduction between the intervention and control groups was not detected until follow-up measures were collected, six years after the cessation of a low-intensity parent-led intervention provided in early childhood. Thus, long-term follow-ups are likely necessary to thoroughly vet the theory of change for this intervention approach. Such studies will be time-consuming and costly, but should be a priority for researchers testing the efficacy of NDBIs for your children with autism and for funding agencies.

Outcome boundedness.

Intervention outcomes that were measured in the same or very similar context as the intervention had significantly larger effects than outcomes measured in contexts which differed from that of the intervention across two or more dimensions (i.e., interaction partner, interaction style, materials, and setting). Though it is expected that learned skills are most readily demonstrated in environments highly similar to the context in which the skills were taught, it is vital that the gains observed as a result of intervention be generalized to flexible and socially valid contexts for us to conclude that we have effected meaningful change for children with autism. Teaching generalization from intervention outset is a core tenet of NDBIs (Schreibman et al., 2015), and though the summary effect of NDBIs on generalized outcomes was positive, our findings suggest that further work is needed to increase intervention effects on generalized skills for young children with autism.

Parent/teacher training CME.

Although our results did not support the notion that NDBI effects were significantly inflated by the presence of parent/teacher training CME, this should not be interpreted as definitive proof that this quality indicator poses no threat to the validity of study results. Parent/teacher training CME arises from a change in the assessment context over the course of the study that may produce changes in the outcome that systematically favor the intervention group. Fundamentally, it is an instrumentation threat to validity. Given that this threat was present in approximately half of all outcomes in studies of NDBIs, and that this effect trended towards significance in the anticipated direction, any interpretation of summary effect estimates should give consideration to the potential influence of this and other more established quality indicators.

Measurement Considerations for Primary Research

Many researchers attempting to measure change affected by intervention believe that measures derived from naturalistic samples have more apparent construct validity, in that they are purportedly more likely to reflect changes that are generalized to unstructured contexts. Moreover, if the goal of a given intervention is to facilitate change in parent responsivity, measuring change during parent-child dyadic interactions is a natural choice. Unfortunately, because naturalistic interactions with caregivers serve as the primary intervention context for NDBIs, measures derived from these interactions likely reflect changes that are confined (or at least potentially confined) to that specific context (e.g., responsive play with parents). Improvements in the intervention group on such measures could reflect only transient change in the dyadic context driven by the caregiver’s increased responsivity attributable to training, rather than developmental change in the child that will generalize across communication partners and interaction styles. Our point is that demonstrating change within the context where such change was effected is only an initial step towards demonstrating that an intervention can induce change that extends beyond the context and targets of intervention. Researchers should avoid overstating the developmental ramifications of such findings, and instead interpret them as preliminary evidence establishing the theorized mechanisms of action, while acknowledging that these effects are potentially transient and subject to the risk of detection bias and parent/teacher training CME.

Future researchers that wish to guard against these risks while still relying on observational measures derived from naturalistic contexts can do so by ensuring the interaction partner within observational measures of child behavior is not trained in the intervention over the course of the study and is not aware of group assignment. For example, in a study examining the effects of a parent-led intervention to support symbolic play development, future researchers could collect observational measures of play using previously developed and validated semi-structured interactions with a naive examiner (e.g., the Developmental Play Assessment; Lifter, 2000; the Structured Play Assessment; Ungerer & Sigman, 1981). Although some studies have employed these types of measures (Boyd et al., 2018; Chang et al., 2016; Kasari et al., 2006), this approach is not yet the norm. Alternatively, researchers may derive observational measures from contexts that feature participants’ untrained teachers or peers as interaction partners (while ensuring they remain naive to group assignment over the course of the study). Yet another approach would be to retain measures of caregiver-child interactions, but use these for mediation analyses that would connect these proximal, context bound effects to developmentally distal and generalized child outcomes that are not threatened by parent/teacher training CME. Doing so would allow researchers to assess whether caregiver-child interactions are the substrate for downstream developmental achievements.

Null Findings

The effects of NDBIs did not vary according to any of the child or intervention characteristics that we hypothesized would moderate the magnitude of effects on outcomes of interest. Notably, summary effects were not moderated by child age at intervention onset. Although it is widely claimed that earlier intervention facilitates greater gains for children with autism, we did not find evidence to support this assertion. While it is possible that the overrepresentation of children aged 2-5 years may obscure a true linear relationship between age at intervention onset and intervention effectiveness, the results of this meta-analysis suggest that children who begin intervention at a later age may experience similar benefits from NDBIs as children who receive intervention starting at a younger age. The absence of this association does not mean that naturalistic interventions provided at early ages (e.g., 2-3 years) are not effective, as several studies included in this meta-analysis demonstrated that even very young children with autism benefitted from early intervention (Dawson et al., 2010; Drew et al., 2002; Estes et al., 2015; Wetherby & Woods, 2006). Rather, we failed to find evidence that the potential for developmental improvements facilitated by NDBIs decreases with advancing age, at least up to age 6 (the highest mean participant age documented in included studies).

We also did not find evidence that intervention effects varied as a function of cumulative intervention intensity. Given that we were unable to code intervention intensity for some studies due to differences in reporting practices, and that there was a risk of incorrect reporting of total intervention hours in studies of parent-implemented interventions, it is possible that measurement error may have limited our ability to detect a potentially true association between intervention intensity and intervention effectiveness. Alternatively, it is possible that NDBIs provided at lower intensities can facilitate substantial developmental gains in children with autism (Brian et al., 2017; Ingersoll, 2012; Kasari et al., 2008).

Finally, we did not find evidence that intervention effects significantly differed by the type of interventionist (e.g., caregiver, clinician, combination, etc). Recent meta-analyses examining the extent to which intervention effects on language and social communication vary according to the type of interventionist have mixed findings (Fuller & Kaiser, 2020; Hampton & Kaiser, 2016; Sandbank et al., 2020b), and the sole prior meta-analysis of NDBIs did not examine this question. Additional primary research might further explore this question by directly comparing caregiver- and clinician-mediated iterations of the same intervention. In the absence of clear evidence supporting added benefits dependent on intervention intensity, interventionist type, and age of intervention onset, clinician recommendations for early intervention should prioritize family-centered intervention approaches which balance the needs of the family and the young child (Sandall, McLean, & Smith, 2000; Sandbank, Bottema-Beutel, & Woynaroski, 2020). Recommendations for high-intensity clinician-led interventions should take into account the potential emotional and financial strain which can be caused when families are compelled to make excessive time commitments to intervention, coordinate the presence of professionals in their home, or interrupt their daily routines to travel to clinics. Intervention providers should consider what amount of intervention is developmentally appropriate for young children, recognizing that we do not yet have definitive empirical support for the notion that more is always better as it relates to NDBIs.

Alignment with Prior Meta-Analysis

Our findings replicate many of those reported in the only prior meta-analysis of NDBIs. Like those recently reported by Tiede and Walton (2019), our summary estimates support the notion that NDBIs have significant positive effects on language, play, and social communication for children with autism. In addition, we expanded on Tiede and Walton’s findings that lower quality studies yield larger effects than more methodologically rigorous ones, by examining specific outcome quality indicators (i.e., risk of parent/teacher training CME) and outcome characteristics (i.e., boundedness and proximity) as putative moderators of intervention effects.

However, in several cases our results depart from prior meta-analytic findings. For example, we did not replicate Tiede and Walton’s finding that NDBIs facilitate a significant reduction of autism symptomatology. This discrepancy may be attributable to the different measures of autism symptomatology represented in the two samples. A larger proportion of autism symptomatology outcomes in the Tiede & Walton meta-analysis were derived from parent report measures, which are at high risk of inflation due to detection bias. We also found little evidence to support the notion that increased intervention intensity was associated with larger intervention effects, which departs slightly from Tiede and Walton’s findings that dosage moderated intervention effects on joint attention (but not on other outcomes). This discrepancy may be due to our different characterization of intervention intensity, as Tiede and Walton only included hours of intervention delivered by trained research staff, while we included reported hours of intervention delivered by caregivers. Finally, while Tiede and Walton selected individual effect sizes from each included study, we employed RVE statistical methods that allowed for the inclusion of all 454 independent effect sizes extracted from the available pool of studies. Our inclusion of the full set of effect sizes contributes to a more precise picture of the state of the literature for this intervention approach.

Limitations

As is the case in any analysis of coded variables, measurement error related to coding is an acknowledged risk, though this risk was minimized by the extensive discrepancy resolution procedures applied to all codes and high interrater reliability in the present study. Additionally, we lacked the power to run more complex models or test various interactions of interest, such as the potential interaction between intervention intensity and outcome boundedness. Similarly, the precision of summary estimates and moderator analyses is limited by the volume and quality of the primary literature on NDBIs. For some target outcomes, such as play, effect sizes were sourced from a single or a small handful of named interventions, which somewhat limits the degree to which we can be confident in generalizing the results from these studies to the broader category of NDBIs. Even so, we do believe the estimates we have calculated are informative. It is notable as well that we were unable to calculate summary effects of NDBIs on a number of outcomes of interest, including brain-based outcomes and indices of sensory and motor function, as such measures have been underrepresented in studies conducted to date. Inclusion of broader neural and behavioral outcome measures is necessary to determine whether NDBIs have the potential to influence the wider range of domains known to be impacted by autism.

Future Primary Research

Future primary research should not only include a robust representation of outcomes that span the continua of boundedness and proximity, but should also endeavor to monitor long-term outcomes to determine whether and to what extent NDBIs lead to downstream developmental gains that are detectable later in life. It is also important that future investigators register or pre-register their clinical trials whenever possible, and distinguish between primary and secondary outcome variable(s) in registrations and trial reports.

In addition, NDBIs are package interventions, and the key components of these packages have yet to be identified. In other words, which components of NDBIs are most crucial and how each of these components affect children with various demographic and behavioral characteristics is currently unknown. The Sequential Multiple Assignment Randomized Trial (SMART) study design, which was developed specifically to help develop adaptive interventions (Almirall, Nahum-Shani, Sherwood, & Murphy, 2014), has been used by prior autism researchers to investigate whether adaptations to intervention or increases in intervention intensity can provide added benefit to children who did not initially appear to benefit from an NDBI (Kasari et al., 2014). Similarly, this study design should be used in future research to investigate whether certain aspects of NDBIs differentially benefit children who do not initially reach intervention goals. Thoughtful research design, incorporating elements such as pre-treatment measurement of child and family characteristics that may predict a differential response to treatment and midpoint measurement of theorized mechanisms of action, in combination with advanced analytic approaches to moderation and mediation analysis, also have great potential to advance our understanding of the subgroups for whom, and the mechanisms by which NDBIs work.

Conclusions

On the whole, NDBIs have accrued a substantial body of supporting evidence for their effects on some, but not all, of the core and related characteristics associated with autism, including social communication, language, play, and cognition. This body of evidence is relatively methodologically rigorous compared to that of other common intervention approaches for children on the spectrum. However, this evidence is nonetheless limited by some methodological concerns, including a prevalence of outcomes subject to high detection bias and parent/teacher training CME. The results of this analysis also support the use of proximity and boundedness as important indicators of outcome quality. Future research should incorporate both distal and generalized outcomes, in order to obtain a more robust understanding of the extent to which NDBIs affect durable, meaningful, and developmentally important changes that benefit children with autism.