Examining relationships between child skills and potential key components of an evidence-based practice in ASD

Abstract

Background:

Research in autism spectrum disorders (ASD) has identified a need to understand key components of complex evidence-based practices (EBP). One approach involves examining the relationship between component use and child behavior.

Aims:

This study provides initial evidence for identifying key components in a specific EBP, Pivotal Response Training (PRT). We examined which components were related to child response and evaluated relationships between provider characteristics, child characteristics and component intensity.

Methods:

Trained coders reviewed archival videos (n = 278) for PRT fidelity and child behavior. We completed multi-level regression and latent profile analysis to examine relationships between intensity of individual or combinations of PRT components and child behavior, and moderators of component use.

Results:

Analyses indicated differential relationships between specific components and child behaviors which may support methods of altering intensity of components to individualize intervention. Profile analysis suggested relatively intensive use of most PRT components, especially antecedent strategies, may maximize child responsivity. Providers with postgraduate education trended toward higher intensity component use. Child characteristics did not moderate use.

Implications:

Careful examination of key components of ASD interventions may helps clarify the mechanisms of action. Recommendations specific to PRT implementation and use of the methodology for other interventions are discussed.

1. Introduction

Autism spectrum disorder (ASD) is a complex, heterogeneous, neurodevelopmental disorder that manifests in early childhood and usually lasts throughout the lifespan. The CDC currently estimates that 1 in 59 children in the US have ASD (Baio et al., 2018), making effective treatment for children with ASD very important. Most current policy related to treating ASD across systems of care requires that providers use evidence-based practices (EBPs; Council for Exceptional Children, 2009; National Conference of State Legislatures [NCSL], 2018; Yell, Katsiyannis, Losinski, & Marshall, 2016). However, there are currently 27 interventions identified as EBPs for ASD, and most are quite complex and comprised of multiple components (Odom, Cox, Brock, & National Professional Development Center on Autism Spectrum Disorders, 2013; Stahmer, Suhrheinrich, & Mandell, 2016; Wong et al., 2015). This complexity presents several challenges related to understanding the causal mechanism of the intervention, developing individualized treatment protocol and training providers to effectively use EBPs.

First, although many of the multiple EBPs for ASD have components in common, causal mechanism of the interventions are not well understood. Most EBP development and efficacy evaluation has focused on a multi-component protocol or manualized program without examining the relative weight of individual components on child outcomes. However, an understanding of the causal mechanisms of the EBP, and the contribution of specific key components is essential. This knowledge will facilitate optimization of current EBPs, facilitate comparison of components across EBPs, and lead to better outcomes for children with ASD (Chorpita & Daleiden, 2009; Stahmer, Suhrheinrich, Mandell et al., 2016).

Next, due to the heterogeneous clinical presentation of individuals with ASD, treatment must also be individualized based on clinical characteristics (Stahmer, Schreibman, & Cunningham, 2011). Researchers have recommended the elimination or adaptation of EBP components in an attempt to address diverse client characteristics (Lau & Brookman-Frazee, 2016; Stirman et al., 2015); however, the field has not yet developed an empirically validated process for how to individualize intervention based on client characteristics.

This leads to an additional challenge in training providers to use complex EBPs. Research indicates that providers are varied in how quickly and completely they learn to use an EBP (Pellecchia et al., 2015; Suhrheinrich, 2015). Moreover, after training, providers show variability in which components are maintained as part of their clinical care (Suhrheinrich et al., 2013). Providers adapt EBPs to their own work and may choose only components they are most comfortable using (Lau & Brookman-Frazee, 2016; Stirman et al., 2015). Empirically-based guidance on which components are key will lead to more effective individualization of interventions for specific children and may simplify training and increase effective EBP implementation (Embry & Biglan, 2008).

In an effort to address these challenges, several researchers now suggest a “common” components approach, which involves identifying the key components within and across specific EBPs to increase intervention accuracy and effectiveness (Weisz et al., 2012). In fact, children’s mental health research links child outcomes to the use of key components of EBPs (Garland et al., 2014). A distillation and matching model in which researchers conceptualize interventions as composites of individual strategies has successfully allowed for empirical matching of specific strategies to clients, settings, and other factors to individualize interventions in childhood disorders, such as anxiety and depression (Chorpita, Daleiden, & Weisz, 2005). Measuring each intervention component separately may help with identifying its relative contribution to the EBP (Sanetti & Kratochwill, 2009; Schulte, Easton, & Parker, 2009). This, in turn, would support individualization of EBPs for individual children and varied settings by providing information about which component will be most effective for a particular child (Pellecchia et al., 2015; Stahmer, Collings, & Palinkas, 2005). One recent application of this method to EBPs for ASD involved measuring intensity of correct implementation of specific components. Abry, Hulleman, and Rimm-Kaufman, (2015) linked variability in providers’ use of specific components of a social-emotional learning intervention to child outcomes and used this strategy to identify the key components of the intervention.

One well-supported EBP is Pivotal Response Training (PRT), a naturalistic developmental behavioral intervention developed for children with ASD (Humphries, 2003; National Autism Center, 2009; Schreibman et al., 2015; Shavelson, 2002; Wong et al., 2014). PRT is a packaged intervention comprised of eight specific components (see Table 1 for a list of PRT components with definitions) designed to work together to increase motivation of the child with ASD to engage in the learning activity. As in other strategies based on the principles of applied behavior analysis, the provider uses some components to create an opportunity for the child to respond (antecedent components) and uses others in response to the child’s behavior (consequence components).

Table 1.

Coding Definitions and Reliability.

Provider Behavior PRT Components	Definitions					Reliability
Student Attention	Child attending to the provider before cue provided					.76
Clear Cues	Cue spoken in clear language or gestural expression.					.76
Developmentally Appropriate Cues	Cue developmentally appropriate and provided at the child’s or slightly above the child’s developmental level.					.76
Novel Cue	Provider presents a cue or materials for the first time.					.81
Shared Control	Provider follows child’s interests and includes preferred materials or activity, moves to new activity if child loses interest.					.75
Maintenance/ Acquisition Task	Maintenance Task: Child correctly responds to cue 80% of trials Acquisition Task: Child correctly responds to cue < 80% of trials.					.78
Direct Reinforcement	Provider uses contingent, tangible reinforcement directly related to the activity.					.78
Turn Taking	Provider takes or facilitates turns while interacting with the child.					.86
Child Behavior Response to Instruction		Definitions				Reliability
Appropriate		Behavior is in range of skills targeted
Goal Directed Attempt		Behavior serves same function as target but less accurate/ complex				.75
Incorrect		Behavior is not correct
No Response		Child does not respond to instruction
Inappropriate		Responses are unrelated to the interaction or disruptive
Child Overall Skill Use						Reliability
	1	2	3	4	5
Participation	None	Limited.	Partial/some	Frequent	Full/Consistent	.94
Functional Object Use						.89
Functional Commun.						.78
Responsivity						.85
Attention						.89

PRT has been studied primarily as a complete package (e.g., Verschuur, Didden, Lang, Sigafoos, & Huskens, 2014). Observations of teachers using PRT highlight the importance of understanding the relative contribution of each component. Few teachers, even after training, meet research standards for correct use across components, and they have consistent difficulty with specific components, such as shared control and introduction of multiple cues (Suhrheinrich et al., 2013). Researchers have not yet examined the relative contribution of the different components of PRT or other similar autism interventions. Additionally, research thus far provides limited guidance for how to adapt the intervention package based on individual child characteristics (Pellecchia et al., 2015; Stahmer et al., 2011).

Research demonstrates a clear need for systematic evaluation of potential key components of EBPs in ASD in order to determine whether or not they can be adapted based on child or setting characteristics, how intensively each component needs to be implemented, and if any components are inert. To that end, our aims were to use archival data of providers using PRT with children with ASD to provide a preliminary analysis of (1) relationships between use of individual PRT components (attention, clear cue, developmentally appropriate cue, novel cue, shared control, maintenance task intersperse, direct reinforcement, turn taking) and concurrent child skills (appropriate, attempt, incorrect, no response, inappropriate) and behavior (participation, object use, communication, responsivity, attention), (2) relationships between use of combinations of components and concurrent child skills and behavior, and to (3) evaluate relationships between provider characteristics (education), child characteristics (autism severity, receptive language) and intensity of PRT components.

2. Method

2.1. Design

This study employed a descriptive design using archival video data from three intervention studies involving providers using PRT with children with ASD. To address the research aims, we completed: (A) detailed coding of archival video data of providers using PRT with children with ASD for (i) quality and intensity of use of each PRT component, and (ii) child behavior and responsivity; (B) multi-level regression analyses to examine the association between use of PRT components and child behavior and responsivity (Aim 1), and latent profile analysis (LPA) to examine whether use of groups of PRT components supported child outcomes (Aim 2); and (C) multi-level regression analyses to identify associations between provider characteristics, children characteristics, and use of PRT components (Aim 3).

2.2. Participants

The archival video data used for this study included 49 providers and 41 children (videos n = 279). All participant data came from three completed PRT research studies (described below). The majority of providers identified as female (n = 47; 95.92%) and undergraduate students (n = 28; 57.14%). We operationalized provider education as either postgraduate education (providers n = 16; videos n = 107) for providers with doctoral or M.A. degrees or teaching credentials or undergraduate education (providers n = 33; videos n = 172) for providers who were undergraduate students, or held a B.A., A.A. degree.) Two children were nested within each provider.

Forty-one children participated, ranging in age from 18 to 117 months old (M = 58 months; SD = 27 months). The study team obtained assessment data through record review of the original studies. All children had an ASD diagnosis confirmed with the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 2000; Lord, Luyster, Gotham, & Guthrie, 2012; Lord, Rutter et al., 2012). For 37 children, the team converted ADOS scores to ADOS-2 comparison scores. Four children had only Total ADOS scores available which could not be converted.

All participating children had also completed a cognitive assessment. Fifteen children received the Mullen Scales of Early Learning (MSEL; Mullen, 1995), 25 received the Preschool Language Scales-IV (PLS-4; Zimmerman, Steiner, & Pond, 2002), and one received the Comprehensive Assessment of Spoken Language (CASL; Carrow-Woolfolk, 1999). Moderator analyses used receptive language age equivalence scores, as receptive language scores were available across studies (i.e., MSEL Receptive Language Subdomain and PLS-IV Auditory Comprehension Subscale) and have been found to be related to outcomes for children with ASD (Howlin, Magiati, & Charman, 2009). One child received the CASL and did not have a receptive language age equivalence. See Table 2 for detailed participant characteristics.

Table 2.

Demographics.

	Provider		Child
	n	%	n	%
Gender
Male	2	4.08	33	80.49
Female	47	95.92	8	19.51
Education Level
Postgraduate Education1	16	32.5	–	–
Undergraduate Education2	33	67.25	–	–
Race
White	28	57.14	25	60.98
Unknown/Not Reported	13	26.53	13	31.71
Asian	5	10.20	2	3.92
American Indian/Alaska Native	2	4.08	0	0
More than one race	1	2.04	1	2.44
Ethnicity
Not Hispanic/Latino	32	65.31	17	41.46
Unknown	13	26.53	14	34.15
Hispanic/Latino	4	8.16	10	24.39
Child Assessments			n	M (SD)
ADOS-2	–	–	37	7.54 (1.88)
Receptive Language Age Equivalence (mos)	–	–
MSEL	–	–	15	9.50 (4.91)
PLS-4	–	–	25	31.40 (20.60)

Note.

¹pH.D., M.A. and/or Teaching Credential;

²Undergraduate student, Bachelor’s or Associate Level; ADOS-2 = Autism Diagnostic Observation Schedule-2, Comparison Scores; MSEL = Mullen Scales of Early Learning, Receptive Language Subdomain Age Equivalence; PLS-IV = Preschool Language Scales-4, Auditory Comprehension Subscale Age Equivalence.

2.3. Procedures

2.3.1. Phase 1: establishing behavioral coding definitions

Prior to video coding, in order to ensure PRT component coding definitions were representative of those used by a range of PRT experts, we used Delphi methodology. The Delphi method is a systematic, interactive process wherein a panel of experts provide feedback through one or more rounds of questionnaires (Dalkey & Helmer, 1963). Responses are summarized anonymously to the panel for a second review to identify common language for key indicators across a range of opinion leaders (Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008; Normand, McNeil, Peterson, & Palmer, 1998).

We obtained PRT behavioral coding definitions from research and community programs with PRT expertise (Odom, Collet-Klingenberg, Rogers, & Hatton, 2010). The research team reviewed and combined the definitions, resulting in 21 provider behaviors and six associated child behaviors. Next, 11 experts agreed to participate in the Delphi process. None of these experts participated as providers in the videos coded in Phase 2. All participants held a Ph.D., had ASD experience, and currently worked as clinicians (n = 3) or university-based research faculty (n = 8). Participants included one PRT program developer, seven clinicians/researchers directly trained by a program developer, and three researchers with expertise in naturalistic developmental behavioral interventions more generally. For each behavior, participants rated whether the definition adequately captured the behavior. If the participant responded “no,” they provided specific feedback or alternative wording. Nine participants (82%) fully completed the survey and provided feedback. Two participants did not respond to reminders to complete the survey.

For 56% of the codes, 88–100% of participants endorsed the definition as adequate. For the six child behaviors, 67–88% of participants endorsed each definition as adequate. For the 21 provider behaviors (including PRT components), 22–100% endorsed the definition as adequate. Feedback was reviewed and integrated into the coding definitions. All nine participants reviewed the revised definitions, and all respondents endorsed 100% of the revised codes as adequate (see Table 1). Expert coders used these definitions to determine frequency of each component in the archival data set in Phase 2.

2.3.2. Phase 2: analyzing PRT components and child behaviors

Phase 2 involved using the definitions developed in Phase 1 to code the archival video data.

2.3.2.1. Coder training.

Each undergraduate student coder trained on a single coding system (see Measures) to reliability criteria (80% agreement across behaviors) across three videos. Once coders reached reliability, they completed coding independently. Reliability assessments on independently coded videos for each coder occurred throughout the coding period to prevent drift, and coders completed retraining as needed.

2.3.2.2. Coding process.

Coders used the middle ten minutes of each session, following common convention that this amount of time is adequate for determining intensity of the intervention component use and to avoid including set up and clean up in the coded clip.

2.4. Measures

2.4.1. Archival data sets

The research team reviewed archival video data and selected usable sessions from a pool of 831 videos from three PRT studies conducted between 2000 and 2011. All studies involved providers using PRT with children with ASD. The team screened videos that included a provider who used PRT with at least two different children with ASD over the course of at least 5 sessions. Videos were screened for codability (e.g., adequate sound), which resulted in a total of 278 usable videos (60 videos from Data Set 1, 138 videos from Data Set 2, and 93 videos from Data Set 3) with an average of 5.94 videos (range = 2–14) per provider participant.

2.4.1.1. Archival data set 1.

Set 1 included PRT videos from a randomized trial of PRT (Schreibman & Stahmer, 2014) for 39 children with ASD, ages 20–45 months at intake. The majority of treatment occurred in-home by 30 bachelor’s level and undergraduate student therapists supervised by master’s level Board Certified Behavior Analysts (BCBA). Therapists received training to fidelity in PRT prior to working with any children, and sessions were regularly coded for use of PRT strategies, with feedback and training provided if fidelity scores dropped below 80% correct. Therapists had 1–5 years of experience working with children with ASD and using behavioral interventions.

2.4.1.2. Archival data set 2.

Set 2 included videos from a single subject examination of PRT (Jobin, 2012). Seven undergraduate student therapists trained and supervised by a master’s level BCBA implemented a majority of sessions in the home, with four participants beginning treatment at 22–29 months of age. Therapists received training to fidelity in PRT prior to working with any children, and sessions were regularly coded for use of PRT strategies, with feedback and training provided if fidelity scores dropped below 80% correct. Therapists had little to no previous experience with children with ASD or behavioral intervention.

2.4.1.3. Archival data set 3.

Set 3 videos came from a single subject study examining the use of PRT in schools with 12 teachers working one-on-one with 40 participating students in preschool to 3rd grade (Stahmer, Suhrheinrich, Mandell et al., 2016). Participating teachers received 12 h of interactive workshop training and up to 7 h of in-classroom coaching from the research team. They did not have fidelity requirements in this study. Participating teachers had an average of 6 years of experience teaching students with ASD and most had some previous training in other behavioral strategies, but not prior training in PRT.

2.4.2. Coding systems and behavioral coding definitions

2.4.2.1. Provider behavior – PRT strategies.

Students coded each teaching trial for the presence or absence of PRT strategies (see Table 1). Frequency data were aggregated across each minute to facilitate comparison with child behavior.

2.4.2.2. Child behavior – response to instruction.

Students coded each teaching trial for the presence/absence of child responses, including appropriate responding, goal-directed attempt, incorrect response, inappropriate response, and no response (see Table 1). Data were aggregated across each minute.

2.4.2.3. Child behavior – overall skill use.

Students coded general child skills, including participation, functional object use, communication, responding, and attention. Each behavior was coded on a 1–5 Likert scale on a per minute basis (see Table 1).

2.4.2.4. Reliability of coding.

For each coding system, we coded 30% of videos (randomly selected across coders) twice for analysis of inter-rater reliability. We used Cohen’s kappa for categorical variables, and percent agreement for continuous (Table 1).

2.5. Data analysis

All analyses were run utilizing MPlus software. As recommended by Enders (2010), the full-information maximum likelihood (FIML) estimation approach to missing data was used for all analyses.

Due to the nested structure of the data (i.e., children [level-1] nested within providers [level-2]), multilevel regression models were used to determine the relationships between intensity of PRT components and (a) child behavior and (b) provider characteristics. Possible clustering effects were initially estimated by examining intraclass correlation coefficients (ICCs). The ICCs for each provider and child behavior indicated that there was significant clustering at the provider level, and thus, multilevel examination of the data was warranted and utilized (see Table 3).

Table 3.

Descriptive Statistics – Provider Strategies and Child Behaviors.

Provider Strategies
PRT Components/Strategies	M	SD	ICC
Student attention	2.92	1.14	.24
Clear cue	2.97	1.15	.24
Dev. appropriate cue	2.98	1.15	.24
Novel cue	1.70	1.08	.32
Shared control	2.94	1.15	.25
Maintenance tasks	0.73	0.80	.31
Acquisition tasks	2.25	1.06	.28
Direct reinforcement	1.13	0.60	.22
Reinforcement of attempts	1.04	0.57	.21
Child Behaviors	M	SD	ICC
Participation	4.24	0.76	.44
Functional object use	4.88	0.21	.14
Communication	3.07	1.10	.30
Responding	3.35	0.96	.43
Attention	4.08	0.61	.38
Appropriate responding	0.98	1.02	.38
Goal-directed attempt	1.28	0.92	.32
Incorrect response	0.09	0.13	.10
Inappropriate response	0.16	0.19	.17
No response	1.11	0.74	.25

Note. The intensity of components and child behaviors were coded as the frequency per one-minute interval, with the exception of participation, functional object use, communication, responding and attention, which were coded as one rating per one-minute interval.

For Aim 1, we examined provider use of each PRT component in relation to within-session child behavior during each minute. We utilized multilevel regression models (i.e., children nested within providers) with child behaviors (i.e., rating/percentage per one-minute interval) as outcome variables and intensity of components (i.e., the frequency of each component per one-minute interval) as predictors.

For Aim 2, a latent profile analysis (LPA) (with provider as a cluster variable to account for nesting) was used to group each provider video into different profiles based on the frequency with which they used particular PRT components. Investigation of model fit was used to determine the appropriate number of profiles. Specifically, we examined Akaike Information Criterion (AIC; Akaike, 1974) Bayesian Information Criterion (BIC; Schwarz, 1978) and the change in deviance statistic to determine the best fitting model to the data (Raudenbush & Bryk, 2002). Once groups were established, we examined profiles using multilevel regression models to determine whether profiles related to child behavioral response. For this analysis, child behavior during session served as level-1 outcome variables and provider profile group served as a level-2 predictor variable.

For Aim 3, we examined relationships between provider and child characteristics and intensity of PRT components use by creating relevant interaction terms and running multilevel regression models. Participants without relevant assessments (e.g., ADOS) were excluded from these analyses.

3. Results

3.1. Description of PRT component use and coding reliability

Table 3 shows the mean and standard deviation for PRT components and child behavior using average rate of use per minute. Descriptively, providers tended to use antecedent strategies more often than consequence strategies. Providers presented acquisition tasks over twice as often as maintenance tasks. The children tended to be responsive and participate in the sessions with good attention. Children exhibited few inappropriate responses but often failed to respond. ICCs ranged from .10 to .44, indicating significant dependencies for the target study variables and supporting the use of a multi-level approach.

3.2. AIM 1: relationship of individual provider behaviors and within-session child behavior

Increased use of several of the antecedent strategies was associated with within-session child behaviors (see Table 4). All of the antecedent strategies, including (1) gaining a child’s attention before presenting the instruction, (2) presenting a clear cue, (3) presenting a developmentally appropriate cue, (4) shared control, (5) and targeting both maintenance and acquisition tasks had positive associations with a majority of the child behaviors including participation, communication, responsivity, and attention.

Table 4.

Aim 1: Relationship of Key Component Use and Child Behavior.

	CPRT Component
	Student Attention	Clear Cue	Dev. App Cue	Shared Control	Turn Taking	Maint & Acq Tasks	Direct Reinf	Reinf Atts
Child Behavior
Participation	B = .24, p < .001	B = .24, p < .001	B = .24, p < .001	B = .25, p < .001	B = −.40, p = .007	B = .39, p < .001	B = .32, p = .013
Functional object use					B = −.21, p = .024
Communication	B = .32, p = .004	B = .31, p = .005	B = .31, p = .004	B = .32, p = .003	B = −1.23, p = .005	B = .63, p < .001
Responsivity	B = .27, p = .002	B = .27, p = .003	B = .28, p = .001	B = .30, p < .001		B = .62, p < .001	B = .54, p = .001
Attention	B = .18, p = .002	B = .18, p = .003	B = .19, p = .002	B = .19, p = .001		B = .38, p < .001	B = .30, p = .009

Note: For ease of presentation only significant results are presented.

Consequence strategies showed a different pattern. Provider use of direct reinforcement was associated with higher levels of child participation, responsivity, and attention, but not communication. The use of turn taking was negatively associated with child participation, object use, and communication. Reinforcement of attempts at correct responding was not associated with any child behaviors.

3.3. AIM 2: relationships between combinations of PRT components and within-session child behavior

3.3.1. Latent profile analysis

The analysis examined whether providers could be grouped based on the frequency of PRT component use. Examination of the fit criteria suggested that a 5-class model was the best fit (AIC = 700,040.97; BIC = 70,313.04; Entropy = 0.976). The AIC and BIC values for the 5-class model were the lowest of all models (for all other class models AIC ranged from 70327.13 to 73735.06 and BIC values ranged from 70551.89 to 73817.87).

The five profiles identified represented varying use of PRT components (see Table 5). For videos fitting within the Moderate Antecedent Frequency Profile (LP1; 12.3%), providers used most of the antecedent components with moderate frequency and rarely used direct reinforcement and turn taking. In the Very Low Frequency Profile (LP2; 36.3%) videos, providers used most of the components with a relatively low frequency, indicating low intensity. In the Low Frequency Profile (LP3; 31.4%) videos, providers used most of the components at a low to moderate rate, except they had higher use of turn taking. In the Moderate Frequency Profile (LP4; 9.6%) videos, providers used most strategies at moderate rates and direct reinforcement strategies and turns at relatively high rates. In the High Frequency Profile (LP5; 10.4%) videos, providers used a majority of strategies at the highest rates with the exception of direct reinforcement, which was used moderately, and turn taking, which was quite low.

Table 5.

Aim 2: Provider Groups Based on Frequency of Component Use.

	Latent Profile 1 Moderate Antecedent Group	Latent Profile 2 Very Low Frequency Group	Latent Profile 3 Low Frequency Group	Latent Profile 4 Moderate Frequency Group	Latent Profile 5 High Frequency Group
% Observation in Group	12.3	36.3	31.4	9.6	10.4
PRT Component	Mean	Mean	Mean	Mean	Mean
Attention	3.73 (0.11)	1.80 (0.11)	2.88 (0.14)	3.69 (0.20)	5.50 (0.19)
Clear Cue	3.77 (0.11)	1.82 (0.12)	2.97 (0.14)	3.71 (0.20)	5.58 (0.19)
Devel Appropriate Cue	3.78 (0.11)	1.83 (0.12)	2.98 (0.14)	3.73 (0.21)	5.58 (0.19)
Shared Control	3.78 (0.11)	1.81 (0.11)	2.90 (0.12)	3.72 (0.21)	5.56 (0.19)
Maintenance Task	1.69 (0.21)	0.48 (0.08)	0.39 (0.08)	1.69 (0.27)	1.83 (0.43)
Direct Reinforcement	0.49 (0.05)	0.99 (0.11)	1.12 (0.12)	2.13 (0.09)	1.06 (0.27)
Turn taking	0.05 (0.02)	0.50 (0.06)	0.42 (0.07)	0.49 (0.08)	0.13 (0.05)

Note: PRT components coded as frequency per one-minute interval.

3.3.2. Profile use and child within-session skills and behaviors

Next, we examined the relationships between the profiles and each of the child behaviors (see Table 6). The latent profile variable was dummy coded such that the Very Low Frequency Profile (LP2) served as the referent profile. Thus, each parameter estimate can be interpreted relative to the level of the outcome variable for the Very Low Frequency Profile (LP2; e.g., relative to the LP2, LP5 was associated with a 0.602 increase in child participation). The results are summarized in Table 6. Generally, use of a majority of strategies at high or moderate frequency (Moderate Frequency and High Frequency profiles) was associated with better child behaviors and more appropriate responding. However, very high use of the antecedent strategies and limited use of direct reinforcement and turn taking also increased inappropriate and no responding. Relative to the Very Low Frequency Profile (LP2), the use of strategies at a moderate frequency (Moderate Antecedent Frequency Profile; LP1) was associated with better communication and appropriate responding and with increased goal-directed attempts. Finally, in comparison to the Very Low Frequency Profile (LP2), the Low Frequency Profile (LP3) was associated with an increase in goal directed attempts, and high rates of child inappropriate and nonresponding.

Table 6.

AIM 2. Relationship between Group Membership and Child Skills and Behaviors.

	β0 Intercept	βLC1	βLC3	βLC4	βLC5
Child Behavior
Participation	4.128***	0.479	−0.157	0.619*	0.602*
Functional object use	4.849***	0.088	0.004	0.040	0.102
Communication	2.874***	1.145**	−0.121	0.600	0.598
Responding	3.221***	0.680*	−0.274	0.893*	0.675*
Attention	4.025***	0.246	−0.176	0.507*	0.447*
Appropriate responding	0.513***	1.960***	−0.001	1.456***	1.310***
Goal-directed attempt	0.789***	0.296	0.714***	0.816*	1.378***
Incorrect response	0.044*	0.202***	−0.019	0.089	0.231***
Inappropriate response	0.179***	−0.017	0.130**	0.015	0.053
No response	0.727***	0.161	0.622***	−0.074	1.451***

Note: LC1: Class 1 – Moderate Antecedent Group; LC3: Class 3 – Low Frequency Group; LC4: Class 4 – Moderate Frequency Group; LC5: Class 5 – High Frequency Group.

^*p < .05.

^**p < .01.

^***p < .001.

3.4. AIM 3: evaluate relationships between provider and child characteristics and intensity of PRT component use

We looked at mean group differences in intensity for each component using multilevel regression models. We used the overall rating for the entire video to facilitate examination of overall performance with child and provider characteristics.

3.4.1. Provider education

Providers in the postgraduate education group had significantly higher use of novel cues (mean difference = 26.26; B = 26.24, p < .001). Those in the undergraduate education group had significantly higher means on direct reinforcement (mean difference = 10.25; B = − 11.97, p < .05), and shared control (mean difference = 11.95; B = −10.23, p < .001).

Providers with postgraduate education were distributed across profiles such that 21% were in the Moderate Antecedent Frequency Profile (LP1), 24% were in the Very Low Frequency Profile (LP2), 29% were in the Low Frequency Profile (LP3), 12% were in the Moderate Frequency Profile (LP4), and 14% were in High Frequency Profile (LP5). Providers with undergraduate education, on the other hand, were more likely to be in LP2 and LP3 (44% and 34% respectively), with very few in LP1, LP4 or LP5 (6%, 6%, and 10%, respectively). These differences were not statistically significant overall (χ² = 8.64, p = .07).

3.4.2. Moderation analyses

Child characteristics (i.e., autism severity, receptive language) did not moderate the relationship between child behavior and provider use of PRT components.

4. Discussion

This project demonstrates a novel method for identifying key components of a multi-component EBP associated with child behavior. This is a first step toward systematic identification of key components of ASD intervention. Clear associations between provider use of individual components and within-session child behavior provide direction for understanding the mechanism of action of interventions like PRT. Additionally, examining combinations of strategies and how they relate to child behavior leads to a better understanding of the intensity of strategy use associated with change in the behavior of children with ASD. One of the major challenges of this specific project is that, due the nature of the data, only within-session behaviors could be examined. Therefore, results cannot be used to determine the predictive value of the use of each component on child outcomes. Still, these data provide evidence of how variability in intensity of PRT component use is related to provider characteristics and child behavior.

When examining the PRT components individually, there appears to be differential association of specific components with child behaviors, which may support individualization of intervention to child characteristics. Prospective studies can determine whether certain components can be emphasized based on the child behaviors being targeted. For example, providing novel cues as a way to broaden attention was associated with increased child responding, but with less child attention. It is possible that children are more likely to respond when cues are presented in new ways but that this strategy may be most effective for children who already have the skills needed to attend to the instruction or lesson. It is also possible that providers used novel cues more often when children were already responsive.

Data suggest that intensity of some components may differentially relate to child behavior. Higher provider use of maintenance tasks, for example, was associated with better child behavior overall. It is our hypothesis that when children receive a higher number of maintenance tasks, they are more likely to respond correctly and receive reinforcement, and therefore have fewer challenging behaviors. One application of this finding is individualize the intervention by varying the rate of maintenance tasks based on the child’s level of challenging behavior. However, it is unclear how a high ratio of maintenance-to-acquisition tasks affects child learning rates. That is, if a provider presents only tasks the child can complete easily, behavior will likely be good, but the child may not learn new skills. Further research on ideal ratios based on child and task characteristics will be important.

Similarly, in the area of consequence components, increased tangible reinforcement of child goal-directed attempts at correct responding was associated with fewer incorrect responses. This is consistent with early studies of this component indicating that children are more motivated to try to respond when a task is difficult if they are regularly rewarded for good trying (Koegel, O’Dell, & Dunlap, 1988). Again, we cannot draw a causal relationship, and it is possible that providers were employing differential reinforcement strategies; however, our data suggest that for young children with limited language skills, praise alone is not associated with responding, attention, or participation.

Understanding the association between key components of PRT and child behavior may also assist with the development of more effective training programs. For example, gaining a child’s attention before providing an instruction is, not surprisingly, associated with a variety of positive child behaviors. Fortunately, this specific component is intuitive for many providers (Stahmer, Suhrheinrich, & Schreibman, 2012), and providers with postgraduate education tend to use this skill often even prior to training in PRT specifically (Suhrheinrich et al., 2013). Gaining attention may serve as a foundational skill for other provider behaviors, so ensuring that young therapists have this skill may support fidelity of the intervention. Providing clear and appropriate cues was not significantly associated with any child behaviors. However, it was used often by providers, and it is possible that it varies with attention. Disentangling these issues could be a direction for future research and may lead to adaptations for clinical practice (i.e., less emphasis on the specificity of the cue delivered).

A next step in this process may be to more closely examine the specific components of PRT in an experimental setting to determine how intensity of each component relates to child outcomes. Preliminary studies indicate that the multiple cues component, found to be challenging for teachers in community classroom settings, could be greatly simplified, and is not always developmentally appropriate (Reed, Stahmer, Suhrheinrich, & Schreibman, 2013; Rieth, Stahmer, Suhrheinrich, & Schreibman, 2015). The turn taking component of shared control was found to impact specific tasks (verbal and object imitation) more than other tasks (complex speech), and to affect performance differentially based on child language level (Rieth, Schreibman, Stahmer, Kennedy, & Ross, 2013). These results are preliminary, but may lead to specific recommendations of when to use these components and when it is appropriate to leave them out of PRT, or how to modify them.

Although preliminary, our profile analysis of PRT suggests of the need for relatively intensive use of most components to maximize the effect of the intervention. That is, use of the majority of components at high frequencies was associated with high child engagement, responsivity, attention to task, goal directed attempts and correct responding. It seems that high frequency use of antecedent strategies can help providers overcome lower use of reinforcement strategies; however, this may also be linked to lower responsivity. Alternatively, moderate use of a majority of antecedent strategies coupled with high use of direct reinforcement is also linked to positive responding. The low frequency profile was associated with goal directed attempts and also with poor responding. Overall, the minimal use of most strategies did not seem especially beneficial. These data help us understand that increasing use of some, but not all, PRT strategies is still beneficial for children with ASD; however, higher frequency use of the majority of strategies is likely to lead to better student performance. The LPA should be considered preliminary. Given the limited sample size at the provider level, the LPA was conducted at the observation level. The fact that these observations were nested within providers was accounted for, however, by using providers as a clustering variable.

Providers with undergraduate education (and likely less experience) were much more likely to use the strategies less intensely even though they had relatively high levels of supervision. This may have implications for training and ongoing coaching in EBPs that require adaptation to child behavior. These EBPs can be challenging for even experienced providers (Mandell et al., 2013) and therefore, a longer apprenticeship period may be required. Use of the antecedent strategies alone at high frequencies was associated with better responding than use of all the strategies at low intensity; thus, emphasis may be placed on these components in on-going training efforts. However, for maximal student outcomes, future research should examine ways to improve the use of consequence strategies as well.

4.1. Limitations

The greatest limitation of this exploration is the concurrent nature of the data. That is, we cannot draw any causality regarding the effect of the use of specific strategies and child outcomes. It is possible that providers are using specific strategies based on child behavior rather than to affect it. Therefore, additional research is needed to examine the use of specific strategies and longer-term outcome in children as well as systematic, prospective studies of combinations of strategies. Additionally, the archival studies did not provide details regarding provider experience with ASD, behavioral interventions and PRT specifically. These factors may be very important for fidelity of the intervention and may affect overall quality of treatment. Finally, the range of child age and functioning level is relatively small in this sample and therefore results cannot be generalized to a broader range of individuals with ASD.

4.2. Conclusions

Careful examination of key components in complex interventions for individuals with autism is a useful way to clarify the mechanisms of action of the intervention. This may allow for more systematic individualization of interventions and simplification of interventions for use in community settings. In addition, key component association with child behavior across interventions may lead to consolidation of the types of strategies needed to affect better outcomes for children with ASD.

What this paper adds.

These analyses represent a first step to inform strategies for associating key components of evidence-based practice (EBP) for children with autism spectrum disorder (ASD) with child responsivity to intervention. This paper targets one naturalistic developmental behavioral intervention, Pivotal Response Training (PRT) to demonstrate methodological process and potential outcomes; however, we envision these analytical methods as applicable to interventions more broadly. Specific to PRT, analyses support recommendations for PRT adaptation based on differential association of individual component use with child behaviors. Data highlight the potential for individualization of intervention components based on child behavior. Additionally, these data add to the literature by identifying the need to support providers with less education and experience to increase the intensity of intervention use.

More generally, the project supports the use of careful examination of archival data from treatment studies of a multi-component intervention to evaluate relationships between use of intervention components and child responsivity and behavioral outcomes as well as to evaluate relationships between provider characteristics and use of intervention components. This model is generalizable to other complex interventions and may lead to a better understanding of key components of these interventions using existing data sets. This is a relatively low cost, low risk method for identifying key components of an intervention associated with child responsivity. Additionally, more studies of this type may lead to a set of common components of effective intervention for children with ASD and move the field away from packaged interventions that may be challenging to integrate in community settings.