Pivotal response treatment: A study into the relationship between therapist characteristics and fidelity of implementation

Abstract

In this study, we investigated the relationship between fidelity of pivotal response treatment implementation and therapist characteristics, such as therapist personality, therapist–child relationship, therapist attitude toward evidence-based practices, and therapist experience. We also explored whether child age and autism symptom severity were related to pivotal response treatment fidelity. Participants were 41 level III certified pivotal response treatment therapists who recorded three 10-min pivotal response treatment sessions and completed four questionnaires to measure therapist characteristics. Results indicated that therapists’ attitude toward evidence-based practices, specifically openness to innovation, and therapists’ experience with pivotal response treatment significantly predicted fidelity of pivotal response treatment implementation. Cross-validation methods largely confirmed these findings. Therapist personality, therapist–child relationship, and child characteristics were not significantly related to pivotal response treatment fidelity. Implications for clinical practice and directions for future research are discussed.

Lay abstract

Pivotal response treatment is a naturalistic behavioral intervention that teaches pivotal skills to children with autism spectrum disorder to produce widespread gains in other skills. Although most children with autism spectrum disorder benefit from pivotal response treatment, intervention outcomes vary considerably among children. Fidelity of intervention implementation (i.e. the extent to which an intervention is implemented as intended) may affect intervention outcomes. In this study, we studied the relationship between fidelity of pivotal response treatment implementation and therapist characteristics, such as therapist personality, therapist–child relationship, therapist attitude toward evidence-based practices, and therapist experience. We also explored whether a child’s age and autism symptom severity were related to pivotal response treatment fidelity. Participants were 41 pivotal response treatment therapists who videotaped three pivotal response treatment sessions and completed four questionnaires to measure therapist characteristics. This study found that therapists’ openness to innovation and their experience with pivotal response treatment predicted fidelity of implementation. Therapist personality, therapist–child relationship, and child characteristics were not related to pivotal response treatment fidelity. The results of this study emphasize that it is important (1) to target therapists’ attitudes toward innovation prior to or during training in pivotal response treatment and (2) to provide therapists with ongoing supervision and feedback after training to increase fidelity of implementation and thus to improve intervention outcomes for children with autism spectrum disorder.

With the increase in the number of children diagnosed with autism spectrum disorder (ASD), it has become imperative to provide effective and efficient interventions to meet increased demands for treatment services (e.g. Baio et al., 2018; Elsabbagh et al., 2012). Interventions based on applied behavior analysis (ABA) or behavioral interventions are recognized as most effective in teaching functional skills and reducing maladaptive behaviors in children with ASD (e.g. Odom, Collet-Klingenberg, Rogers, & Hatton, 2010; Reichow, 2012; Wong et al., 2015). Behavioral interventions include both structured or analog interventions that aim to teach single responses in a structured one-to-one teaching setting (e.g. discrete trial teaching [DTT]) and naturalistic interventions that focus on teaching developmentally appropriate and prerequisite skills in a variety of natural settings (e.g. incidental teaching, Early Start Denver Model, and pivotal response treatment (PRT); Cowan & Allen, 2007; Lang, Hancock, & Singh, 2016; Roane, Fisher, & Carr, 2016; Schreibman et al., 2015).

PRT is a naturalistic behavioral intervention that targets pivotal skills in children with ASD to produce widespread gains in other—often untargeted—skills, using a combination of motivational techniques (Koegel, Ashbaugh, & Koegel, 2016). A systematic review found evidence for the effectiveness of PRT for increasing self-initiations in children with ASD and producing collateral improvements in language, communication, play, affect, and maladaptive behavior (Verschuur, Didden, Lang, Sigafoos, & Huskens, 2014). However, intervention outcomes varied considerably among children. Research on variables that predict outcomes of PRT is still limited, but some studies suggest that higher cognitive and expressive language skills at baseline, positive affect, appropriate toy contact, decreased social avoidance, and decreased stereotyped or repetitive vocalizations are associated with a positive response to PRT in preschool children with ASD (Fossum, Williams, Garon, Bryson, & Smith, 2018; Schreibman, Stahmer, Barlett, & Dufek, 2009; Sherer & Schreibman, 2005; Smith, Flanagan, Garon, & Bryson, 2015).

In addition to child characteristics, fidelity of intervention implementation may affect intervention outcomes, with higher levels of fidelity resulting in better outcomes (e.g. DiGennaro Reed & Codding, 2014; Durlak & DuPre, 2008; Pellecchia et al., 2015). Fidelity of implementation, also known as treatment integrity or procedural fidelity, refers to the extent to which an intervention is implemented as intended and encompasses several aspects: (a) adherence, the degree to which therapists reliably and accurately implement intervention procedures as prescribed, (b) exposure or dosage, the amount of intervention delivered, (c) differentiation, the extent to which only prescribed intervention procedures are implemented, and (d) competence, the degree to which prescribed intervention procedures are delivered well (e.g. Dane & Schneider, 1998; Perepletchikova, Treat, & Kazdin, 2007; Schoenwald et al., 2011; Wainer & Ingersoll, 2013). Adherence is the aspect of fidelity that has been measured most often and has the strongest relationship with intervention outcomes (Durlak & DuPre, 2008; Schoenwald et al., 2011). Therefore, this study focused on adherence as a measure of fidelity. Generally, a fidelity of implementation level at or greater than 80% is considered acceptable (e.g. Reichow, Volkmar, & Cicchetti, 2008).

Fidelity of implementation is affected by different factors at multiple levels (e.g. Aarons, Hurlburt, & Horwitz, 2011; Damschroder et al., 2009; Dingfelder & Mandell, 2011). The science of dissemination and implementation has developed a variety of models to describe these factors to promote consistent and effective implementation of research findings in practice (e.g. Aarons et al., 2011; Dingfelder & Mandell, 2011). Most of these models divide the process of implementation in several phases, including exploration, adoption or preparation, implementation, and sustainment or maintenance (Aarons et al., 2011; Dingfelder & Mandell, 2011). Factors that may impact implementation outcomes, such as fidelity, during these different phases include sociopolitical factors (e.g. legislation, policies, and funding) and organizational factors (e.g. organizational culture or climate, implementation leadership, and the quality and amount of training or ongoing supervision; Aarons et al., 2011; Dillenburger, 2017; Reid & Fitch, 2011), but individual factors, such as therapist or child characteristics, may also affect fidelity of implementation (e.g. Aarons et al., 2011; Perepletchikova & Kazdin, 2005).

Research on the relationship between therapist characteristics and fidelity of PRT implementation is currently lacking. Two studies investigated this relationship for DTT (Peters-Scheffer, Didden, Korzilius, & Sturmey, 2013; Symes, Remington, Brown, & Hastings, 2006). Furthermore, the association between personal characteristics and fidelity of implementation or job performance in general has been explored in various settings and for various jobs (e.g. Barrick & Mount, 1991; Barrick, Mount, & Judge, 2001; Beidas et al., 2015; Judge & Zapata, 2015; Lange, Van der Rijken, Busschbach, Delsing, & Scholte, 2017; Pagoto et al., 2007; Reding, Chorpita, Lau, & Innes-Gomberg, 2014; Tschuschke et al., 2015; Weck, Grikscheit, Jakob, Höfling, & Stangier, 2015).

Personality is a therapist characteristic that may be associated with fidelity of implementation. Peters-Scheffer et al. (2013) found that therapists with higher levels of openness to experience demonstrated lower procedural fidelity of DTT. Research on the association between personality traits and job performance in general has indicated that openness to experience is strongly and positively related to performance in jobs that require creativity (Judge & Zapata, 2015). Furthermore, a positive relationship between conscientiousness and job performance has been reported for different occupational groups (Barrick & Mount, 1991; Barrick et al., 2001).

In addition to therapist personality, therapist–child relationship might affect the fidelity of PRT implementation. Symes et al. (2006) reported that therapists considered competent, likable, motivated, and compliant children to facilitate implementation of early intensive behavioral intervention (EIBI), whereas challenging behavior or a lack of progress hindered EIBI implementation, which may suggest that the therapist–child relationship is associated with the fidelity of implementation. Also, several studies have reported that a better therapeutic alliance is related to higher treatment integrity of psychosocial treatments (e.g. Tschuschke et al., 2015; Weck et al., 2015). In contrast, Peters-Scheffer et al. (2013) found that therapists who perceived the relationship between themselves and the child as more positive displayed lower procedural fidelity of DTT.

Other therapist characteristics that might affect procedural fidelity are therapists’ attitudes toward evidence-based practices (EBPs) and therapists’ experience. Several studies have indicated that therapists’ attitudes toward EBPs, particularly openness to the use of EBPs and perception of EBP appeal, predict fidelity of implementation (e.g. Beidas et al., 2015; Pagoto et al., 2007; Reding et al., 2014). Negative attitudes toward EBPs may be a barrier to correct implementation, whereas positive attitudes may facilitate implementation. In recent years, the number of studies examining the relationship between attitudes toward EBPs and implementation outcomes has increased (e.g. Brookman-Frazee & Stahmer, 2018; Locke et al., 2019). Also, some studies found an association between therapist experience and fidelity of implementation, with highly experienced therapists displaying higher levels of fidelity (e.g. Lange et al., 2017). However, other studies reported a negative relationship between therapist experience and fidelity of implementation (e.g. Tschuschke et al., 2015).

Child characteristics may also be related to fidelity of implementation (Perepletchikova & Kazdin, 2005; Symes et al., 2006). Numerous studies have demonstrated that child characteristics predict outcomes of behavioral interventions, including PRT (e.g. Ben-Itzchak & Zachor, 2011; Fossum et al., 2018; Granpeesheh, Dixon, Tarbox, Kaplan, & Wilke, 2009; Perry, Blacklock, & Geier, 2013; Smith et al., 2015; Vivanti, Prior, Williams, & Dissanayake, 2014). The impact of child characteristics on implementation fidelity has been examined far less frequently, but a significant relationship between classroom type (preschool vs elementary) and teachers’ fidelity of PRT has been reported, suggesting that child age may be related to implementation fidelity (Suhrheinrich, Dickson, Rieth, Lau, & Stahmer, 2016). In addition, child age and autism symptom severity may affect teacher behavior directed toward preschoolers with ASD in inclusive classrooms (Irvin, Boyd, & Odom, 2015). Also, some studies reported that fidelity decreased with increased levels of challenging behavior in children with developmental disabilities (Carr, Taylor, & Robinson, 1991; McConnachie & Carr, 1997).

The relationship between fidelity of implementation and intervention outcomes for children with ASD stresses the importance of identifying factors that predict fidelity. However, due to differences between interventions, it is yet unclear whether findings from previous studies can be generalized to PRT. PRT is a naturalistic behavioral intervention and compared to more structured behavioral interventions, such as DTT or EIBI, intervention sessions are more loosely structured, and teaching episodes (i.e. opportunities or trials) are child initiated (Cowan & Allen, 2007; Delprato, 2001; Schreibman et al., 2015). This involves following the child’s lead, incorporating child choice, and presenting opportunities in the context of child-preferred activities, which necessitates therapists’ creativity and flexibility and thus higher levels of openness to experience (Koegel, 2011). In addition, PRT involves contingent and natural reinforcement upon the child’s response or reasonable attempt to strengthen response–reinforcement contingencies, which demands therapists’ accuracy and preciseness and therefore higher levels of conscientiousness (Koegel et al., 2016; Koegel & Koegel, 2006). In naturalistic behavioral interventions, such as PRT, teaching and learning are embedded in activities that contain meaningful social interactions (Schreibman et al., 2015). A positive interaction or relationship between therapist and child may thus facilitate both teaching and learning. In a positive relationship, the therapist’s effort to elicit a response from the child is more likely to be reinforced by the child (i.e. the child correctly responds to the therapist’s opportunity) and thus implementing PRT with this child will be highly reinforcing for the therapist, which may result in higher levels of fidelity (Detrich, 1999; Symes et al., 2006). In summary, the correct implementation of several components of PRT seems to require certain therapist characteristics.

Therefore, this study aimed to investigate the relationship between therapist characteristics and fidelity of PRT implementation in treatment facilities for children with ASD in the Netherlands. In addition, the impact of child characteristics on the fidelity of PRT implementation was explored. First, we expected that higher levels of openness and conscientiousness would predict higher fidelity of PRT implementation. Second, we expected that a more positive therapist–child relationship would predict higher PRT fidelity. Third, we hypothesized that a more positive attitude toward EBPs would predict higher fidelity of PRT implementation, in particular higher levels of openness to the use of EBPs and a more positive perception of EBP appeal. Fourth, we expected that higher levels of experience would predict higher PRT fidelity. Finally, we hypothesized that the fidelity of PRT implementation would be higher for therapists who implemented PRT with younger children and children with less severe ASD symptoms.

Method

Participants and setting

Therapists were recruited at treatment facilities for children with ASD in the Netherlands and were eligible for participation in this study if they (a) were certified at PRT level III (i.e. demonstrated fidelity of PRT implementation at or greater than 80% with three children across three different activities) prior to participation in this study and (b) provided PRT at least monthly. Only certified therapists were included to ensure that all therapists had the same amount of training in PRT, so that variation in fidelity could not be explained by variation in training in PRT.

A total of 41 therapists participated in this study. They were working as direct-care staff member (n = 30, 73%), psychologist (n = 8, 20%), teacher (n = 1, 2%), speech language pathologist (n = 1, 2%), or researcher (n = 1, 2%). Therapists (39 females) had a mean age of 35.0 years (standard deviation (SD) = 7.01, range = 25–52). On average, they had 10.0 years of experience in working with children with ASD (SD = 4.88, range = 1–22) and 18.2 months of experience in providing PRT (i.e. number of months after obtaining PRT level III certification; SD = 15.5, range 0–68). One therapist participated in this study immediately after obtaining PRT level III certification and thus this therapist’s experience in providing PRT was 0 months. Therapists provided PRT daily (n = 23, 56%), weekly (n = 16, 39%), or monthly (n = 2, 5%). Each therapist was asked to select a child with ASD to form a dyad. A total of 41 children (33 males) participated in this study and all had a clinical diagnosis of ASD according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000) or Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5; American Psychiatric Association, 2013) criteria. For 39 children, the clinical diagnosis was confirmed by the score on the Social Responsiveness Scale (SRS; Roeyers, Thys, Druart, De Schryver, & Schittekatte, 2011). Two children had a score just below the threshold for ASD. Mean score on the SRS was 78.9 (SD = 10.3, range = 60–108). Children were aged between 3 and 15 years (mean = 8.68, SD = 3.41).

Informed consent was obtained from each therapist, the parents of each child, and children older than 12 years. The study was approved by the Ethics Committee of the Faculty of Social Sciences of the Radboud University, Nijmegen, The Netherlands (ECG2013-1304-100a).

Procedures and measures

Therapists were instructed to conduct three PRT sessions with the child during an age-appropriate everyday activity, such as playing with construction toys, drawing, or baking, and to record these PRT sessions using a video camera. They were instructed to record sessions that (a) lasted at least 10 min and (b) were recorded in a one-to-one situation. In addition, therapists were asked to complete three questionnaires.

Personality traits

The Dutch version of the NEO Five-Factor Inventory (NEO-FFI) was used to measure openness to experience and conscientiousness (Hoekstra, Ormel, & De Fruyt, 2007). Both subscales consisted of 12 items. Therapists rated each item on a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Raw subscale scores were calculated (range = 5–60), with higher scores indicating higher levels of openness or conscientiousness. Cronbach’s alphas for openness to experience and conscientiousness were 0.68 and 0.76, respectively, indicating questionable to acceptable reliability (Hoekstra et al., 2007). Test–retest reliability was good for both subscales (i.e. r > 0.80) and construct validity was acceptable (Hoekstra et al., 2007).

Therapist–child relationship

Therapists completed the Dutch version of the Student–Teacher Relationship Scale (STRS) to measure the therapist–child relationship (Koomen, Verschueren, & Pianta, 2007). The STRS is a 28-item questionnaire consisting of three subscales: (a) closeness (i.e. positive aspects of the therapist–child relationship, such as warmth and open communication), (b) conflict (i.e. negative aspects of the therapist–child relationship, such as disharmonious interactions), and (c) dependency (i.e. negative aspects of the therapist–child relationship related to possessive behavior). Items were rated on a five-point scale ranging from 1 (definitely does not apply) to 5 (definitely applies). Raw total scores were calculated (range = 28–140). High raw total scores indicate a relatively low level of conflict and dependency, a relatively high level of closeness, and an overall positive therapist–child relationship. Cronbach’s alpha for the total scale was 0.89 and the test–retest reliability coefficient for the total scale was 0.70, indicating acceptable to good reliability (Koomen, Verschueren, van Schooten, Jak, & Pianta, 2012; Koomen et al., 2007). Construct validity was good (Koomen et al., 2012). Although the STRS has been developed for children aged 3–12 years, reliability and construct validity were also satisfactory for children older than 12 years (Dekker, 2008).

Attitude toward evidence-based interventions

The Dutch translation of the Evidence-Based Practice Attitude Scale (EBPAS) was used to assess the therapists’ general attitudes toward EBPs (Aarons, 2004; Van Sonsbeek et al., 2015). The EBPAS is a 15-item questionnaire consisting of four subscales: (a) appeal (i.e. the extent to which a therapist would adopt an EBP if it was intuitively appealing or was being used by colleagues who were positive about it), (b) requirements (i.e. the extent to which a therapist would adopt an EBP if it was required by a supervisor or an agency), (c) openness to innovation (i.e. the extent to which a therapist is generally willing to use or try new interventions and EBPs), and (d) divergence (i.e. the extent to which a therapist perceives EBPs as not useful and less important than clinical experience). Therapists rated items on a five-point scale ranging from 0 (not at all) to 4 (to a very great extent). Raw subscale and total scale scores were calculated, with higher scores indicating more positive attitudes toward the adoption of EBPs. Cronbach’s alphas for appeal and openness to innovation were 0.75 and 0.77, respectively, with a total scale alpha of 0.72. This indicates acceptable reliability (Van Sonsbeek et al., 2015). Support for the construct and content validity of the EBPAS has been provided by several studies (e.g. Aarons et al., 2010; Van Sonsbeek et al., 2015).

Fidelity of PRT implementation

Fidelity of PRT implementation (i.e. adherence to prescribed intervention procedures) was measured using partial interval recording (Cooper, Heron, & Heward, 2013). Ten minutes of each videotape were coded by a naïve observer. If videotapes lasted more than 10 min, 10 min in the middle of the videotapes were observed. If videotapes lasted less than 10 min, the entire videotape was observed. Videotapes were divided into 10 1-min intervals. Whereas fidelity is usually determined by scoring each PRT component separately (e.g. Nefdt, Koegel, Singer, & Gerber, 2010; Robinson, 2011; Suhrheinrich, Rieth, Dickson, Roesch, & Stahmer, 2019), this study used a measure of PRT fidelity that includes a sequence of correctly implemented PRT components that constitute a three-term contingency as proposed by Huskens, Reijers, and Didden (2012), because these three-term contingencies increase the frequency of responding and thus enhance learning (e.g. Albers & Greer, 1991; Cooper et al., 2013). The following components of fidelity were recorded per interval: (a) following the child’s lead, (b) implementing three-term contingency learning trials, and (c) interspersing maintenance and acquisition tasks. Operational definitions of these components and subcomponents are presented in Table 1.

Table 1.

Operational definitions of components of fidelity of PRT implementation.

Component	Operational definition
1. Following the child’s lead	The therapist followed the child’s lead by (a) Allowing the child to choose an activity, object, or topic and following the child’s choice, for example “What do you want to do?” or “Do you want to play with play dough or bubbles?” (b) Following the child’s spontaneous choice, for example, by playing with trains if the child indicated he or she wanted to play with trains without being asked; (c) Being involved with the child for at least 10 s of the interval by being nearby the child and looking at him or her, making comments, or asking questions to him or her (e.g. Bryson et al., 2007; Huskens, Reijers, and Didden, 2012).
2. Implementing three-term contingency learning trials	The therapist implemented a three-term contingency learning trial if at least one the following sequences of subcomponents occurred during the 1-min interval: (a) Presenting a clear opportunity, child initiation or attempt, and reinforcing the child’s initiation or attempt naturally and contingently; (b) Presenting a clear opportunity, prompting the child to initiate, child initiation or attempt, and reinforcing the child’s initiation or attempt naturally and contingently; (c) Presenting a clear opportunity, prompting the child to initiate at least once, no child initiation or attempt, and providing no reinforcement (Huskens et al., 2012; Verschuur, Huskens, & Didden, 2019; Verschuur, Huskens, Verhoeven, & Didden, 2017).
	1. Presenting a clear opportunity	The therapist presented a clear opportunity by (a) Shared control: the therapist had control over an object the child desired or needed during the activity. (b) In sight, out of reach: the object the child desired or needed during the activity was visible, but out of the child’s reach; the object was not in the therapist’s possession. (c) Out of sight, out of reach: the object the child desired or needed during the activity was invisible and out of the child’s reach; the object was not in the therapist’s possession. (d) Waiting: the therapist did nothing and waited for 3 s, when the routine of the activity expected the therapist to act or when the child needed help to carry out an action. (e) Interrupting a routine: the therapist did something that did not fit in the routine of the activity, for example, throwing the dice while it was the child’s turn. (f) Carrier phrase: the therapist sang a familiar song or said a familiar rhyme/saying, but omitted the last word, for example, “Ready, set . . .” and then remained silent. (g) Making a statement: the therapist made a statement or comment without giving details to the child, for example, “I have done something fun yesterday,” and then remained silent.
	2. Child initiation	The child began or directed a social interaction, spontaneously or prompted, for example, by saying “Ball?” or “Can you help me?” or “Which movie did you see?”
	3. Prompting the child to initiate	If the child did not initiate within 5 s after the therapist presented a clear opportunity, the therapist offered help by prompting. Three types of prompts were recorded: (a) Time delay prompt: the therapist was silent for 3 s, while giving the child a questioning look and/or making a sign to stimulate the child to initiate. (b) Open-ended question prompt: the therapist asked an open question to stimulate the child to initiate, for example “What could you ask me now?” (c) Verbal model prompt: the therapist modeled the verbal utterance that the child could use to initiate. The therapist continued prompting until the child initiated, until the therapist gave three prompts (children who were not verbally fluent) or until a verbal model prompt (children who were verbally fluent) was given.
	4. Reinforcing the child’s initiation or attempt naturally and contingently	The therapist reinforced the child’s initiations or reasonable attempts naturally and contingently by responding to this initiation within 2 s. Natural and contingent reinforcement was recorded if (a) the response was the therapist’s first behavior after the child’s initiation or attempt and (b) the response was a natural consequence of the child’s question (i.e. in everyday life the response to this question is equal). Natural and contingent reinforcement was also recorded if the therapist withheld reinforcement if the child did not initiate after being prompted at least once.
	An example of a correct three-term contingency learning trial would be holding a marble while playing with a marble run and immediately giving a marble to the child when he or she asked for it. A three-term contingency learning trial would also be correct if the therapist controlled the ball, prompted the child to ask for the ball by saying “ball,” and did not give the ball to the child if he or she did no reasonable attempt to say “ball.”
3. Interspersing maintenance and acquisition tasks	The therapist interspersed maintenance and acquisition tasks if he or she presented at least two different opportunities (e.g. shared control and interrupting a routine) or used at least two different prompts (e.g. time delay prompt and verbal model prompt) in two consecutive intervals (e.g. Bryson et al., 2007).

PRT: pivotal response treatment.

Observers were instructed to view the entire interval and to record subsequently whether or not each component (i.e. following the child’s lead, implementing three-term contingency learning trials, and interspersing maintenance and acquisition tasks) had been implemented during the interval. A plus (+) or minus (–) was recorded if the component had or had not been implemented during the interval. To record whether or not the therapist implemented three-term contingency learning trials, observers first recorded each sequence of subcomponents using numbers (e.g. shared control = 1, time delay prompt = 2, child initiation = 3, and natural and contingent reinforcement = 4) and subsequently recorded a plus (+) or minus (–) if the therapist did or did not correctly implement a three-term contingency learning trial during the interval. The data sheet that was used to score the fidelity of PRT implementation is provided as Supplemental Material. For each videotape, the total percentage of fidelity was calculated by dividing the number of intervals in which all three components were scored a plus by the total number of intervals, multiplied by 100. For each therapist, the average percentage of fidelity was calculated across three videotapes.

A second naïve observer independently coded 33% of the videotapes, evenly distributed across dyads, to determine the interobserver agreement. Prior to coding, both observers received intensive training in using the recording system for fidelity of PRT implementation. Observer training included (a) discussing definitions, examples, and non-examples using written guidelines and video examples; (b) practicing coding using 1-min fragments; (c) discussing any discrepancies and revision of written guidelines as necessary; and (d) independent coding of 10-min training videotapes (Ledford, Lane, & Gast, 2018). Observer training continued until interobserver agreement was acceptable (i.e. prevalence-adjusted and bias-adjusted kappa [PABAK] ⩾ 0.60) in two consecutive training videotapes (Byrt, Bishop, & Carlin, 1993; Cicchetti et al., 2006; Kennedy, 2005). Ongoing training sessions were conducted every 2 weeks to minimize observer drift and to promote interobserver agreement. Interobserver agreement was assessed for total fidelity of PRT implementation and per component by calculating PABAK (Byrt et al., 1993). For total fidelity of PRT implementation, mean PABAK was 0.62 (SD = 0.31, range = 0.00–1.00), indicating good interobserver agreement (Cicchetti et al., 2006). However, interobserver agreement was less than 0.60 for 34% of the videotapes, indicating that interobserver agreement was insufficient in many instances. For “following the child’s lead,” “implementing three-term contingency learning trials,” and “interspersing maintenance and acquisition tasks,” the mean PABAKs were 0.99 (SD = 0.06, range = 0.60–1.00), 0.66 (SD = 0.25, range = 0.00–1.00), and 0.55 (SD = 0.36, range = –0.11 to 1.00), respectively. This indicates fair to excellent interobserver agreement, but interobserver agreement was insufficient in 29% of the videotapes for “implementing three-term contingency learning trials” and 59% for “interspersing maintenance and acquisition tasks.”

Data analyses

First, preliminary analyses were conducted to explore correlations between therapist characteristics (i.e. personality, therapist–child relationship, attitude toward EBPs, and experience with ASD and PRT), child characteristics (i.e. child age and ASD symptom severity), and the outcome variable (i.e. total fidelity of PRT implementation). Normality was checked using quantile–quantile (Q-Q) plots and by examining each variable’s distribution for skewness and kurtosis. Several variables were not normally distributed, including the outcome variable. Also, several variables had outliers. In non-normal distributions and when outliers are present, Pearson’s correlation coefficient is less robust than Spearman’s correlation coefficient (e.g. De Winter, Gosling, & Potter, 2016). We calculated both correlation coefficients and, since they did not differ much in magnitude or direction, we only reported Pearson’s correlations.

Subsequently, a multiple linear regression model was constructed with total fidelity of PRT implementation as the outcome variable and variables that significantly correlated with PRT fidelity as predictors. In addition, because of the relatively small sample size and the non-normal distribution of several variables, we used bootstrapping to estimate confidence intervals around regression coefficients, drawing 1000 samples from the data, and we applied a leave-one-out cross validation (LOOCV) to evaluate the robustness of the regression model (Efron & Tibshirani, 1993; Field, Miles, & Field, 2012; James, Witten, Hastie, & Tibshirani, 2013). With LOOCV, data from all participants except one were used to construct the regression model, after which the omitted participant was used to validate the model. This was repeated until each participant in the sample was used once for validation. The use of bootstrap and cross-validation methods is recommended if samples are small or if the assumption of normally distributed errors is violated (e.g. Field, 2017; Field et al., 2012) and is also applied in autism research (e.g. Davidson, Kaushanskaya, & Weismer, 2018; Hellinckx, Roeyers, & Van Waelvelde, 2013; Supekar et al., 2013).

Finally, because of differences in means of the three components of fidelity, we also performed exploratory analyses to determine the relationship between therapist and child characteristics and each component by repeating the above-described analyses for each component of PRT fidelity separately.

The significance level was set at α = 0.05. Following Cohen’s guidelines, R² was interpreted as small (0.01), medium (0.09), or large (0.25; Cohen, 1992). Analyses were conducted using R (R Core Team, 2018). We used the “boot” package for bootstrapping (Canty & Ripley, 2017; Field et al., 2012) and the “caret” package for cross validation (Kuhn, 2008).

Results

Fidelity of PRT implementation

The average total fidelity of PRT implementation across therapists was 30.2% (SD = 17.5%, range = 6.67%–90.0%). Five therapists achieved 80% fidelity of implementation during at least one 10-min PRT session and for only one therapist the average total fidelity across sessions exceeded 80%, indicating that the overall fidelity of PRT implementation was low. For the components “following the child’s lead,” “implementing three-term contingency learning trials,” and “interspersing maintenance and acquisition tasks,” the average percentages of fidelity were 99.5, 52.5, and 35.1, respectively (see Table 2). On average, therapists implemented 7.85 three-term contingency learning trials per 10-min PRT session (SD = 4.83; range = 2–23).

Table 2.

Descriptive statistics for fidelity of PRT implementation (%).

Component	Mean	Median	SD	Range
Total fidelity of PRT implementation	30.24	26.67	17.52	6.67–90.00
Following the child’s lead	99.51	100.00	1.41	93.33–100.00
Implementing three-term contingency learning trials	52.48	50.00	19.00	16.67–95.00
Interspersing maintenance and acquisition tasks	35.09	29.63	19.58	0.00–94.45

PRT: pivotal response treatment; SD: standard deviation.

Predictors of fidelity of PRT implementation

Therapist characteristics and Pearson’s correlation coefficients are presented in Tables 3 and 4. Total fidelity of PRT implementation was not significantly related to openness to experience, conscientiousness, or therapist–child relationship. Also, there was no significant relationship between the fidelity of PRT implementation and therapists’ general attitude toward EBPs or attitude related to the appeal of an EBP. However, therapists’ openness to innovation was significantly and positively correlated with fidelity, indicating that therapists who were more willing to use or try new interventions and EBPs had higher levels of fidelity. Therapists’ experience in working with children with ASD was not significantly associated with the fidelity of PRT implementation, but we found a significant positive relationship between fidelity and therapists’ experience with PRT, indicating that therapists with more experience with PRT implemented PRT with higher fidelity. There was no significant association between the fidelity of PRT implementation and child age or autism symptom severity.

Table 3.

Descriptive statistics for therapist characteristics.

Variable	Mean	SD	Range
Personality
Openness to experiences	37.22	5.61	26–51
Conscientiousness	47.66	4.91	39–57
Therapist–child relationship
Total	102.78	12.41	75–130
Attitude toward EBPs
Total	43.12	4.42	36–55
Appeal	13.05	1.72	8–16
Openness	11.46	1.89	8–16

SD: standard deviation; EBPs: evidence-based practices.

Table 4.

Pearson’s correlations between fidelity of PRT implementation and therapist/child characteristics.

Variable	1	2	3	4	5	6	7	8	9	10	11
1. Total fidelity of PRT implementation	–
2. Openness to experiences	0.08	–
3. Conscientiousness	−0.04	−0.19	–
4. Therapist–child relationship (total)	0.01	0.05	0.11	–
5. Attitude toward EBPs (total)	0.20	−0.08	0.33*	0.08	–
6. Attitude toward EBPs (appeal)	0.13	−0.17	0.13	−0.01	0.58**	–
7. Attitude toward EBPs (openness)	0.49**	−0.03	0.17	−0.04	0.52**	0.38*	–
8. Experience with ASD	0.15	−0.13	0.10	−0.05	−0.11	0.25	0.15	–
9. Experience with PRT	0.41**	−0.13	0.01	0.12	0.01	0.06	0.13	0.41**	–
10. Child age	−0.20	−0.15	0.04	−0.28	0.09	0.02	0.14	−0.44**	−0.32*	–
11. ASD symptom severity	−0.24	−0.31*	0.04	−0.06	−0.16	−0.05	0.07	−0.12	−0.26	0.23	–

PRT: pivotal response treatment; EBPs: evidence-based practices; ASD: autism spectrum disorder.

^*p < 0.05; **p < 0.01.

A multiple linear regression analysis was conducted to determine whether openness to innovation and experience with PRT would predict the total fidelity of PRT implementation. Table 5 displays the results of the regression analysis. Both openness to innovation and experience with PRT significantly predicted the fidelity of PRT implementation and accounted for 36% of the variance in fidelity (F(2, 38) = 10.68, p < 0.001), indicating a large effect. Bootstrapped confidence intervals for both openness to innovation and experience with PRT did not include zero and thus confirmed that both predictors significantly predicted the fidelity of PRT implementation. After cross validation, openness to innovation and experience with PRT explained 24% of the variance in fidelity. The reduction in R² suggests that the regression model was not fully robust, but the cross-validated R² still indicated a medium to large effect.

Table 5.

Multiple regression analysis to predict total fidelity of PRT implementation from openness to innovation and experience with PRT.

Dependent variable	Independent variable	B	SE of B	β	p
Total fidelity of PRT implementation	Constant	−24.14 (−48.78 to −3.23)	14.03		0.09
	Attitude toward EBPs (openness)	4.11 (2.30 to 6.02)	1.22	0.44	0.002
	Experience with PRT	0.40 (0.11 to 1.12)	0.15	0.35	0.01

PRT: pivotal response treatment; SE: standard error; EBPs: evidence-based practices.

95% bias-corrected and accelerated confidence intervals are reported in parentheses. Confidence intervals are based on 1000 bootstrap samples.

Exploratory analyses of predictors of fidelity components

“Following the child’s lead” was not significantly correlated with any of the therapist or child characteristics (see Supplemental Material). Therefore, no regression model was constructed for this component.

“Implementing three-term contingency learning trials” was significantly related to therapists’ openness to innovation (r = 0.40, p = 0.01), therapists’ experience with PRT (r = 0.39, p = 0.01) and child age (r = −0.36, p = 0.02; see Supplemental Material). Multiple linear regression indicated that openness to innovation and child age predicted this component of PRT fidelity (see Table 6). Bootstrapped confidence intervals indicated that only openness to innovation was a significant predictor. Together, openness to innovation, experience with PRT, and child age explained 38% of the variance in “implementing three-term contingency learning trials” (F(3, 37) = 7.55, p < 0.001), indicating a large effect. The cross-validated R² was 24%, which indicates a medium to large effect, but also suggests that the regression model was not fully robust.

Table 6.

Exploratory multiple regression analysis to predict fidelity of PRT components.

Dependent variable	Independent variable	B	SE of B	β	p
Implementing three-term contingency learning trials	Constant	15.34 (−14.24 to 39.75)	15.93		0.34
	Attitude toward EBPs (openness)	4.25 (1.90 to 6.44)	1.34	0.42	0.003
	Experience with PRT	0.28 (−0.08 to 0.92)	0.17	0.23	0.11
	Child age	−1.92 (−3.51 to 0.12)	0.78	−0.34	0.02
Interspersing maintenance and acquisition tasks	Constant	−17.69 (−42.57 to 16.97)	15.74		0.27
	Attitude toward EBPs (openness)	3.74 (0.60 to 5.82)	1.36	0.36	0.009
	Experience with PRT	0.54 (0.16 to 1.02)	0.17	0.43	0.002

PRT: pivotal response treatment; SE: standard error; EBPs: evidence-based practices.

No regression model was constructed for “following the child’s lead,” since none of the therapist and child characteristics was significantly correlated with this component. 95% bias-corrected and accelerated confidence intervals are reported in parentheses. Confidence intervals are based on 1000 bootstrap samples.

The component “interspersing maintenance and acquisition tasks” was significantly associated with therapists’ openness to innovation (r = 0.42, p = 0.01) and therapists’ experience with PRT (r = 0.48, p < 0.01; see Supplemental Material). Multiple linear regression indicated that both therapist characteristics predicted this component of PRT fidelity (see Table 6). Bootstrapped confidence intervals confirmed these results. Openness to innovation and experience with PRT accounted for 36% of the variance in “interspersing acquisition and maintenance tasks” (F(2, 38) = 10.49, p < 0.001), indicating a large effect. The cross-validated R² was again lower (i.e. 25%), but still indicated a large effect.

Discussion

This study examined the relationship between fidelity of PRT implementation and therapist characteristics in 41 certified PRT therapists working with children with ASD. We hypothesized that higher levels of openness to experience and conscientiousness, a more positive therapist–child relationship, a more positive attitude toward EBPs, and higher levels of experience would predict higher fidelity of PRT implementation. In addition, we explored whether child age and autism symptom severity were related to fidelity of PRT implementation. Our results indicated that openness to experience, conscientiousness, the relationship between therapist and child, child age, and severity of autism symptoms were not significantly related to total fidelity of PRT implementation. Therapists’ attitude toward EBPs, specifically openness to innovation, and therapists’ experience with PRT both significantly and positively predicted total fidelity of PRT implementation, indicating that therapists who were more willing to use or try new interventions and had more experience with PRT showed higher fidelity. These therapist characteristics together accounted for 36% of the variance in fidelity of PRT implementation. Cross validation indicated that these therapist characteristics may explain a lower but nevertheless medium to large amount of variance in fidelity in another sample. Exploratory analyses to determine the relationship between therapist and child characteristics and components of PRT fidelity confirmed that therapists’ openness to innovation positively predicted “implementing three-term contingency learning trials” and “interspersing maintenance and acquisition tasks”; therapists’ experience with PRT only predicted the latter. Child age was negatively associated with “implementing three-term contingency learning trials,” suggesting that therapists implemented more learning trials for younger children, but the child’s age was not a significant predictor of this component of PRT fidelity. None of the therapist or child characteristics was significantly related to “following the child’s lead.”

Although numerous studies have demonstrated associations between personality traits and procedural fidelity or job performance (e.g. Barrick & Mount, 1991; Barrick et al., 2001; Judge & Zapata, 2015; Peters-Scheffer et al., 2013), we found no relationship between therapists’ openness to experience and conscientiousness and fidelity of PRT implementation. It is important to note that previous studies generally included large samples (e.g. Barrick et al., 2001). Compared to these studies, our sample was relatively small and thus might have had not enough power to detect personality traits’ effects (Green, 1991). Research within a larger sample of therapists is warranted to examine whether fidelity of PRT implementation is associated with personality traits and, if that is the case, how therapist training in PRT could be designed to incorporate or foster these traits to increase each therapist’s fidelity of PRT implementation to an acceptable level.

We also found no association between the fidelity of PRT implementation and the therapist–child relationship, which is in contrast to previous research (e.g. Peters-Scheffer et al., 2013; Tschuschke et al., 2015; Weck et al., 2015). Again, the small sample size may explain this divergence of findings. Yet, another explanation could be that scores on the STRS merely reflect how therapists perceive their relationship with the child in general, which may differ from the quality of the therapist–child interaction during videotaped PRT sessions (Beurkens, Hobson, & Hobson, 2013). It is possible that the quality of the interaction between therapist and child more directly affects the fidelity of PRT implementation than the therapist–child relationship. Future research should address this topic.

Consistent with other studies (e.g. Beidas et al., 2015; Pagoto et al., 2007; Reding et al., 2014), we found that therapists’ attitude toward EBPs predicted the fidelity of implementation, but only openness to innovation predicted fidelity and not attitude related to the appeal of an EBP. It has been suggested that openness to innovation is akin to the personality trait of openness to experience (e.g. Aarons, 2005; Barrick & Mount, 1991), but in our study openness to innovation and openness to experience were uncorrelated and only openness to innovation predicted fidelity of PRT implementation. This finding might be explained by the difference between attitudes and traits (Ajzen, 2005). Although both attitudes and traits refer to latent constructs that can be inferred from observable responses, attitudes are evaluative responses directed at an external target that can change rapidly as new information or experiences become available, whereas traits are tendencies to respond in a particular manner that focus on the individual himself or herself and are much more resistant to change (Ajzen, 2005). Previous studies have demonstrated that negative attitudes toward EBPs can be improved by training (e.g. Allen & Bowles, 2014; Bearman, Wadkins, Bailin, & Doctoroff, 2015). Providing education regarding attitudes toward EBPs may be as important as training in a specific EBP. Thus, to improve the fidelity of PRT implementation, it is important to target therapists’ attitudes toward EBPs and specifically their openness to innovation prior to or during training in PRT.

Overall fidelity of PRT implementation was low (i.e. 30.2%), compared to other studies (e.g. Mohammadzaheri, Koegel, Rezaee, & Rafiee, 2014; Robinson, 2011) and given that all therapists were certified PRT trainers who had demonstrated fidelity at or greater than 80% prior to participation in this study. This finding suggests that therapists may not maintain their fidelity levels after training. If certified therapists do not correctly implement PRT in practice, fidelity may be even lower for therapists without formal training in PRT, which is often the case in community-based services (e.g. Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012). As a result, PRT may be less effective in community settings (e.g. Durlak & DuPre, 2008). We also found that therapists with more experience with PRT displayed higher levels of fidelity, which indicates that therapists’ fidelity of PRT implementation may increase once their experience increases. These findings stress the importance of the sustainment phase in the implementation process. Factors that may facilitate the sustained use of EBPs include ongoing funding, ongoing monitoring of fidelity, supervision, and feedback (Aarons et al., 2011; Bond et al., 2014; Schoenwald et al., 2011). Funding is necessary to continue monitoring of fidelity and to provide ongoing supervision and feedback, which are critical for continued EBP use and increasing fidelity of implementation (Novins, Green, Legha, & Aarons, 2013; Schoenwald et al., 2011). The low overall level of fidelity could be due to the way we determined fidelity of PRT implementation. To assess fidelity, we scored 10 min in the middle of each videotape, whereas therapists were allowed to self-select 10-min fragments during the process of obtaining PRT certification. PRT is designed for implementation in natural situations, suggesting that therapists should be able to implement PRT with high fidelity throughout the day and across activities, and thus during an entire videotape. However, it is possible that only therapists with more experience with PRT continuously display this level of fidelity, because they easily or naturally identify opportunities to implement PRT throughout the day, whereas less experienced therapists may achieve adequate levels of fidelity only during prepared activities or in self-selected minutes of a videotape. For future research, it is important to examine how levels of fidelity of PRT implementation vary throughout the day and to investigate strategies to achieve a continuous high level of fidelity.

Child characteristics (i.e. child age and autism symptom severity) were not related to the total fidelity of PRT implementation, but child age was negatively associated with “implementing three-term contingency learning trials,” which is consistent with Suhrheinrich et al. (2016). Since bootstrapping did not confirm that child age was a significant predictor of this component of fidelity, we should be careful to draw firm conclusions. However, this finding suggests that it may be more difficult to implement three-term contingency learning trials for older children, because these children play or work more independently compared to younger children, and implies that specifically targeting therapist training based on children’s ages may be important to increase fidelity for this component of PRT (Suhrheinrich et al., 2016).

Therapist characteristics, specifically openness to innovation and experience with PRT, accounted for 36% of variance in total fidelity of PRT implementation. This suggests that there are other factors that might affect fidelity as well. These possible factors are described in various implementation models and include other therapist characteristics (e.g. perceived workload), additional child characteristics (e.g. challenging behavior and rate of progress or response to intervention), and organizational characteristics (e.g. provision of ongoing feedback and supervision, organizational culture or climate, and implementation leadership; Aarons et al., 2011; Ransford, Greenberg, Domitrovich, Small, & Jacobson, 2009; Symes et al., 2006). Further research is warranted to examine the relationship of these factors with fidelity.

Several limitations should be considered regarding this study. First, although interobserver agreement for total fidelity of PRT implementation was good on average, in 34% of the videotapes interobserver agreement was insufficient, which has affected the accuracy of our data. Therefore, our findings should be interpreted with caution. Interobserver agreement may have been low in several instances, because total fidelity of PRT implementation and two of its components (i.e. “implementing three-term contingency learning trials” and “interspersing maintenance and acquisition tasks”) involved recording of multiple behaviors (Cooper et al., 2013). In addition, due to bad acoustics and background noises, the audio quality of several videotapes was poor, which may also have affected the interobserver agreement. Second, due to the cross-sectional design, we could not determine the direction of the relationship between therapist characteristics and the fidelity of PRT implementation. Longitudinal research is required to test for causal relationships between these variables. Third, as already mentioned, the sample size was relatively small, limiting the possibility to detect small effects. In addition, as this small sample size was expected, we did not collect data on additional relevant therapist, child, and organizational characteristics and thus could not examine these factors in this study. Fourth, we used a self-report measure to assess the therapist–child relationship. Since self-report measures are often biased (Baumeister, Vohs, & Funder, 2007), this may have affected our findings. Fifth, we defined experience with PRT as the number of months after obtaining PRT level III certification. This definition does not consider how often therapists provided PRT nor the amount of ongoing supervision or support they received. However, these factors are likely to facilitate experience (Stein & Lambert, 1984). Additional research is necessary to assess the influence of these factors. Sixth, the generalizability of our findings may be limited, as our coding system for fidelity of PRT implementation differed from the fidelity measure that is generally used and our participants were certified therapists who were highly trained compared to most community service providers. Finally, the results of the cross validation indicated that our regression model was not fully robust. Therefore, the results of this study need to be confirmed in a larger sample of certified PRT therapists in future research.

Despite these limitations, this study provides insight into the relationship between therapist characteristics and fidelity of PRT implementation. Both openness to innovation and experience with PRT predicted the fidelity of PRT implementation, stressing the importance of targeting therapists’ attitudes toward innovation prior to or during training in PRT and providing them with ongoing supervision and feedback after training. Future research should identify other factors that predict fidelity of PRT implementation to achieve high levels of fidelity and thus to improve intervention outcomes for children with ASD.