Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Dec 23.
Published in final edited form as: Prof Psychol Res Pr. 2017 Nov 2;48(6):481–489. doi: 10.1037/pro0000149

Training Community-Based Clinicians in Parent-Child Interaction Therapy: The Interaction Between Expert Consultation and Caseload

Carrie B Jackson 1, Amy D Herschell 2, Kristen F Schaffner 3, Nicholas A Turiano 4, Cheryl B McNeil 5
PMCID: PMC8697733  NIHMSID: NIHMS962765  PMID: 34955593

Abstract

Professional psychologists are increasingly encouraged to utilize evidence-based treatments (EBTs), and therefore have a need to participate and provide the most efficient training methods for these treatments. Multicomponent trainings, which commonly include ongoing support, are more effective than brief methods such as 1-day workshops or reading treatment manuals. The present study examined the effectiveness of 1 form of ongoing support, consultation, as part of a multicomponent training protocol. Thirty-two community-based clinicians were trained in Parent-Child Interaction Therapy (PCIT) as part of a statewide implementation effort, and data were collected on clinician and implementation outcomes at pre-, mid-, and posttraining. Simple and multiple linear regression analyses were conducted to predict posttraining knowledge, skill, acceptability, and feasibility, as well as to examine clinician variables that might moderate these relations. Greater consultation call attendance significantly predicted higher posttraining skill; however, this association was qualified by a significant interaction with PCIT caseload. Implications for training guidelines are discussed.

Keywords: consultation, implementation, training, parent training

There continues to be a divide between clinical practice and research in psychology, with significant barriers in transporting evidence-based treatments (EBTs) to community settings, where many individuals receive mental health services (LeJeune & Luoma, 2015). Widening the gap between research and practice, many community-based clinicians have not received any formal training in EBTs (Rodríguez, Southam-Gerow, O’Connor, & Allin, 2014; Weissman et al., 2006). At the same time, many national and international initiatives have called for the increased use of these treatments, and many state mental health systems have mandated clinician use of EBTs (Glisson & Schoenwald, 2005; Rieckmann, Bergmann, & Rasplica, 2011). This has resulted in a challenge for a large number of clinicians to seek out effective training in EBTs within a short amount of time.

Although there is a pressing need to train clinicians in EBTs, there is little agreement on the most effective way to do so (Jensen-Doss, Hawley, Lopez, & Osterberg, 2009). Baldwin and Ford’s training transfer model (see Baldwin & Ford, 1988) provides a framework for understanding different factors that may influence training outcomes. Three training input factors are hypothesized to influence training outcomes: (1) training design, such as training content and components, (2) individual trainee characteristics, including demographics and previous skills, and (3) work-environment factors, such as organizational climate and supervisory support. These input factors may directly influence immediate learning of training material, and also indirectly affect the transfer of these learned skills to the workplace (Baldwin & Ford, 1988). Of particular relevance to the current article are the training design and individual trainee characteristics input factors.

With regard to training design, research supports that individual training components such as workshops and treatment manuals are mostly ineffective for increasing clinician knowledge and skill (Herschell, Kolko, Baumann, & Davis, 2010). These training methods may be too brief to allow clinicians to fully integrate newly learned skills into their current caseload (Beidas, Edmunds, Marcus, & Kendall, 2012). A growing body of literature recommends that multicomponent trainings, which commonly include an element of ongoing support (e.g., consultation), are the most effective strategy for training clinicians (Herschell et al., 2010).

Consultation has been defined as communication between an expert in a field, the consultant, and a consultee who seeks advice from the consultant for improvement of a current work issue (Caplan, 1970). Within the field of mental health, consultation call attendance has been linked to improvements in clinician knowledge, skill, and adherence (Beidas et al., 2012; Edmunds, Beidas, & Kendall, 2013; Schoenwald, Chapman, Sheidow, & Carter, 2009; Schoenwald, Sheidow, & Letourneau, 2004; Stirman et al., 2017). Although these studies are an important step in understanding the impact of consultation on clinician outcomes, these studies have primarily examined the effects of training protocols as a whole, as opposed to dismantling training design factors that promote positive outcomes.

Researchers have also begun to examine the influence of consultation on implementation outcomes. Proctor et al. (2011) proposed a taxonomy of implementation outcomes to assist in analyzing the effectiveness of implementation efforts, namely, that implementation outcomes are interrelated with and precede client outcomes; therefore, improving implementation may subsequently benefit clients (Proctor et al., 2011). One study has linked consultation calls to one of Proctor et al.’s (2011) implementation outcomes, specifically clinician fidelity; however, other implementation outcomes have not been examined (Schoenwald, Carter, Chapman, & Sheidow, 2008). The current study focuses on the implementation outcomes of acceptability and feasibility. Acceptability has been defined as a perception that a given implementation effort is agreeable or satisfactory (Proctor et al., 2011). For example, clinicians, administrators, or stakeholders may hold that a given treatment is satisfactory on the basis of several dimensions (e.g., content, complexity; Hides, Lubman, Elkins, Catania, & Rogers, 2007; Proctor et al., 2011). Feasibility is the degree to which this effort or treatment may successfully be utilized (Proctor et al., 2011). Clinicians may perceive a treatment to be acceptable, but that it is not feasible given a lack of training, space, or resources to appropriately carry out the intervention (Karlsson & Bendtsen, 2005; Myers, Valentine, & Melzer, 2007; Proctor et al., 2011). Further research is necessary to understand the impact of consultation on additional implementation outcomes, such as acceptability and feasibility.

Baldwin and Ford’s (1988) training transfer model also suggests that individual trainee characteristics may influence training outcomes. Although research in other fields (e.g., industrial–organizational psychology) has found evidence to support the influence of trainee characteristics on training outcomes (e.g., Velada, Caetano, Michel, Lyons, & Kavanagh, 2007), results within the field of mental health remain mixed. Several studies have found that individual clinician characteristics such as attitudes, gender, and theoretical orientation may influence the self-reported use of EBTs (Beidas, Edmunds, et al., 2014; Ditty, Landes, Doyle, & Beidas, 2015). However, other studies have found that factors related to a clinician’s therapy experience (e.g., caseload, licensure, years of experience) do not affect outcomes (Bearman et al., 2013; Campbell, Raja, & Grining, 1999; Lewis & Simons, 2011). Therefore, additional research is necessary to understand the role of clinician characteristics on clinician (e.g., knowledge, skill) and implementation outcomes (e.g., acceptability, feasibility).

Given the lack of research on consultation and the inconsistent findings related to individual trainee characteristics, the present study examined the consultation component of a multicomponent training protocol for PCIT. In the current study, we analyzed if consultation calls predicted posttraining clinician knowledge and skill, as well as the implementation outcomes of acceptability and feasibility. Additionally, we sought to investigate if clinician characteristics (caseload, licensure, baseline knowledge, years of experience, and experience with preschool children) altered the strength of these relationships.

Method

Parent Study

The present study utilized data from a parent study funded by the National Institute of Mental Health (R01 MH095750). The parent study is a statewide trial examining three implementation methods (training models) for PCIT including (1) a cascading training model (CM), a hierarchical training method, (2) distance education (DE), an online-computer based course, and (3) a learning collaborative (LC), which addresses the clinical, supervisor, administrator, and senior leader levels of an agency and emphasizes collaboration across agencies. The CM involves training senior clinicians within an organization, who then have the ability to train other clinicians within their organization. DE involves an individual clinician completing and interacting with a Web-based training course. The LC model, originally developed by the Institute for Health Improvement, and adapted to behavioral health by the National Child Traumatic Stress Network, involves training multiple levels within an organization (e.g., clinicians, supervisors, senior leaders) with a specific focus on implementation within an organization. Further details of the training conditions may be found in the study protocol of the parent study (Herschell et al., 2015). Fifty eligible agencies were randomized to one of the three aforementioned training conditions. The parent study examines clinician-level (e.g., knowledge), clinic-level (e.g., adoption), and client-level (e.g., disruptive behavior problems) outcomes.

Participants

Participants from the CM condition of the parent study were included in the present study. The CM condition was chosen to examine consultation given that this was the only condition that included video review of therapy sessions to measure skill. For inclusion, clinicians had to (a) be currently employed at a clinic meeting inclusion criteria, (b) hold a master’s- or doctoral-level degree, (c) be licensed in his or her field or receiving supervision from a licensed individual, (d) be able to provide treatment to families appropriate for PCIT, (e) have no previous training in PCIT but were receptive to training, and (f) be willing to participate in various study tasks (e.g., completing assessments, videotaping sessions). The current study only involved senior clinicians who were directly trained by project trainers as opposed to also including clinicians who had been trained by senior clinicians within their organization.

Participants were 32 clinicians from 16 agencies, with 2 clinicians per agency, across the Commonwealth of Pennsylvania. Table 1 contains demographic information for the participants. Twenty-nine (91%) of the participants were female, 31 (97%) of the participants were White, 32 (100%) held a master’s degree, and 11 (34%) were professionally licensed.

Table 1.

Demographic Information (n = 32)

Variable N %
Gender
  Male 3 9
  Female 29 91
Race
  White 31 97
  Unknown/Not reported 1 3
Education
  Masters degree 32 100
Professional licensed
  Yes 11 34
  No 21 66
Open in a new tab

Procedure

All procedures were approved by the (University of Pittsburgh’s) Institutional Review Board. Clinicians completed questionnaires at four separate time-points: baseline, 6 (mid), 12 (post), and 24 months (1 year follow-up). Relevant to the proposed study, at baseline, clinicians completed the Clinician Background form. The Usage Rating Profile-Intervention scale was administered at 6, and 12 months. The PCIT Coaches Quiz was completed across all time-points. The Clinician PCIT Skills Checklist was completed throughout training, consultation calls, and video-review of tapes that clinicians submitted.

Incentives were included for participation of both clinicians and agencies. Clinicians received free training, Continued Education credits, and payment for completion of questionnaires ($25 for baseline and 6-month follow-up, $30 for 12-month follow-up, and $40 for 24-month follow-up). Agencies received free PCIT training for their clinicians and a stipend ($1,000) for initial startup costs and necessary equipment for utilizing PCIT in their organization.

Trainers

Three trainers participated in the cascading condition and were balanced across training conditions and across consultation call groups. For the purposes of this study, the trainers and consultants were the same experts, and are hereby referred to as trainers. The trainers were Caucasian females in their late twenties and had extensive training and expertise in PCIT, disruptive behavior disorders, and child welfare. Two of the trainers received their doctorate in clinical child psychology, and one of the trainers received her doctorate in school psychology.

Training

The CM utilized in this study is consistent with PCIT International Training guidelines, and was carefully developed and operationalized through an expert consensus process with PCIT Master Trainers (Scudder & Herschell, 2015). Training was conducted in four waves across the Commonwealth of Pennsylvania as training across the state could not be completed at once. Training consisted of an initial 40 hours of face-to-face contact with two of three expert PCIT trainers, an advanced live training of 16 hours utilizing real cases 6 months after the initial training, and ongoing biweekly contact with a PCIT trainer over 12 months. A training integrity checklist was completed at each in-person training to assess that training content was administered according to protocol. Two clinicians from each organization were included in each training group, with each training group including an average of eight clinicians total (range = 6–8, SD = 2.00).

Consultation calls

Included in the 12 months of intensive training, clinicians had the opportunity to attend up to 24 1-hr consultation calls conducted by one of three expert consultants, who were also trainers for the in-person training. In total, 96 consultation calls were conducted across four consultation call groups. Trainers led consultation calls according to the PCIT Case Consultation Outline and Record, which aligns with Nadeem and colleagues’ (2013) core components of consultation.

Treatment

Parent-Child Interaction Therapy (PCIT; Eyberg & Funderburk, 2011; McNeil & Hembree-Kigin, 2010) is a “well-established” manualized EBT for youth ages 2.5 to 7 years old with externalizing behavior problems (McNeil & Hembree-Kigin, 2010). PCIT focuses on improving the quality of parent–child interactions and increasing child compliance, while also reducing disruptive behaviors such as aggression and defiance. PCIT is divided into two phases of treatment, Child-Directed Interaction (CDI) and Parent-Directed Interaction (PDI). During the CDI phase, parents learn nondirective play skills that work to engage the child in play and improve the quality of the parent– child relationship. During the PDI phase, parents include skills such as commands and consequences (e.g., time-out) with the goal of reducing disruptive behaviors and increasing compliance (Eyberg & Funderburk, 2011; McNeil & Hembree-Kigin, 2010).

Measures

Clinician background form

Basic demographic information for clinicians was collected utilizing the Clinician Background Form, adapted from an existing community practitioner demographic form (Kolko et al., 2012). This form consists of demographic items such as licensure status as well as items related to previous clinical experiences such as years of experience and therapy experience with preschool children.

PCIT case consultation outline and record

Trainers completed this measure following each consultation call to track clinician attendance and consultation content. This outline contains sections for trainers to indicate (a) clinician attendance and participation, (b) clinician caseload, (c) trainer perspectives of the most impactful content areas on the consultation call, (d) clinician-reported barriers to implementation, and (e) amount of time engaged in different consultation content areas. Consultation content areas in the outline align with Nadeem and colleagues’ (2013) examination of core components of the consultation process. Trainers estimate the amount of time spent engaging in various consultation activities including continued training on key topics, problem-solving implementation barriers (e.g., lack of referrals), reviewing cases, holding clinicians accountable (e.g., submitting parent– child assessments), building skills (e.g., role plays, feedback on skill demonstration), planning for sustainability, structuring the call (e.g., attendance), discussing administrative concerns (e.g., technical issues with website), and other topics.

PCIT coaches quiz

The PCIT Coaches Quiz (PCQ; Funderburk & Nelson, 2014) measures clinician knowledge of PCIT treatment protocol and skills. This 24-item multiple-choice questionnaire includes items related to PCIT protocol, example scenarios, and evaluating mock in-session coding sheets. Scoring of this measure corresponds with the number of items that a clinician answers correctly, with scores ranging from 0 to 24.

Clinician PCIT skills checklist

This 19-item checklist assesses skill and was developed according to competencies that align with the Parent-Child Interaction Therapy (PCIT) International Training Guidelines (2013). These competencies were reviewed and approved by an expert PCIT panel, and include a combination of evaluation at the in-person training, consultation calls, and video review. Completed by the PCIT trainer, this checklist addresses core competencies of PCIT including PCIT assessment, CDI skills, PDI skills, PCIT general skills, and completion of training cases. Trainers indicate the date by which each clinician achieved mastery on each core competency.

Usage rating profile-intervention

The Usage Rating Profile-Intervention measure (URP-I; Chafouleas, Riley-Tillman, Briesch, & Chanese, 2008) evaluates implementation outcomes. The URP-I is a 35-item 6-point Likert-type self-report measure, which ranges from 1 to 6 (strongly disagree to strongly agree), developed to understand intervention usage as determined by four subscales: acceptability, understanding, feasibility, and systems support. The acceptability and feasibility subscales were utilized as they are consistent with the taxonomy of implementation outcomes proposed by Proctor et al. (2011). Results of an exploratory factor analysis support the measure’s use of four subscales, and evidence for internal consistency reliability was established with an alpha greater than .70 for all subscales (Chafouleas, Briesch, Riley-Tillman, & McCoach, 2009).

Data Analysis

All data analyses were conducted in Statistical Package for the Social Sciences (version 24). Knowledge was measured at pretraining (Time-point 1) and posttraining (Time-point 3). A knowledge change score was calculated by subtracting pretraining scores from posttraining scores, with a positive change score indicating an increase in knowledge. Although change scores have more recently been accepted as a way of analyzing change with two time-points, it is still necessary to adjust for baseline levels (Glymour, Weave, Berkman, Kawachi, & Robins, 2005). Therefore, a mean knowledge score was computed for each participant at pretraining and posttraining and included in the model as a predictor. This method has previously been utilized to correct issues with change scores, although it provides a more conservative estimate of actual change (Turiano et al., 2012). For skill, acceptability, and feasibility, a posttraining (Time-point 3) score was utilized as the outcome.

To examine whether consultation call attendance was associated with clinician outcomes (knowledge change, skill) and implementation outcomes (acceptability, feasibility), we estimated a series of regression analyses for the four separate outcomes. For each outcome, in Model 1 we included consultation call attendance and in Model 2 we included consultation call attendance, and individual clinician characteristics (caseload, licensure, baseline knowledge, years of experience, experience with preschool children).

Moderation analyses were conducted to determine if clinician characteristics altered the association between consultation call attendance and knowledge change, posttraining skill, acceptability, and feasibility. Based on previous research, the following variables were hypothesized to be moderators that could impact the strength of consultation’s effect on posttraining outcomes: PCIT caseload, licensure, baseline knowledge, years of experience, and experience with preschool children. Moderation analyses were conducted through hierarchical regression analyses. In step 1, consultation call attendance and the hypothesized moderator (i.e., PCIT caseload) were entered into the model. At step 2, a centered interaction term of both variables was entered into the model.

Results

Descriptive Statistics

Preliminary analyses were conducted to determine if violations occurred in normality, linearity, homoscedasticity, and multicollinearity. Skewness and kurtosis values fell within acceptable limits between −2 and +2, indicating normality of data. Scatterplots indicated a linear relationship between predictor variables and the outcomes of knowledge, skill, acceptability, and feasibility. Homoscedasticity was examined through visual inspection of residual scatterplots with standardized predicted values plotted against standardized residual values. An analysis of variance inflation factor and tolerance levels for the regression models indicated no violations in multicollinearity.

Table 2 presents correlations between all study variables. Participants attended an average of 17.60 (M = 74%, SD = 28.2%) of 24 consultation calls. Participants had an average PCIT caseload of 3.81 families (SD = 4.68), with five (15.6%) clinicians completing two cases over the course of training and consultation calls. Mean knowledge percentage scores increased from baseline (M = 39.7, SD = 10.40) to posttraining (M = 72.1, SD = 13.60). Participants met an average of 10.25 (SD = 3.65) competencies throughout training. The average acceptability rating for participants was 69.4 (SD = 6.40), while the average feasibility rating for participants was 39.74 (SD = 10.43).

Table 2.

Bivariate Correlations Between All Study Variables (n = 32)

Variables M (SD) 1 2 3 4 5 6 7 8 9
1. Consult. attendance 74.06 (28.16)
2. Baseline knowledge 39.74 (10.43) −.07
3. Years exp. 10.22 (7.47) .18 −.10
4. Therapy exp. preschool children 2.53 (.98) .08 −.04 .03
5. Licensure .34 (.48) .03 −.01 .40* .22
6. PCIT caseload 3.81 (4.68) .41* .14 .17 .02 −.16
7. Knowledge change 32.97 (13.29) .52** −.35 .38* .26 .13 .37*
8. Skill 10.25 (3.65) .66* .08 .21 .03 .10 .70* .50**
9. Acceptability 69.40 (6.35) .36* −.01 −.10 .31 −.27 .31 .40* .24
10. Feasibility 36.13 (6.16) −.06 .22 −.22 −.06 −.11 .05 −.22 .04 .51**
Open in a new tab
*

p < .05.

**

p < .01.

Consultation Call Content

On average, 5.10 (SD = 1.20) content topics were discussed per consultation call. Consultants spent an average of 19.32 min (SD = 7.54) reviewing cases, 12.65 min (SD = 7.00) on continued training topics, 12.13 min (SD = 3.39) structuring the call, 5.85 min (SD = 2.26) problem-solving implementation barriers, 3.23 min (SD = 2.00) building skills, 1.47 min (SD = .79) on other topics, and less than one minute on administrative concerns (M = .81, SD = .52), sustainability planning (M = .47, SD = .36), and holding clinicians accountable (M = .18, SD = .18).

Clinician Knowledge and Skill

Table 3 presents the results of the regression analyses on clinician knowledge change and skill. Consultation call attendance (Model 1) significantly predicted greater change in knowledge (F(1, 27) = 9.90, β = .52, p < .01, R2 = .27) and greater posttraining skill (F(1, 30) = 23.10, β = .66, p < .001, R2 = .44). Model 2 predicting change in knowledge from consultation call attendance, caseload, years of experience, licensure, and therapy experience with preschool children was not significant, F(5, 28) = 2.5, p = .72. The same multiple regression model predicting skill was significant (F(6, 30) = 9.0, p < .001, R2 = .69), with consultation call attendance (β = .47, p < .01), and caseload uniquely predicting skill (β = .54, p < .001).

Table 3.

Regression Analyses Predicting Clinician Knowledge Change and Skill

Knowledge change Skill
Model 1 Model 2 Model 1 Model 2
Variable B (SE B) β B (SE B) β B (SE B) β B (SE B) β
Consultation attendance .06 (.02) .48* .04 (.02) .34 .09 (.02) .66*** .06 (.02) .47**
Baseline knowledge .12 (.25) .09 .01 (.04) .03
Licensure −.17 (1.34) −.02 1.36 (1.02) .17
PCIT caseload .10 (.15) .15 .42 (.10) .54***
Therapy exp. preschool children .47 (.64) .14 −.57 (.46) −.15
Years experience .09 (.19) .19 −.03 (.07) −.05
R2 .290 .360 .435 .691
Open in a new tab

Note. Model 1 includes only consultation call attendance; Model 2 includes consultation call attendance and clinician characteristics.

*

p < .05.

**

p < .01.

***

p < .001.

Implementation Outcomes

Table 4 presents the results of the regression analyses from consultation call attendance predicting acceptability and feasibility. Consultation call attendance significantly predicted greater acceptability (F(1, 28) = 4.30, β = .36, p < .05, R2 = .13), but not feasibility, F(1, 28) = .10, p = .75. The multiple linear regression model was not significant in predicting acceptability, F(6, 28) = 2.5, p = .06, or feasibility, F(6, 28) = .26, p = .95.

Table 4.

Regression Analyses Predicting Acceptability and Feasibility

Acceptability Feasibility
Model 1 Model 2 Model 1 Model 2
Variable B (SE B) β B (SE B) β B (SE B) β B (SE B) β
Consultation attendance .08 (.04) .36* .07 (.04) .40 −.01 (.04) −.06 −.00 (.05) −.02
Baseline knowledge −.01 (.08) −.02 .11 (.12) .20
Licensure −3.43 (2.08) −.31 .75 (2.94) .06
PCIT caseload .20 (.21) .18 .06 (.30) .05
Therapy exp. preschool children 1.73 (.95) .33 −.62 (1.33) −.10
Years experience −.02 (.14) −.03 −.07 (.20) −.08
R2 .132 .403 .066
Open in a new tab

Note. Model 1 includes only consultation call attendance; Model 2 includes addition of clinician characteristics.

*

p < .05.

**

p < .01.

***

p < .001.

Moderation Analyses

PCIT caseload was the only moderator found to be significant in the relationship between consultation call attendance and skill (see Figure 1). At step 1 of the hierarchical linear regression, consultation call attendance and caseload significantly predicted skill (F(2, 32) = 22.5, p < .001, R2 = .71, with consultation call attendance uniquely predicting skill (β = .45, p < .01). At step 2, the interaction term (consultation call attendance * caseload) was entered into the model significantly predicting skill (b = .02, p < .05).

Figure 1.

Figure 1

Open in a new tab

Results of the moderating effect of caseload on the relation between consultation call attendance and skill. Low case = 1 SD below the mean; high case = 1 SD above the mean.

Discussion

Overall Findings

As there is a growing need to acquire skill in a variety of EBTs, it is important that the most effective trainings are available to the community of professional psychologists. Although brief workshops and trainings are commonly utilized training methods, the current study lends support to a growing body of literature demonstrating the importance of ongoing support as a component of training protocols for EBTs (Herschell et al., 2010; Lyon, Stirman, Kerns, & Bruns, 2011). Our initial hypotheses predicted that greater consultation call attendance would be associated with greater change in knowledge and posttraining skill, acceptability, and feasibility. These hypotheses were partially supported. Simple linear regression analyses supported the importance of consultation call attendance in predicting knowledge change, skill, and acceptability. Yet when additional clinician variables were included in the model, consultation call attendance only significantly predicted posttraining skill. Overall, these findings suggest that consultation call attendance is uniquely associated with increased clinician skill in PCIT, despite various clinician characteristics.

Notably, the relation between consultation call attendance and skill was qualified by a significant interaction between consultation call attendance and PCIT caseload. This interaction suggests that clinicians with a high caseload who attend a greater number of consultation calls reap the most benefits in skill. Furthermore, this implies that increases in clinician skill occur as a result of the interplay between training and experience with cases. Additionally, clinicians with a low caseload demonstrated a similar skill level regardless of the amount of consultation calls attended, lending support to this finding. Thus, increasing clinician skill requires a combination of ongoing support as well as the opportunity to utilize newly learned skills with clients. However, clinicians who had a high caseload, but attended few consultation calls, demonstrated the lowest skill level in the current study. One possible explanation for this finding is that clinicians who had a high PCIT caseload but did not attend additional consultation calls may have had such a high caseload that it prohibited them from full participation in training. For example, within the field of child welfare, having an above-average caseload has been related to inadequate training and supervision time (Yamatani, Engel, & Spjeldnes, 2017). Therefore, it is possible that this group of clinicians were also unable to attend consultation calls due to a lack of available time. Further research should seek to understand the factors that distinguish clinicians who were able to attend a great number of consultation calls from those who were not.

Interestingly, this study did not find support for the importance of consultation calls in predicting implementation outcomes of feasibility and acceptability. It was hypothesized that clinicians who attended a greater number of consultation calls would perceive PCIT as being more feasible and acceptable to utilize with their clients. However, this study did not find evidence in support of this hypothesis. It is possible that improving implementation outcomes may be more related to organizational characteristics as opposed to training design factors. Research is beginning to reveal that organizational characteristics, such as organizational climate and supervisory support, influence the degree to which a clinician implements a treatment (Beidas, Cross, & Dorsey, 2014; Ditty et al., 2015; Glisson, 2002). It is possible that organizational-level characteristics not included in the current study may have affected acceptability and feasibility. Therefore, additional research should explore if organizational factors are also related to the implementation outcomes of feasibility and acceptability.

While the present study did find PCIT caseload to be a moderator of consultation call attendance and skill, no other significant moderators were found. This finding differs from previous research that suggests the importance of clinician characteristics in the implementation of evidence-based treatments (Beidas, Edmunds, et al., 2014; Lewis & Simons, 2011; Velada et al., 2007). Prior research has not examined the influence of clinician characteristics on consultation call outcomes. However, the growing research dedicated to individual clinician characteristics suggests that it should be an area for future research within the training literature.

Limitations

Directionality of associations

This study has examined the correlational association between consultation call attendance, clinician variables, and clinician and implementation outcomes. While evidence in support of the importance of consultation call attendance and caseload on clinician outcomes was found, the nature of this association is unclear. The current study hypothesized that consultation call attendance is a predictor of clinician caseload, suggesting that clinicians who attend more consultation calls have a higher caseload. However, it is also possible that clinician caseload predicts consultation call attendance. In this example, clinicians who are treating more families may be likely to attend more calls, given that a core component of consultation calls involves reviewing cases. Similarly, clinicians who have a small number of cases may be more likely to attend fewer consultation calls. To further understand the impact of consultation, experimental methods comparing consultation groups to individuals who did not receive consultation should be utilized. Unfortunately, in the current study, a majority of clinicians attended at least one consultation call, and therefore, comparisons between these two groups could not be made. Additional research should seek to clarify the directionality of this association, while utilizing three waves of measurement to analyze change as well as experimental methods.

Measurement

Several measures included in the current study (e.g., Clinician PCIT Skills Checklist, PCIT Case Consultation Record) have not demonstrated adequate psychometric evidence, limiting the interpretations that can be drawn. While the Clinician PCIT Skills Checklist was utilized given that it aligns with PCIT International’s Training Guidelines (2013) and certification process, it has not demonstrated adequate reliability or validity in assessing clinician skill. The field of implementation science as a whole has been limited by a lack of measures that have appropriate psychometric properties, in addition to relying upon study-developed measures (Martinez, Lewis, & Weiner, 2014). The lack of adequate measurement weakens the findings of the current study.

In addition to the lack of psychometric evidence for measures in the current study, several items were completed through video review of sessions with clients (e.g., CDI Coach Session). However, if clinicians were unable to enroll clients on their caseload into the study due to a variety of reasons (e.g., lack of consent, lack of referrals), clinicians were unable to meet competencies requiring video review. In order to reduce this issue, during the consultation phase, trainers attempted to problem-solve the barrier of clinicians not receiving enough referrals. Additionally, clinicians were encouraged to enroll a large number of families in the study in order to submit videos given the possibility of families dropping out of treatment or not consenting to the study.

Nesting

The current study involves clinicians, who are nested within a supervisor, who is nested within an organization. However, given the small sample size utilized in the current study, hierarchical linear modeling procedures could not be utilized. Therefore, the interpretations that can be made from this study are limited in that the variations in clinician skill may also be due in part to supervisor or organizational factors.

Sample size

While a power analysis was conducted and the sample size was determined to be appropriate for the simple linear regression analyses described previously, it is possible that the study was underpowered to detect moderation effects. Further research should examine how therapist characteristics influence the association between consultation call attendance and outcomes. Additionally, the current study was underpowered to perform growth curve analyses that may have been more amenable to our hypotheses in examining change over time as a result of consultation call attendance.

Future Directions

These findings demonstrate the importance of analyzing training design components that influence clinician knowledge and skill outcomes. The consultation calls in the current study primarily focused on reviewing clinician cases; however, additional research should focus on targeting which content areas optimize training outcomes. For example, some research has found support for the importance of skill-building activities in supervision for promoting clinician fidelity (Beidas, Cross, et al., 2014). The current study found that consultants did not devote substantial time to building skills; however, it may be an important aspect of ongoing support. Additional research should focus on the mechanisms and content areas that underlie the effectiveness of ongoing support strategies such as consultation. Additionally, the current study examined one form of ongoing support, consultation, in regard to these outcomes. While consultation is widely utilized in large-scale training initiatives, other forms of ongoing support (e.g., supervision) should also be examined.

In addition, the present study has highlighted the importance of the training design input factor in producing positive training outcomes. Although research has examined how individual clinician characteristics (Beidas, Edmunds, et al., 2014; Lewis & Simons, 2011; Velada et al., 2007) and organizational characteristics (Beidas et al., 2015; Ditty et al., 2015) influence implementation outcomes, these input factors remain largely unstudied in relation to training clinicians in EBTs. Further research should examine other training input factors (e.g., work-environment factors) of Baldwin and Ford’s (1988) model that influence training outcomes for clinicians.

Prior research on training outcomes has generally isolated one of Baldwin and Ford’s (1988) training input factors when examining outcomes. This study attempted to understand the interaction between training design and individual trainee characteristics by examining moderators of consultation’s effectiveness. While PCIT caseload was found to be a moderator of consultation’s effectiveness, it is likely that other individual trainee characteristics or organizational factors may moderate this relationship. Additional research should seek to disentangle the interactions between training design, individual trainee characteristics, and organizational characteristics as these are the complex relationships found in real-world settings that professional psychologists are trained within. Understanding the interface between training design and real-world implementation settings may result in improved training effectiveness, ultimately leading to the increased availability and accessibility of EBTs in community settings.

Public Significance Statement.

This study advances the importance of consultation as a training component for professional psychologists. Additionally, it highlights the significance of gaining experience through therapy cases in combination with attending consultation calls to become skillful at providing an evidence-based treatment.

Acknowledgments

We thank the study participants and statewide steering committee for their insightful comments and participation in this project; Shelley Hiegel, Kristen Schaffner, Ashley Scudder, and Sarah Taber-Thomas for their organization of training efforts and recruitment of participants; Jonathan Hart, Allison Leonard, Christopher Owen, and Jeffrey Rounds for their assistance with data collection and data analysis. This research was funded by a grant from the National Institute of Mental Health (NIMH; R01 MH095750; A Statewide Trial to Compare Three Training Models for Implementing an EBT; PI: Herschell).

Biographies

Carrie B. Jackson received her MS in psychology from West Virginia University. She is currently a graduate student in the psychology doctoral program, clinical child specialization, at West Virginia University. Her areas of professional interest include implementation science, training methods for mental health providers, the sustainability of evidence-based treatments in community settings, and child maltreatment.

Amy D. Herschell received her PhD in psychology, clinical child specialization, from West Virginia University. She is currently an associate professor of psychology and family medicine at West Virginia University and an associate professor of psychiatry and psychology at the University of Pittsburgh School of Medicine. Her areas of professional interest include studying the implementation of evidence-based treatments in community settings, community-based behavioral health services and perspectives, developing and testing implementation models, and young children with externalizing behavior problems and a history of harsh parenting.

Kristen F. Schaffner received her PhD in school psychology from Duquesne University. She is currently a licensed psychologist and nationally certified school psychologist, working as a project coordinator at West Virginia University, Morgantown, West Virginia and research project clinician at the University of Pittsburgh Medical Center. Her areas of professional interest include disruptive behavior disorders, young children, Parent–Child Interaction Therapy, training in evidence-based practice, community implementation, as well as early care and education.

Nicholas A. Turiano received his PhD in human development and family studies from Purdue University. He is currently an assistant professor at West Virginia University in the Life Span Developmental Psychology program. His research program focuses on how personality traits are associated with various health outcomes such as mortality. Specific emphasis is placed on exploring how behavioral, social network, and physiological processes explain why personality is associated with health.

Cheryl B. McNeil received her PhD in clinical psychology from the University of Florida. She is currently a professor of psychology in the Clinical Child program at West Virginia University. Her clinical and research interests are focused on program development and evaluation, specifically with regard to abusive parenting practices and managing the disruptive behaviors of young children in both the home and school settings.

Contributor Information

Carrie B. Jackson, West Virginia University

Amy D. Herschell, West Virginia University and University of Pittsburgh

Kristen F. Schaffner, West Virginia University and University of Pittsburgh

Nicholas A. Turiano, West Virginia University

Cheryl B. McNeil, West Virginia University

References

  1. Baldwin TT, Ford JK. Training transfer: A review and directions for future research. Personnel Psychology. 1988;41:63–104. doi: 10.1111/j.1744-6570.1988.tb00632.x. [DOI] [Google Scholar]
  2. Bearman SK, Weisz JR, Chorpita BF, Hoagwood K, Ward A, Ugueto AM The Research Network on Youth Mental Health. More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research. 2013;40:518–529. doi: 10.1007/s10488-013-0485-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Beidas RS, Cross W, Dorsey S. Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice. 2014;21:1–11. doi: 10.1016/j.cbpra.2013.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Beidas RS, Edmunds J, Ditty M, Watkins J, Walsh L, Marcus S, Kendall P. Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety? Administration and Policy in Mental Health and Mental Health Services Research. 2014;41:788–799. doi: 10.1007/s10488-013-0529-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beidas RS, Edmunds JM, Marcus SC, Kendall PC. Training and consultation to promote implementation of an empirically supported treatment: A randomized trial. Psychiatric Services. 2012;63:660–665. doi: 10.1176/appi.ps.201100401. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, Mandell DS. Individual and organizational factors related to community clinicians’ use of therapy techniques in a large public mental health system. Journal of the American Medical Association Pediatrics. 2015;34:355–368. [Google Scholar]
  7. Caplan G. The theory and practice of mental health consultation. New York, NY: Basic Books; 1970. [Google Scholar]
  8. Campbell R, Raja S, Grining PL. Training mental health professionals in violence against women. Journal of Interpersonal Violence. 1999;14:1003–1013. doi: 10.1177/088626099014010001. [DOI] [Google Scholar]
  9. Chafouleas SM, Briesch AM, Riley-Tillman TC, McCoach DB. Moving beyond assessment of treatment acceptability: An examination of the factor structure of the Usage Rating Profile–Intervention (URP-I) School Psychology Quarterly. 2009;24:36–47. doi: 10.1037/a0015146. [DOI] [Google Scholar]
  10. Chafouleas SM, Riley-Tillman TC, Briesch AM, Chanese JM. Generating usable knowledge: Initial development and validation of the Usage Rating Profile for Interventions (URP-I) Canadian Journal of School Psychology. 2008;23:175–189. doi: 10.1177/0829573508323989. [DOI] [Google Scholar]
  11. Ditty MS, Landes SJ, Doyle A, Beidas RS. It takes a village: A mixed method analysis of inner setting variables and dialectical behavior therapy implementation. Administration and Policy in Mental Health and Mental Health Services Research. 2015;42:672–681. doi: 10.1007/s10488-014-0602-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Edmunds JM, Beidas RS, Kendall PC. Dissemination and implementation of evidence-based practices: Training and consultation as implementation strategies. Clinical Psychology: Science and Practice. 2013;20:152–165. doi: 10.1111/cpsp.12031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Eyberg S, Funderburk B. Parent-child interaction therapy protocol. Gainesville, FL: PCIT, International Publishing; 2011. [Google Scholar]
  14. Funderburk B, Nelson M. PCIT Coaches Quiz 2014 [Google Scholar]
  15. Glisson C. The organizational context of children’s mental health services. Clinical Child and Family Psychology Review. 2002;5:233–253. doi: 10.1023/A:1020972906177. [DOI] [PubMed] [Google Scholar]
  16. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research. 2005;7:243–259. doi: 10.1007/s11020-005-7456-1. [DOI] [PubMed] [Google Scholar]
  17. Glymour MM, Weuve J, Berkman LF, Kawachi I, Robins JM. When is baseline adjustment useful in analyses of change? An example with education and cognitive change. American Journal of Epidemiology. 2005;162:267–278. doi: 10.1093/aje/kwi187. [DOI] [PubMed] [Google Scholar]
  18. Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review. 2010;30:448–466. doi: 10.1016/j.cpr.2010.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Herschell AD, Kolko DJ, Scudder AT, Taber-Thomas S, Schaffner KF, Hiegel SA, Mrozowski S. Protocol for a statewide randomized controlled trial to compare three training models for implementing an evidence-based treatment. Implementation Science. 2015;10:133. doi: 10.1186/s13012-015-0324-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hides L, Lubman DI, Elkins K, Catania LS, Rogers N. Feasibility and acceptability of a mental health screening tool and training programme in the youth alcohol and other drug (AOD) sector. Drug and Alcohol Review. 2007;26:509–515. doi: 10.1080/09595230701499126. [DOI] [PubMed] [Google Scholar]
  21. Jensen-Doss A, Hawley KM, Lopez M, Osterberg LD. Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice. 2009;40:417–424. doi: 10.1037/a0014690. [DOI] [Google Scholar]
  22. Karlsson A, Bendtsen P. Acceptability of a computerized alcohol screening and advice routine in an emergency department setting—A patient perspective. Addictive Behaviors. 2005;30:767–776. doi: 10.1016/j.addbeh.2004.08.022. [DOI] [PubMed] [Google Scholar]
  23. Kolko DJ, Baumann BL, Herschell AD, Hart JA, Holden EA, Wisniewski SR. Implementation of AF-CBT by community practitioners serving child welfare and mental health: A randomized trial. Child Maltreatment. 2012;17:32–46. doi: 10.1177/1077559511427346. [DOI] [PubMed] [Google Scholar]
  24. LeJeune JT, Luoma JB. The integrated scientist–practitioner: A new model for combining research and clinical practice in fee-for-service settings. Professional Psychology: Research and Practice. 2015;46:421–428. doi: 10.1037/pro0000049. [DOI] [Google Scholar]
  25. Lewis CC, Simons AD. A pilot study disseminating cognitive behavioral therapy for depression: Therapist factors and perceptions of barriers to implementation. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:324–334. doi: 10.1007/s10488-011-0348-x. [DOI] [PubMed] [Google Scholar]
  26. Lyon AR, Stirman SW, Kerns SEU, Bruns EJ. Developing the mental health workforce: Review and application of training approaches from multiple disciplines. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:238–253. doi: 10.1007/s10488-010-0331-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implementation Science. 2014;9:118. doi: 10.1186/s13012-014-0118-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. McNeil CB, Hembree-Kigin TL. Parent-Child Interaction Therapy. 2. New York, NY: Springer; 2010. [Google Scholar]
  29. Myers KM, Valentine JM, Melzer SM. Feasibility, acceptability, and sustainability of telepsychiatry for children and adolescents. Psychiatric Services. 2007;58:1493–1496. doi: 10.1176/ps.2007.58.11.1493. [DOI] [PubMed] [Google Scholar]
  30. Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration and Policy in Mental Health and Mental Health Services Research. 2013;40:439–450. doi: 10.1007/s10488-013-0502-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Parent-Child Interaction Therapy International. PCIT Certification. 2013 Retrieved from http://www.pcit.org/pcit-certification.html.
  32. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Hensley M. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Rieckmann T, Bergmann L, Rasplica C. Legislating clinical practice: Counselor responses to an evidence-based practice mandate. Journal of Psychoactive Drugs. 2011;43:27–39. doi: 10.1080/02791072.2011.601988. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Rodríguez A, Southam-Gerow MA, O’Connor MK, Allin RB., Jr An analysis of stakeholder views on children’s mental health services. Journal of Clinical Child and Adolescent Psychology. 2014;43:862–876. doi: 10.1080/15374416.2013.873982. [DOI] [PubMed] [Google Scholar]
  35. Schoenwald SK, Carter RE, Chapman JE, Sheidow AJ. Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35:379–394. doi: 10.1007/s10488-008-0181-z. [DOI] [PubMed] [Google Scholar]
  36. Schoenwald SK, Chapman JE, Sheidow AJ, Carter RE. Long-term youth criminal outcomes in MST transport: The impact of therapist adherence and organizational climate and structure. Journal of Clinical Child and Adolescent Psychology. 2009;38:91–105. doi: 10.1080/15374410802575388. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Schoenwald SK, Sheidow AJ, Letourneau EJ. Toward effective quality assurance in evidence-based practice: Links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology. 2004;33:94–104. doi: 10.1207/S15374424JCCP3301_10. [DOI] [PubMed] [Google Scholar]
  38. Scudder A, Herschell AD. Building an evidence-base for the training of evidence-based treatments in community settings: Use of an expert-informed approach. Children and Youth Services Review. 2015;55:84–92. doi: 10.1016/j.childyouth.2015.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Stirman SW, Pontoski K, Creed T, Xhezo R, Evans AC, Beck AT, Crits-Christoph P. A non-randomized comparison of strategies for consultation in a community–academic training program to implement an evidence-based psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research. 2017;44:55–66. doi: 10.1007/s10488-015-0700-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Turiano NA, Pitzer L, Armour C, Karlamangla A, Ryff CD, Mroczek DK. Personality trait level and change as predictors of health outcomes: Findings from a national study of Americans (MIDUS) The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences. 2012;67B:4–12. doi: 10.1093/geronb/gbr072. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Velada R, Caetano A, Michel JW, Lyons BD, Kavanagh MJ. The effects of training design, individual characteristics and work environment on transfer of training. International Journal of Training and Development. 2007;11:282–294. doi: 10.1111/j.1468-2419.2007.00286.x. [DOI] [Google Scholar]
  42. Weissman MM, Verdeli H, Gameroff MJ, Bledsoe SE, Betts K, Mufson L, Wickramaratne P. National survey of psychotherapy training in psychiatry, psychology, and social work. Archives of General Psychiatry. 2006;63:925–934. doi: 10.1001/archpsyc.63.8.925. [DOI] [PubMed] [Google Scholar]
  43. Yamatani H, Engel R, Spjeldnes S. Child welfare worker caseload: What’s just right. Social Work. 2017;54:361–368. doi: 10.1093/sw/54.4.361. [DOI] [PubMed] [Google Scholar]

RESOURCES