Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective

Abstract

Evidence-based practice (EBP), a preferred psychological treatment approach, requires training of community providers. The systems-contextual (SC) perspective, a model for dissemination and implementation efforts, underscores the importance of the therapist, client, and organizational variables that influence training and consequent therapist uptake and adoption of EBP. This review critiques the extant research on training in EBP from an SC perspective. Findings suggest that therapist knowledge improves and attitudinal change occurs following training. However, change in therapist behaviors (e.g., adherence, competence, and skill) and client outcomes only occurs when training interventions address each level of the SC model and include active learning. Limitations as well as areas for future research are discussed.

The American Psychological Association (APA) and the American Academy of Child and Adolescent Psychiatry (AACAP) support the provision of evidence-based practice (EBP; American Academy of Child and Adolescent Psychiatry, 2006; American Psychological Association, 2005). However, a report by the United States Surgeon General (1999) suggests that the majority of clients with mental illness do not receive EBP. There are obstacles in the dissemination and implementation (DI) of EBP into clinical practice (e.g., criticism of treatment manuals, inadequate training, and unsupportive organizational climates). Understanding how to best disseminate EBP is paramount to reducing the gap between research and practice (Addis & Krasnow, 2000; Hayes, 2002; Herschell, McNeil, & McNeil, 2004).

Multiple terms have been used interchangeably, and at times inaccurately, in this area (Kendall & Beidas, 2007). EBP¹ as defined by the American Psychological Association (2005) is “the integration of the best available research with clinical expertise.” ESTs refer to psychological interventions that have been evaluated scientifically (e.g., a randomized controlled trial, RCT) and satisfy the criteria outlined in Chambless and Hollon (1998). DI research includes the purposeful distribution of relevant information and materials to therapists (i.e., dissemination) and the adoption and integration of EPB into practice (i.e., implementation; Lomas, 1993). Our focus is on training as it relates to DI research: How does training influence therapist knowledge and behavior (adherence, competence, and skill), and how does the therapist’s context (organizational support and client population) influence adoption and implementation of interventions?

To understand training as it relates to DI, a systems-contextual (SC) approach is warranted. The SC perspective recommends that when considering the effects of training on therapist behavior, contextual factors such as therapist variables, organizational support, quality of training program, and client variables (Sanders & Turner, 2005; Turner & Sanders, 2006) should be examined. Studying the outcomes of training without addressing contextual variables is insufficient because training and potential implementations occur within a system. Effective DI may occur when therapists are trained appropriately and when the context supports behavior change (Sanders & Turner, 2005). Such an approach is holistic and provides an understanding of how training influences an individual within a system. Each component of the SC model (i.e., quality of training, practitioner variables, client variables, and organizational support) will be operationalized and expanded upon below.

The quality of training provided to therapists refers to the availability of training for individuals interested in implementing an EBP, as well as the content and method of the training. The current “gold standard” of training in EBP includes a workshop, a manual, and clinical supervision (Sholomskas, Syracuse-Siewert, Rounsaville, Ball, & Nuro, 2005). The quality of training in EBP is likely to vary based on when therapists received their graduate training. Therapists trained before 1995 are unlikely to have had an emphasis on EBP, while those trained after 1995 have had varied exposure to EBP during training (Karekla, Lundgren, & Forsyth, 2004). A 1995 survey of training directors found that doctoral programs in clinical psychology covered about 50% of EBP (specifically—ESTs) and that most internships did not require competence in EBP (specifically—ESTs; Crits-Christoph, Frank, Chambless, Brody, & Karp, 1995). Similarly, a survey of 200 trainees in APA-accredited programs found that 32% had no coursework in EBP (specifically—ESTs) and over 65% had not read any of the major Task Force reports regarding EBP (specifically—ESTs). Those who had read the Task Force reports tended to be cognitive-behavioral in orientation (Karekla et al., 2004). These findings suggest that experienced therapists are not likely to have received training in EBP, and that newer therapists may have received training in some aspects of EBP.

Importantly in our view, the content and method (i.e., overall quality) of the training program is crucial to successful DI. Some suggest that training content is important and prefer a focus on principles, rather than teaching the details of manuals (Abramowitz, 2006; Hayes, 2002; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004). Miller et al. (2004) noted that the appropriate training approach includes an emphasis on principles and the “underlying spirit” of a treatment rather than a focus on techniques. Further empirical study is needed to determine the appropriate focus of training content.

Training method is an important vehicle through which change in therapist behavior may be achieved. Current training methods include passively delivered didactic lectures (e.g., the format of a continuing education workshop) despite findings that this type of instruction has limited effects on behavior change (El-Tannir, 2002). Active learning, an alternative to passive learning, is an interactive process that uses action and reflection. Active learning is useful for skills that must be employed within a clinical context (Cross, Matthieu, Cerel, & Knox, 2007) and has been employed successfully in the Triple P-Positive Parenting Program (Triple P) training program (Sanders & Turner, 2005). Cross et al. (2007) endorse active learning methods (modeling, practice opportunities, building self-efficacy, and interaction among learners), which may be accomplished through behavioral role-plays. Including behavioral role-plays as part of training improved behavior in both clinical and nonclinical employees in suicide prevention efforts (Cross et al., 2007; Matthieu, Cross, Batres, Flora, & Knox, 2008). Change in behavior may be less robust without the inclusion of behavioral role-plays in training (Wyman et al., 2008). Further empirical study of the contribution of active learning to effective DI efforts is needed.

Therapist attributes may be an important aspect of whether training produces differential learning and subsequent behavior change. Therapist variables include individual attributes such as clinical experience, theoretical orientation, and therapist attitudes towards EBP. Conflicting evidence exists regarding whether or not prior clinical experience influences skill acquisition. One study found that prior general clinical experience did not influence therapist skill acquisition in cognitive therapy (CT), but that prior specific clinical experience in CT facilitated skill acquisition (James, Blackburn, Milne, & Reichfelt, 2001). Another study comparing practicing therapists to trainee therapists suggested that clinical experience did not moderate skill attainment (DeViva, 2006).

Therapist attitudes towards EBP have been examined in survey format (e.g., Addis & Krasnow, 2000; Najavits, Weiss, Shaw, & Dierberger, 2000), with mixed results. Some suggest that therapists hold favorable attitudes towards EBP (Najavits et al., 2000), whereas others suggest that therapists hold unfavorable attitudes towards EBP (Addis & Krasnow, 2000). Theoretical orientation may be an individual difference that explains these mixed results: In the first study, participants were mainly identified as cognitive-behavioral, whereas in the second study, a large variety of theoretical orientations were surveyed. Another therapist difference explaining these discrepant findings may be clinical experience: Those earlier on in their careers (e.g., predoctoral interns) may hold more favorable attitudes towards EBP when compared with those who are more advanced in their careers (e.g., practicing therapists; Aarons, 2004). Few studies have explicitly examined therapist attributes and attitudes towards EBP as predictors of training outcomes, which may inform who will benefit most from training.

Variables relating to organizational support (e.g., clinical supervision and organizational environment) can also impact therapist training outcomes. Reading a manual and attending a workshop may start the transfer of knowledge (i.e., dissemination), but ongoing supervision may be needed for actual therapist behavior change and skillful implementation (Bazelmans, Prins, Hoogveld, & Bleijenberg, 2004; Herschell et al., 2004; Kendall & Southam-Gerow, 1996). Research on evidence-based supervision is sparse (Ellis, Krengel, Ladany, & Schult, 1996; Holloway & Neufeldt, 1995), but it is likely that learning occurs and confidence is built during the supervisory process. Competent supervisors may be related to therapist adherence and competence (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002), two therapist behaviors that are crucial for fidelity to a treatment (Perpepletchikova & Kazdin, 2005). In addition, continued supervision may decrease therapist psychological barriers to adoption of EBP (specifically ESTs; Luoma et al., 2007).

The particulars on supervision differ by intervention, but one feasible model comes from Turner and Sanders’s (2006) Triple P approach. Supervision includes updates on Triple P research through newsletters, conferences, a website, and a question and answer forum. The approach includes self-monitoring and self-regulation, which allows therapists to direct their own learning and skill acquisition following training (Sanders & Turner, 2005). Other organizational support variables important for therapist behavior change include organizational openness to change and an organizational structure that supports implementation of EBP (Zazzali et al., 2008).

Client variables must be accounted for when considering successful therapist training outcomes. Client variables include a therapist’s belief that a particular EBP can be useful for his or her client population when considering the severity and risk factors of such clients (Turner & Sanders, 2006). Some believe that research samples are not representative of community samples because the selection criteria of RCTs are not generally inclusive and comorbidity is not highly represented (Westen, Novotny, & Thompson-Brenner, 2004). However, other researchers (e.g., Stirman, DeRubeis, Crits-Christoph, & Rothman, 2005) have demonstrated that these differences may be overstated. Stirman et al. (2005) mapped charts of individuals seeking treatment under managed care to the criteria of nearly 100 RCTs and identified that 80% of these individuals would be eligible for at least one RCT, and the majority did not have more complex diagnostic profiles than participants included in RCTs. It is important to address therapists, concerns that a treatment is viable for their clients and that a treatment allows for flexible application (Kendall & Beidas, 2007). Additionally, given the ultimate goal of DI (i.e., client access to EBP), client outcomes following therapist training are key dependent variables that must be examined.

This review examines studies that train therapists in EBP from an SC perspective as recommended by Sanders and Turner (2005) and Turner and Sanders (2006). This perspective is ecological in nature in that it views DI as occurring through complex bidirectional processes between the practitioner, practitioner’s environment, and quality of the training (Sanders & Turner, 2005). Additionally, the perspective views the practitioner as embedded within a broader working environment that influences practitioner implementation of innovation (Sanders & Turner, 2005). The primary aim of this review was to identify training studies and consider which level of the SC model (i.e., therapist variables, client variables, organizational support, and training) was addressed. Secondarily, we characterize the quality of the training, with an emphasis on training method (e.g., active and / or passive learning strategies) and content (i.e., a focus on principles or the teaching of a manual session by session).

METHODS

A systematic literature review for the 18-year period spanning 1990–2008 was conducted. Published journal articles, abstracts, and books were identified via Psych-Info and PubMed. Key words included “training,” “dissemination,” “implementation,” “adherence,” “competence,” and “therapists.” Additionally, reference lists of relevant articles guided the identification of appropriate studies. Studies were included if they trained service providers (e.g., social workers, psychologists, physicians, substance abuse counselors, secondary school staff, master’s-level clinicians, and nurses) in EBP or ESTs for at-risk or clinical populations.² Training studies not affiliated with an evidence base were not included, nor were trainings in single techniques. All studies included in this review focused on training in EBP (see Table 1). Unpublished theses or dissertations were not included.

Table 1.

Evidence-based practices (EBPs) included in this study

EBP	References	Classifying body deeming the treatment to be an EBP
Dialectical behavior therapy for borderline personality disorder	Hawkins and Sinha (1998)	A
Motivational interviewing for substance abuse	Baer et al. (2004); Hunter et al. (2005); Miller and Mount (2001); Miller et al. (2004); Rubel et al. (2000); Schoener et al. (2006); Saitz et al. (2000)	A
Cognitive behavior therapy for eating disorders	McVey et al. (2005)	A
Interpersonal therapy for eating disorders	McVey et al. (2005)	A
Cognitive-behavioral therapy for child anxiety	Beidas et al. (2009)	A
Behavioral therapy for anxiety	Gega, Norman, & Marks (2007)	A
Trauma-focused cognitive-behavioral therapy	National Crime Victims Research & Treatment Center (2007)	C
Intervention for youth suicide	Chagnon et al. (2007); Cross et al. (2007); Wyman et al. (2008)	B
Cognitive and / or cognitive-behavioral therapy for substance abuse	Crits-Christoph et al. (1998); Morganstern et al. (2001); Sholomskas et al. (2005); Siqueland et al. (2000); Watkins et al. (2008)	A
Dynamic therapy for substance abuse	Crits-Christoph et al. (1998); Siqueland et al. (2000)	A
Drug counseling for substance abuse	Crits-Christoph et al. (1998); Luoma et al. (2007); Siqueland et al. (2000)	B
Triple P parenting program for at-risk youth	Sanders, Murphy-Brennan, et al. (2003)	B
Time-limited dynamic psychotherapy for adults with Axis I and II disorders	Bein et al. (2000); Henry, Schacht, et al. (1993); Henry, Strupp, et al. (1993)	A
Behavioral family therapy for relatives caring for family member with schizophrenia	Brooker and Butterworth (1993); Fadden (1997)	A
Cognitive-behavioral therapy for youth and young adults presenting to primary care	Maunder et al. (2008)	A
Contingency management for youth substance abuse	Henggeler, Chapman, et al. (2008); Henggeler, Sheidow, et al. (2008)	C
Multisystemic therapy for youth substance abuse	Henggeler, Sheidow, et al. (2008)	C

Note. As cited in A, Chambless & Ollendick (2001); B, Substance Abuse and Mental Health Services Administration (www.nationalregistry.samhsa.gov); C, Special section of Journal of Clinical Child and Adolescent Psychology (Silverman, Pina, & Viswesvaran, 2008).

We operationalized the four levels of the systems-contextual approach to DI as follows: (a) therapist variables: Study included at least one measure of therapist attitudes and / or a comprehensive measure of clinical experience and theoretical orientation,³ (b) organizational support: Study included at least one measure of organizational characteristics and / or provided ongoing consultation or supervision, (c) quality of training processes:⁴ Study provided training and described training sufficiently so that characteristics of training could be identified, and (d) client variables: Study included at least one measure of severity, risk factors, and resiliency of therapist’s client population, or included a measure of client treatment outcome (e.g., diagnostic interview).

We operationalized the training method as employing passive (e.g., didactic presentation, lectures, and seminars) and / or active learning strategies (e.g., behavioral role-plays, feedback, coaching, and experiential exercises). Training content was defined as focusing on principles of the treatment (i.e., the underlying foundational principles of the EBP) or teaching the manual and going through the treatment session by session.

Outcome measures included both self-reported (e.g., therapist attitudes) and independently rated (e.g., therapist adherence) variables. Studies were classified as using standardized measures if the report indicated adequate psychometric properties (i.e., reliability: intraclass coefficients or kappa values above 0.60; Landis & Koch, 1977; adequate validity) and if the measure was used in more than one research report. Studies were classified as using nonstandardized investigator-created measures if psychometric properties were not reported, were not adequate, or if the measure was only used in one study. Studies that included the outcome measures of knowledge,⁵ adherence, and skill / competence were emphasized given the importance of these constructs in the transportation of a treatment (Perpepletchikova & Kazdin, 2005). Additionally, we note whether gains made in various outcome measures (i.e., knowledge, adherence, and skill / competence) suggest therapist proficiency. An 80% score was used to mark proficiency in therapist knowledge and behavior. Eighty percent was used as the cutoff, consistent with training conditions used in other evaluations of EBP as the criterion to be satisfied prior to being certified to deliver a treatment (e.g., Seng, Prinz, & Sanders, 2006; Sholomaskas et al., 2005; Walkup et al., 2008).

RESULTS

Many DI studies (32) of training have been reported, but few (3) assess and measure all domains of the SC model. Studies in this review are organized as follows: those that address (a) training, (b) training and organizational support, (c) training and therapist variables, (d) training and client variables, (e) training, organizational support, and therapist variables, (f) training, organizational support, and client variables, and (g) training, organizational support, therapist variables, and client variables.⁶

Training

Description of Studies

Eight studies focused on training (see Table 2). The following EBPs were studied: (a) dialectical behavior therapy (DBT) for borderline personality disorder (Hawkins & Sinha, 1998), (b) motivational interviewing (MI) for adult substance abuse (Baer, Rosengren, Dunn, Wells, & Ogle, 2004), (c) CBT and interpersonal therapy (IPT) for eating disorders (ED) (McVey et al., 2005), (d) CBT for youth anxiety (Beidas, Barmish, & Kendall, 2009), (e) trauma-focused CBT (tf-CBT) for traumatized youth (National Crime Victims Research & Treatment Center, 2007), (f) behavior therapy (BT) for anxiety (Gega, Norman, & Marks, 2007), (g) the Question, Persuade, Refer (QPR) prevention program for youth suicide (Cross et al., 2007), and (h) Triple P for externalizing behaviors in youth (Sanders, Tully, Turner, Maher, & McAuliffe, 2003). One study used an RCT design (Gega et al., 2007), whereas the other studies used a nonrandomized, convenience-sample, pre–post quasi-experimental design. Study sample sizes ranged from 20 to 4,387 participants and educational level ranged from bachelor’s to postgraduate degrees (e.g., MD, PhD, and PsyD). Most participants were community mental health providers.

Table 2.

Systems-contextual level: Training

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Baer et al. (2004)	22	MI	Adult subst. use	Addiction and mental health clinicians	2-day 14-hour workshop with brief didactics and experiential activities	Experiential activities (small group practice)	1. HRQ 2. coded MISC (OQ, CR, R:Q, MI-C behavior, MI-Inc behavior)	Convenience sample; pre–post assessment; no control group; no RA; 2-month f / u; SM	1. Increase in HRQ + R:Q, OQ, MI-C at pre–post 2. Increase in HRQ + R:Q maintained at f / u
Beidas et al. (2009)	20	CBT	Child anxiety	PsyD trainees	2-hour didactic workshop plus manual	None	1. Rated A 2. Rated S 3. K	Convenience sample; pre–post assessment; no control group; no RA; no f / u; ICM	1. Increase in K pre–post assessment 2. No increase in A, or S
Cross et al. (2007)	76	QPR	Youth suicide prevention	Nonclinical employees	1-hour workshop plus role-play	Experiential behavioral role- play	1. dec K 2. per K 3. self-efficacy 4. Sa 5. diffusion of K 6. rated S	Convenience sample; pre–post assessment; no control group; no RA; 6-week f / u; ICM	1. dec K, per K, increased pre– post. 2. high Sa at postassessment. 3. High diffusion at postassessment 4. S ranged between 11.79 / 15 at postassessment and 11.43 / 15 at f / u (no sig. diff).
Gega, Norman & Marks (2007)	92	BT	Prim. care	Nursing students	4-hour computer training or 4-hour workshop	None	1. K 2. rated S 3. Sa	Randomized controlled trial with crossover; no f / u; ICM	1. Both conditions improved K, rated S, and Sa equally when provided alone. No added benefit to doing both trainings.
Hawkins and Sinha (1998)	109	DBT	BPD	Community clinicians	Not described	Not described	1. K	Convenience sample; pre–post assessment; no control group; no RA; no f / u; ICM	1. K increased pre–post 2. K predictors: study group attendance, time applying tx, reading, peer consult
McVey et al. (2005)	3315	CBT / IPT	ED	Health-care practitioners with professional training	3-hour to 4-day intensive workshops and manual	Not described	1. per K 2. comfort in treating or teaching about ED	Convenience sample; pre–post assessment; no control group; no RA; no f / u; ICM	1. Increase in per K and comfort pre–post
National Crime Victims Research & Treatment Center (2007)	4,387	tf-CBT	Youth trauma	Mental health professionals	10-hour web-based seminar	Interactive computer program (e.g., streaming video)	1. K 2. Sa	Convenience sample; pre–post assessment; no control group; no RA; no f / u; ICM	1. High Sa at postassessment 2. Sig. increase in K for all 10 modules
Sanders, Tully, et al. (2003)	32	Triple P	Youth behavior problems	General physicians	Workshop	Role-play and feedback	1. S 2. Sa 3. Confidence	Convenience sample; pre–post assessment; waitlist comparison; SM	1. Higher S in trained group 2. Higher satisfaction 3. Higher confidence

Note. MI, motivational interviewing; HRQ, Helpful Responses Questionnaire; MISC, Motivational Interviewing Skills Code (OQ, open questions; CR, complex reflections; R:Q, ratio of reflections to questions; MI-C, motivational interviewing consistent; MI-Inc, motivational interviewing inconsistent); RA, random assignment; f / u, follow-up assessment; SM, standardized measures; CBT, cognitive-behavioral therapy; A, adherence; S, skill; K, knowledge; ICM, investigator-created measures; QPR, question, persuade, refer; dec K, declarative knowledge; per K, perceived knowledge; Sa, satisfaction; BT, behavior therapy; DBT, dialectical behavior therapy; BPD, borderline personality disorder; tx, treatment; IPT, interpersonal therapy; ED, eating disorders; tf-CBT, trauma-focused CBT.

Training Method and Content

One study exclusively included passive learning (e.g., didactic presentation; Gega et al., 2007), whereas four included both passive and active learning (e.g., experiential learning, interactive computer exercises, role-plays; Baer et al., 2004; Cross et al., 2007; National Crime Victims Research & Treatment Center, 2007; Sanders, Tully, et al., 2003). Two studies did not provide sufficient description of the training to identify training method (Hawkins & Sinha, 1998; McVey et al., 2005). When sufficiently described, training content included a focus on reviewing each session of the treatment protocol (Beidas et al., 2009; National Crime Victims Research & Treatment Center, 2007; Sanders, Tully, et al., 2003) or a focus on principles and / or the underlying spirit of the treatment (Baer et al., 2004; Gega et al., 2007). Training duration varied widely from one hour (Cross et al., 2007) to five days (Hawkins & Sinha, 1998). Three studies used manuals to supplement the didactic presentation (Beidas et al., 2009; McVey et al., 2005; Sanders, Tully, et al., 2003).

Outcome Measures

The outcome measures for the majority of the studies were self-reports, with the exception of independently rated adherence and skill. Self-reported outcomes included knowledge, satisfaction, level of comfort with providing treatment, self-efficacy, training experiences, and diffusion of knowledge. Two studies (Baer et al., 2004; Sanders, Tully, et al., 2003) used psychometrically sound standardized measures, whereas the other studies used investigator-created nonstandardized measures.

Summary of Studies

Studies using only self-report questionnaires (e.g., knowledge and satisfaction) were considered less rigorous than studies that included independently rated adherence and / or skill given the finding that knowledge change is often not generalized as actual behavior change (Miller, Sorensen, Selzer, & Brigham, 2006). Surveyed mental health professionals (n = 3,315) reported a statistically significant increase in perceived knowledge of ED, an increase in level of comfort in treating ED, and increased comfort teaching others about ED treatment after receiving differential doses of training (McVey et al., 2005). Training varied across participants, ranging from three hours to four days, and knowledge was not objectively measured.

Another study objectively measured declarative knowledge and concluded that community mental health providers (n = 109) were able to reach proficiency in DBT after receiving unspecified dosages of training (training varied across participants; Hawkins & Sinha, 1998). In the group that attended the highest dose of training (n = 36; a five-day intensive workshop), the highest percentage of correct answers on a knowledge test was 67.6. Although the authors conclude that participants learned the protocol, they did not achieve proficiency according to the 80% criterion. Another study objectively measuring knowledge found that a multimedia web-based distance learning program for mental health professionals seeking training in tf-CBT significantly increased participants’ knowledge (National Crime Victims Research & Treatment Center, 2007) from 61.9% to 82.1%, which suggests that participants reached proficiency in their knowledge of tf-CBT.

Five studies measured self-reported knowledge and independently rated therapist behavior. Following a one-hour training workshop for nonclinical employees (n = 76) in the QPR suicide prevention program (Cross et al., 2007), participants reported satisfaction with training, and significant increases in both perceived knowledge (from 36% to 62%) and declarative knowledge (from 69% to 85%). A subset of participants (n = 26) completed a behavioral role-play from which skill was coded. Fifty-five percent of participants reached a satisfactory skill level (a score of 12 of 15), whereas 45% did not.

Similar findings were observed after 20 trainees read a manual and attended a 2.5-hour CBT workshop for youth anxiety (Beidas et al., 2009). Participants showed significant increases in knowledge (all reached the 80% criterion). Following training, participants completed a behavioral role-play from which adherence and skill were coded. After training, none of the participants reached 80% adherence, whereas 67% reached proficient levels of skill in CBT for child anxiety. A similar study compared the effectiveness of differing training modalities (i.e., computer training versus a workshop) on BT for anxiety. Following training, nursing students (n = 92) showed comparable improvement in knowledge, skills, and satisfaction in both conditions. Note that although knowledge improved significantly after training, participants were below the proficiency level (average 71%). Additionally, skill was rated on average as 4.3 of 8 points (Gega et al., 2006).

In a different treatment modality (i.e., MI), addiction and mental health counselors (n = 22) received 24 hours of training. After training, 53% of clinicians were rated as proficient in their MI skills, and at two-month follow-up, 42% were rated as proficient (Baer et al., 2004). A study that trained general practitioners in the Triple P program for externalizing youth found that a brief training emphasizing active learning produced significant improvements in rated skill. Clinicians also reported increased satisfaction and confidence in treatment delivery (Sanders, Tully, et al., 2003).

Conclusions

Both perceived and declarative knowledge increase after receiving training in an EBP. Change in declarative knowledge is especially important given the implications it may have on treatment delivery. However, it remains unclear if knowledge gains indicate proficiency in treatment delivery (e.g., Hawkins & Sinha, 1998), and further empirical study is necessary.

Of those studies that included independently rated behavior, proficiency in therapist skill ranged from 54% to 67% at post-training and 42% at follow-up (Beidas et al., 2009; Cross et al., 2007; Gega et al., 2006, Sanders, Tully, et al., 2003). After training, none of the participants reached proficiency in adherence (see Beidas et al., 2009). This finding is alarming when considering that skill and adherence (Perpepletchikova & Kazdin, 2005) are crucial to the transportation of a treatment from research to practice.

Limitations

Study-specific limitations include high attrition rates (Beidas et al., 2009; National Crime Victims Research & Treatment Center, 2007), practice effects, low content validity (e.g., only four questions per section to assess knowledge; National Crime Victims Research & Treatment Center, 2007), participants receiving differential training collapsed into one larger sample (Hawkins & Sinha, 1998; McVey et al., 2005), the examination of skill in only a subset of the sample (Cross et al., 2007), and an inability to procure all outcome variables (Baer et al., 2004).

Training and Organizational Support

Description of Studies

Six studies focused on both training and organizational support (see Table 3). The EBP training included (a) CBT for adult substance use (Sholomskas et al., 2005), (b) CT, dynamic therapy (DP), and drug counseling (DC) for adult substance abuse (Crits-Christoph et al., 1998), (c) group drug counseling (GDC) for adult substance abuse (Luoma et al., 2007), (d) Triple P for youth behavior problems (Sanders, Murphy-Brennan, & McAuliffe, 2003), (e) time-limited dynamic psychotherapy (TLDP) for adults (Henry, Strupp, Butler, Schacht, & Binder, 1993), and (f) MI for substance abuse (Moyers et al., 2008). Three studies used a nonrandomized, convenience-sample, pre–post, quasi-experimental design (Crits-Christoph et al., 1998; Henry, Strupp, et al., 1993; Sanders, Murphy-Brennan, et al., 2003). Two studies randomly assigned participants to a training condition and included a comparison condition (Luoma et al., 2007; Moyers et al., 2008), whereas another study attempted random assignment but was unable to do so (Sholomskas et al., 2005). Study samples ranged from 16 to 331 therapist participants, with educational levels ranging from bachelor’s to postgraduate degrees (e.g., MD, PhD, and PsyD). Most were community mental health providers.

Table 3.

Systems-contextual level: Training and organizational support

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Crits-Christoph et al. (1998)	65	CT, DP, DC	Adult subst. use	Therapists and counselors	Manual + 4 two- day workshops (didactic, role- play discussion) + supervision	Role-plays	1. S	Convenience sample; pre–post assessment; SM; no control group; no f / u; no RA	1. CT showed per training case (across case). 2. DC and DP showed learning within cases but not across.
Henry, Strupp, et al. (1993)	16	TLDP	Adults Axis I and Axis II	Psychiatrists and psychologists	50 weekly 2-hour workshops (didactic), manual, supervision	None	1. S	Convenience sample; pre–post assessment; SM; no control group; no f / u; no RA	1. Some S increased and some decreased
Luoma et al. (2007)	30	GDC	Adult subst. use	Community clinicians	6-hour workshop (didactics, q / a, role- plays) + eight 1.5-hour consultations from an ACT and relapse prevention model	Role-plays	1. Sa 2. Self-reported adoption 3. Burnout	Pre–post assessment; 2- to 4-month f / u; RA; control group; ICM and SM	1. Sa high in both groups 2. Adoption and personal accomplishment higher in intervention
Moyers et al. (2008)	129	MI	Adult subst. use	Behavioral health providers	Self-directed training vs. workshop vs. workshop + training enrichment	Varied per training condition	1. MI competence	Pre–post assessment; 4-month f / u; RA; comparison group; SM	1. Improved competence in all groups 2. No difference between workshop and workshop + training enrichment 3. Decay in competence at f / u
Sanders, Murphy- Brennan, et al. (2003)	331	Triple P	At-risk youth	General therapists	Reading, 2-day workshop, 1-day accreditation, access to web support network	Active skills training	1. Perceived S 2. Sa	Convenience sample; pre–post assessment; ICM; no control group; no f / u; no RA	1. High Sa 2. Increased perceived S
Sholomskas et al. (2005)	78	CBT	Adult subst. use	Community clinicians	Manual only vs. manual + 20- hour web training vs. 3-day didactic workshop + three 1-hour supervision	Web-based role-plays and in vivo role- plays	1. Rated A 2. Rated S 3. K	Not fully randomized; pre–post group assessment; control group; no f / u (post-Sup); SM	1. Increased S and A workshop + supervision when compared with manual + web at post + f / u. 2. Increased S, and A in Manual + web Compared with Manual only. 3. No Difference in Knowledge

Note. CT, cognitive therapy; DP, dynamic psychotherapy; DC, drug counseling; S, skill; SM, standardized measure; RA, random assignment; f / u, follow-up; TLDP, time-limited dynamic psychotherapy; GDC, group drug counseling; q / a, questions and answers; ACT, Acceptance and Commitment Therapy; Sa, satisfaction; ICM, investigator-created measure; MI, motivational interviewing; CBT, cognitive-behavioral therapy; A, adherence; K, knowledge; post-Sup, postsupervision.

Training Method and Content

All training included passive learning (e.g., didactic presentation). Five studies included active learning strategies (e.g., experiential learning, in vivo and / or computer role-plays, active learning skills; Crits-Christoph et al., 1998; Luoma et al., 2007; Moyers et al., 2008; Sanders, Murphy-Brennan, et al., 2003; Sholomskas et al., 2005). Training content included a focus on reviewing each session of the treatment protocol (Crits-Christoph et al., 1998; Sholomskas et al., 2005) or a focus on principles (Henry, Strupp, et al., 1993; Moyers et al., 2008). Training time varied from 6 (Luoma et al., 2007) to 100 hours (Henry, Strupp, et al., 1993). Five studies required the use of manuals to supplement the didactic presentation (Crits-Christoph et al., 1998; Henry, Strupp, et al., 1993; Moyers et al., 2008; Sanders, Murphy-Brennan, et al., 2003; Sholomskas et al., 2005).

Organizational Support

One study provided peer and web-based support (Sanders, Murphy-Brennan, et al., 2003), whereas another used group consultation to overcome therapist barriers in the use of newly acquired skills (Luoma et al., 2007). Other work (Crits-Christoph et al., 1998; Henry, Strupp, et al., 1993; Sholomskas et al., 2005) included supervision as part of the training, ranging from 3 one-hour supervisions (Sholomskas et al., 2005) to 50 two-hour weekly supervisions (Henry, Strupp, et al., 1993). One study provided supervision that concentrated on feedback on specific techniques and also provided additional didactic training (Moyers et al., 2008).

Outcome Measures

Most studies used self-reported outcome measures, with the exception of independently rated adherence and skill. Self-reported outcomes included knowledge, satisfaction, therapist perception of alliance, adoption, and burnout. Five studies (Crits-Christoph et al., 1998; Henry, Strupp, et al., 1993; Moyers et al., 2008; Sanders, Murphy-Brennan, et al., 2003; Sholomskas et al., 2005) used standardized psychometrically sound measures, and Luoma et al. (2007) used a combination of investigator-created nonstandardized measures and standardized psychometrically sound measures.

Summary of Studies

Physicians (n = 331) were trained in Triple P (Sanders, Murphy-Brennan, et al., 2003) and had access to peer support networks and web-based support. After training, participants reported a statistically significant increase in how well they believed they were trained to manage behavior problems, increased confidence in conducting parent consultation, and high satisfaction with the training. In another study, therapists (n = 30) were randomly assigned to a psychologically focused group consultation after attending a one-day workshop to overcome psychological barriers in the use of new treatment techniques (Luoma et al., 2007). In relation to the comparison group, the intervention group reported more adoption of the treatment at two- and four-month follow-ups, and a greater sense of personal accomplishment at four-month follow-up.

Two interventions focused on the effects of EBP training on therapist behavior as part of larger treatment outcome studies (Crits-Christoph et al., 1998; Henry, Strupp, et al., 1993). Both studies used manuals, didactic workshops, and ongoing supervision as part of training, although one study provided significantly more supervision (50 two-hour weekly sessions; Henry, Strupp, et al., 1993). After receiving training and supervision in TLDP (n = 16), therapist adherence and general skills (e.g., greater use of open-ended questions) were more highly rated. Unexpectedly, certain therapist skills decreased (e.g., less optimistic, less supportive, and more authoritative; Henry, Strupp, et al., 1993). By contrast, Crits-Christoph et al. (1998) found that following training in both DP and DC, therapist skill increased as each case progressed (i.e., within case), while CT therapists performed more skillfully across cases (n = 65; Crits-Christoph et al., 1998). In other words, only CT therapists were able to transfer their learning from case to case. By the fourth training case, CT therapists were scoring an average of 46 out of a possible 48 points, suggesting great improvement and proficiency in comparison with their first training case.

The most methodologically rigorous investigations in this category both included random assignment to varying training methods. One study employed an investigation of which condition: (a) manual only, (b) manual + web-based training, and (c) manual + didactic training + supervision) was most effective in improving therapist knowledge, adherence, and skill in CBT for substance abuse (Sholomskas et al., 2005). The highest dosage of training (manual + didactic training + supervision) produced the highest levels of therapist adherence and skill, with the manual + web training evidencing intermediate scores and the manual alone having the lowest levels of adherence and skill after training and at follow-up. It is of note that in the highest dosage of training, only 54% of clinicians reached proficiency levels in adherence and skill (Sholomskas et al., 2005). Knowledge was not significantly different between the three groups. Another study randomly assigned therapists to one of three conditions: (a) workshop only, (b) workshop + training enrichments (i.e., six supervision calls, specific feedback on behavior), and (c) self-directed training (Moyers et al., 2008). All conditions showed gains in competence from baseline to post-training with effect sizes in the medium to large range; however, these gains declined at four-month follow-up. Contrary to expectations, improved competence was not observed in the group that received training enrichments (i.e., feedback and consultation calls).

Conclusions

After receiving training and follow-up organizational support, therapists’ perceptions of their behavior (e.g., confidence and personal accomplishment) increase (Luoma et al., 2007; Sanders, Murphy-Brennan, et al., 2003). In some studies, the perception of behavior change is matched by actual therapist behavior change; however, these changes in behavior are not maintained at follow-up. When considering specific EBP packages, some evidence supports the notion that MI competence can be gained after attending a one-day workshop. However, these gains were not maintained at follow-up, and a number of training enrichments did not result in expected increased skill.

Contradictory evidence exists regarding the effect of training in DP on therapist adherence. One study suggests that certain DP-related skills improve, while others deteriorate after training, particularly skills that influence the therapeutic relationship (Henry, Strupp, et al., 1993). A follow-up study suggests that DP skills influencing the alliance may decrease in the first training case following training, but that this effect may diminish as the therapist treats more clients (Crits-Christoph et al., 1998). Thus, further supervision and more than one training case may reverse an initial awkward phase as the therapist grows comfortable implementing the EBP.

Similarly, contradictory evidence exists regarding the effect of CT and CBT training on therapist behavior. One study demonstrates that CT training plus supervision allows skills to improve, suggesting that CT therapists are able to apply what they have learned in training and supervision to new cases. However, in another study of CBT training, approximately half of the therapists were not trained to an acceptable criterion in adherence and skill, despite receiving the gold standard in the field of training (Sholomskas et al., 2005).

Limitations

Study-specific issues include limited data on supervisory practices, self-reported skill (Luoma et al., 2007; Sanders, Murphy-Brennan, et al., 2003), biased skill ratings from supervisors (Crits-Christoph et al., 1998), comparisons across training interventions that differ in time, difficulty in randomization (Sholomskas et al., 2005), and difficulty in procuring samples of therapist sessions and follow-up assessment (Moyers et al., 2008).

Training and Therapist Variables

Description of Studies

Few studies focused on both training and therapist variables (see Table 4).³ The EBP training included (a) MI for adult substance use (Rubel, Sobell, & Miller, 2000; Saitz, Sullivan, & Samet, 2000), (b) behavioral family therapy (BFT) for families caring for a relative with schizophrenia (Brooker & Butterworth, 1993), and (c) an intervention training program for youth suicide prevention (Chagnon, Houle, Marcoux, & Renaud, 2007). Three studies used a nonrandomized, convenience-sample, pre–post, quasi-experimental design (Brooker & Butterworth, 1993; Rubel et al., 2000; Saitz et al., 2000); only one study randomly assigned participants to a training condition and included a comparison condition (Chagnon et al., 2007). Sample sizes ranged from 8 to 87 therapist participants, with educational levels ranging from bachelor’s to postgraduate degrees (e.g., MD, PhD, and PsyD). Participants included psychiatric nurses, mental health clinicians, community members, and health-care workers.

Table 4.

Systems-contextual level: Training and therapist variables

References	n	EST	Patient	Therapist	Training	Active Elements	Outcome Variables	Design	Results
Brooker and Butterworth (1993)	8	BFT	Schizo. patients and their families	Community psychiatric nurses	1. 2-day intro. 2. 1-week skills training using role-play and video-feedback	1. Role-play 2. Videotaped feedback concerning progress in skills.	1. Rated S 2. A	Convenience sample; nonrandomized; 6- and 12-month f / u; no control group; SM	1. Competent— very good S 2. 6-month f / u skill improved 3. Strengthen A held prior to training 4. Changed A to BFT as best tx 5. Increase in hours spent with family
Chagnon et al. (2007)	71	SAM	At-risk youth	Community members	3-day workshop	Role-play	1. A 2. K 3. Rated S	Randomized; comparison group; 6-month f / u; ICM and SM	1. At post-training, improvement in A, K, and S 2. At f / u, A maintains but K and S decrease
Rubel et al. (2000)	44	MI	Adult subst. abuse	Mental health therapists	2-day workshop (didactic & experiential) & recommended manual	1. Role-play 2. Fishbowl exercises	1. K 2. Rated S 3. A*	Convenience sample; nonrandomized; no f / u; no control group; ICM and SM	1. K and S improved from pre- to post- training
Saitz et al. (2000)	70	MI	Adult subst. abuse	Health-care providers	200-min workshop (50 min didactic, 120 min role- play, 30 min discussion)	Role-plays with feedback	1. Self-reported change in clinical practice 2. K** 3. A** 4. C** 5. S**	Convenience sample; nonrandomized; no f / u; no control group; ICM	1. Post-training overall A were higher 2. Self-report large impact on clinical practice 3. No change in K, S, or C

Note. BFT, behavioral family therapy; schizo. schizophrenia; S, skill; A, attitudes; f / u, follow-up; SM, standardized measures; tx, treatment; SAM, Suicide Action Montreal; K, knowledge; ICM, investigator-created measures; MI, motivational interviewing;

A*, measured at pre, but does not appear to have been readministered at post; C, confidence;

^**only collected pre–post for a sample of the cohort.

Training Method and Content

All of the studies included both passive learning (e.g., didactic presentation) and active learning strategies (e.g., role-plays with feedback, experiential learning, and fishbowl exercises). When sufficiently described, training content included a focus on principles (Rubel et al., 2000). Training duration varied from 200 min (Chagnon et al., 2007) to seven days (Brooker & Butterworth, 1993). One study recommended a manual to supplement the didactic presentation (Rubel et al., 2000).

Therapist Variables

Studies measured therapist attitudes and theoretical orientation. Two studies surveyed substance abuse attitudes (Rubel et al., 2000; Saitz et al., 2000), one study surveyed attitudes about schizophrenia and its treatment (Brooker & Butterworth, 1993), and another measured attitudes towards intervening with suicidal individuals (Chagnon et al., 2007). Two studies assessed theoretical orientation (Brooker & Butterworth, 1993; Rubel et al., 2000).³

Outcome Measures

For the majority of the studies, self-reported outcome measures were used, with the exception of independently rated skill. Self-reported outcomes included knowledge, attitudes, and change in practice. Saitz et al. (2000) used investigator-created nonstandardized measures, Brooker and Butterworth (1993) used standardized measures, and two studies utilized both types of measures (Chagnon et al., 2007; Rubel et al., 2000).

Summary of Studies

A survey of practicing clinicians (n = 70) trained on MI indicated that training made an impact on their practice with substance abuse patients, particularly in regard to asking formal alcohol screening questions and providing substance abuse counseling (Saitz et al., 2000). Those participants who completed pre- and post-training measures showed slightly higher attitudes towards substance abuse after training, although knowledge, confidence in ability to take a substance history, and self-reported skillfulness did not change.

More methodologically rigorous studies included independent ratings of skill. Results from a 12-hour MI training workshop indicated that both knowledge and rated skill (i.e., percentage of motivational statements to overall statements) improved from pre- to post-training in mental health therapists (n = 44; Rubel et al., 2000). However, participants only scored 67% on the knowledge test—which does not indicate training to proficiency. Attitudes, measured pretraining, indicated a psychosocial conceptualization of substance abuse. A seven-day BFT workshop for community psychiatric nurses demonstrated that rated skills were at least satisfactory two months after training, with several skills improving further at six-month follow-up (Brooker & Butterworth, 1993). At six-month post-training, nurses were proficient in both core and adaptive skills. Attitudes also changed following training, such that belief in a psychosocial theory of schizophrenia and the usefulness of BFT programs increased.

The most methodologically rigorous study investigated the effects of an intervention training in community members who frequently interacted with youth at risk of suicidal behavior (n = 78). Strengths of this study included randomization and a comparison group. In comparison with control subjects, participants in the intervention group showed significant knowledge, attitude, and skill improvement. At follow-up, knowledge and skill improvement significantly diminished, although change in attitudes was maintained. Although participants improved in knowledge and skill at post-training, scores were below proficiency level (i.e., knowledge 69%, skill 67%).

Conclusions

Survey data suggest that training in an EBP has a lasting impact on clinical practice and self-reported skill (i.e., Saitz et al., 2000), but actual behavior change (e.g., rated skill) does not necessarily take place when studies provide training without taking into account the different facets of the SC model—providing training and measuring therapist variables are not sufficient for DI efforts. Additionally, there appears to be a lack of therapist proficiency reached after training. The reported results of three studies indicated that training successfully changed knowledge and independently rated skill to a proficient level, but in two of these studies, exploration of the reported means suggests that this was not the case.

Attitudes shifted following training, and were maintained at follow-up. Attitudes improved towards EBP in all studies that included a pre- to post-training assessment (Brooker & Butterworth, 1993; Chagnon et al., 2007; Saitz et al., 2000). Attitude change was also sustained even when other behaviors diminished at follow-up (Chagnon et al., 2007). It may be that current training efforts engender shifts in participant perceptions and attitudes, but are unable to bring about lasting skill change.

Limitations

Study-specific limitations include small sample sizes, highly motivated participants (Brooker & Butterworth, 1993), low participant response rate (Rubel et al., 2001), and the use of a nonvalidated videotape technique to assess for skill acquisition (Chagnon et al., 2007).

Training and Client Variables

Description of Studies

Two studies focused on training and client variables (see Table 5). The EBP training included (a) MI for adult substance use (Miller & Mount, 2001) and (b) QPR for youth suicide prevention (Wyman et al., 2008). One study used a nonrandomized, convenience-sample, pre–post, quasi-experimental design (Miller & Mount, 2001), whereas the other study randomly assigned participants to a training condition and included a comparison condition (Wyman et al., 2008). Study sample sizes ranged from 22 to 249 participants, with education levels at the bachelor or master’s level. Participants included probation officers, community corrections officers, and secondary school staff.

Table 5.

Systems-contextual level: Training and client variables

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Miller and Mount (2001)	22	MI	Adult subst. abuse	PO and CCO	1. 2-day workshop 2. Manual 3. 6 optional (90 min) f / u discussion*	Small group practice	1. Self-reported S and C 2. Rated S 3. Client R	Convenience sample; nonrandomized; 4-month f / u; no control group; SM	1. Increase in self-reported S and C post-training 2. Modest increases in rated S at post-training, which were somewhat maintained at 4-month f / u on client 3. No effect responses
Wyman et al. (2008)	249	QPR	Youth suicide prevention	Secondary school staff	1. 1.5-hour workshop 2. 30-min refresher course several months later	None	1. K 2. Self-reported A, B, and Co 3. Student survey	Randomized; comparison group; 1-year f / u; ICM	1. Increase in K, A, B 2. Small effect in staff asking students re: suicide-related behaviors

Note. MI, motivational interviewing; PO, probation officers; CCO, community corrections officers;

^*Not attended by participants; f / u, follow-up; S, skill; C, confidence; R, responses; SM, standardized measures; QPR, question, persuade, refer; K, knowledge; A, appraisals; B, Behavior; Co, Communication; ICM, investigator-created measures.

Training Method and Content

Both studies included passive learning (e.g., didactic presentation), whereas one included active learning (e.g., small group practice; Miller & Mount, 2001). Training content was not sufficiently described in either study. Training duration varied from 2 (Wyman et al., 2008) to 15 hours (Miller & Mount, 2001). Miller and Mount (2001) used a manual to supplement the didactic presentation.

Client Variables

In one study, client in-session statements were used as a proxy for treatment outcome (Miller & Mount, 2001), whereas in the other study, students were surveyed regarding school staff communication in relation to suicidal ideation (Wyman et al., 2008). Note that neither study utilized a measure of severity, risk factors, and / or resiliency of therapists’ client population as predictor variables for training efficacy (as recommended by Sanders & Turner, 2005); instead measures of client factors were included as outcome variables.

Outcome Measures

The outcome measures included self-report (Wyman et al., 2008) and independently rated behavior (Miller & Mount, 2001). Self-reported outcomes included knowledge, skill, appraisals, behavior, and staff-student communication. Independently rated behavior included rated MI skill. One study utilized investigator-created nonstandardized measures (Wyman et al., 2008), whereas the other study utilized both investigator-created nonstandardized and standardized psychometrically sound measures (Miller & Mount, 2001).

Summary of Studies

Wyman et al. (2008) measured the effect of QPR training on therapist knowledge and behavior. Secondary school staff (n = 249) received two hours of didactic training on suicide prevention. In comparison with nontrained staff, knowledge, self-reported appraisals, and behavior changes were reported at one-year follow-up. Student responses (i.e., client variables) suggested a small effect (d = 0.18) in being asked more frequently about suicide by staff members (one of the main skills of QPR). Although a statistically significant effect was present in terms of knowledge acquisition (d = 0.41), trained staff scored an average of 76% on the knowledge test, while nontrained staff scored an average of 72% (both below proficiency levels).

Miller and Mount (2001) investigated the effect of training in MI on community corrections and probation officers working with adult substance abusers (n = 22). Participants received 15 hours of didactic training, which included small group practice and a manual. Self-reports by therapists indicated large increases in MI skills, while observational measures reflected more modest skill changes after training that were somewhat retained at four-month follow-up. Clients did not show the response changes found to be predictive of better outcome with MI.

Conclusions

The findings suggest that self-reported perceptions of change are frequently not matched by actual behavior change. Neither study was able to engender significant change at the client level, suggesting that the main point of initiating these training studies (i.e., to improve client response to treatment) may not be occurring. Lack of therapist behavior change may help explain this finding. This is concerning given that therapist confidence was very high in regard to their own skill, suggesting that they may not feel that they need further training or consultation in the implementation of the EBP (Miller & Mount, 2001).

An important consideration for both of these studies is that client response was included as an outcome variable. However, client variables (e.g., severity and resiliency) were not considered as moderators of training effectiveness. Such data need to be co-varied in statistical analyses to depict the interactions between training and client variables. For example, a severe population (such as suicidal youth) may be more difficult to access; so, while training may have succeeded in therapist behavior change, client variables may make the treatment more difficult to implement.

Limitations

Study-specific limitations included low staff enrollment (Wyman et al., 2008) and a small sample size with participants choosing to not attend optional follow-up discussion sessions that may have contributed valuable organizational support (Miller & Mount, 2001). High therapist attrition and / or low participation in follow-up training in DI research must be studied systemically due to the frequency of this occurrence in research studies (e.g., Baer et al., 2004; Miller & Mount, 2001).

Training, Organizational Support, and Therapist Variables

Description of Studies

Several studies (n = 5) focused on training, organizational support, and therapist variables (see Table 6).³ The EBP training studied included (a) BFT for families (Fadden, 1997), (b) CBT in primary care (Maunder, Milne, & Cameron, 2008), (c) CBT for adult substance abuse (Morganstern, Morgan, McCrady, Keller, & Carroll, 2001), (d) contingency management (CM) for youth substance abuse (Henggeler, Chapman, et al., 2008), and (e) CT, DP, and DC for adult substance abuse (Siqueland et al., 2000). Three studies used a nonrandomized, convenience-sample, pre–post, quasi-experimental design with no comparison group (Henggeler, Chapman, et al., 2008; Maunder et al., 2008; Siqueland et al., 2000), whereas one study surveyed participants after training (Fadden, 1997). The most rigorous study included randomization and a comparison group (Morganstern et al., 2001). Sample sizes ranged from 25 to 432 participants. Educational level ranged from bachelor to postgraduate level (i.e., PhD or MD), and participants included community mental health therapists, nurses, and physicians.

Table 6.

Systems-contextual level: Training, organizational support, and therapist variables

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Fadden (1997)	86	BFT	Families with schizo. member	Nursing-level clinicians	1. Orientation 2. Manual 3. 3-day skills workshop 4. Ten 1.5-hour supervisions 5. Ongoing monthly supervision	Role-plays, videotaped demonstrations of sessions	1. Number of families seen 2. Barriers to use of tx 3. Impact of BFT on work	Convenience sample; nonrandomized; no f / u; no pre-; no control group; ICM	1. 70% reported using BFT 2. Mean families seen = 1.7 3. 66% reported difficult to use 4. Therapists in the comm. or where more staff members were trained in BFT saw more families 5. Little attitude change
Henggeler, Chapman, et al. (2008)	432	CM	Subst. abuse in adol.	Public sector clinicians	1. Manual 2. 1-day didactic and experiential workshop	Experiential role-plays	1. Adop 2. A 3. Org*	Convenience sample; nonrandomized; 6-month f / u; no control group; ICM + SM	1. 58% reported use 2. Pract. variables predicted adop 3. Pract. and org. variables predicted A
Maunder et al. (2008)	25	CBT	Adult and youth in PC	MDs	1. 3- to 4-hour workshop 2. Manual 3. Four 1.5-hour case discussions	Case discussion groups	1. K 2. Self-reported use 3. Att and Sa	Convenience sample; nonrandomized; no f / u; no control group; ICM + SM	1. High Sa 2. Increased K 3. Increased use
Morganstern et al. (2001)	29	CBT	Adult subst. abuse	Subst. abuse counselors	1. 100 hours training (35-hour didactic training and feedback, supervision, and manual)	Role-plays and feedback	1. Rated A 2. Rated S 3. Att and Sa	Randomized; comparison group; no f / u; ICM and SM	1. High Sa 2. High Att toward CBT 3. Proficient A & S
Siqueland et al. (2000)	62	CT, DP, DC	Adult subst. abuse	Subst. abuse counselors	1. Manual 2. Four 2-day workshops 3. Unspecified supervision	Role-plays	1. Rated A 2. Rated C	Convenience sample; nonrandomized; no f / u; no control group; ICM and SM	1. In CT general experience and specific experience in CT predicted C

Note. BFT, behavioral family therapy; schizo., schizophrenia; tx, treatment; f / u, follow-up; ICM, investigator-created measures; CM, contingency management; subst., substance; adol., adolescent; Adop, adoption; A, adherence; Org, organizational variables; SM, standardized measures; CBT, cognitive-behavioral therapy; PC, primary care; MDs, physicians; K, knowledge; Att, attitudes; Sa, satisfaction; S, skill; C, competence; CT, cognitive therapy; DP, dynamic therapy; DC, drug counseling.

^*not included as an outcome variable, but measured organizational characteristics as predictors.

Training Method and Content

All studies included both passive learning (e.g., didactic presentation) and active learning strategies (e.g., experiential role-plays: Fadden, 1997; Henggeler, Chapman, et al., 2008; Morganstern et al., 2001; Siqueland et al., 2000; case discussion groups: Maunder et al., 2008). Training content included going through session by session (Henggeler, Chapman, et al., 2008; Siqueland et al., 2000) and a focus on principles of the treatment (Maunder et al., 2008), with one study including both a focus on session-by-session administration and principles of the treatment (Morganstern et al., 2001). Training duration varied from one day (Henggeler, Chapman, et al., 2008) to 100 hours (Morganstern et al., 2001). All studies used a manual to supplement the didactic presentation.

Organizational Support

Most studies (Fadden, 1997; Maunder et al., 2008; Morganstern et al., 2001; Siqueland et al., 2000) included ongoing supervision as part of the training intervention. Supervision time ranged from four 1.5-hour case discussion groups (Maunder et al., 2008) to 65 hours of supervision (Morganstern et al., 2001). One study did not include supervision but included measures of organizational characteristics (i.e., organizational readiness for change; Henggeler, Chapman, et al., 2008).

Therapist Variables

Two studies surveyed participant attitudes towards the patient population before and after training (Fadden et al., 1997; Morganstern et al., 2001), and two studies assessed participant attitudes towards the training received and treatment modality (i.e., CBT; Maunder et al., 2008; Morganstern et al., 2001). One study measured pretraining experience (e.g., number of cases and supervision hours received) as a potential predictor of post-training competence (Siqueland et al., 2000), whereas another study measured pretraining demographics and experience (e.g., years of experience and caseload characteristics) and therapist attitudes towards EBP as a predictor of adherence, adoption, and implementation of an EBP (Henggeler, Chapman, et al., 2008).³

Outcome Measures

Outcome measures included self-report (Fadden, 1997; Henggeler, Chapman, et al., 2008; Maunder et al., 2008) or self-report and independently rated behavior (Morganstern et al., 2001; Siqueland et al., 2000). Self-reported outcomes included declarative knowledge, implementation, acceptability of treatment, use of treatment, barriers to use, attitudes towards patient population / treatment modality, and adoption. Independently rated behavior included adherence, skill, and competence. Four studies used both investigator-created nonstandardized and standardized psychometrically sound measures (which were often modified for the purposes of each study; Henggeler, Chapman, et al., 2008; Maunder et al., 2008; Morganstern et al., 2001; Siqueland et al., 2000), and one study used an investigator-created nonstandardized survey (Fadden, 1997).

Summary of Studies

Three studies utilized only self-report questionnaires to observe the impact of training on therapist knowledge, attitudes, and behavior. Therapist participants trained in BFT (n = 86) were surveyed after completing a 39-hour four-phase training 9 months to 3.5 years after completing the training (Fadden, 1997). No pretraining assessment was completed. Seventy percent of therapists reported they had used BFT in their work since training; however, the average number of families seen was 1.7. Forty percent of families were seen by 8% of the trained therapists, indicating that a small proportion of trainees was seeing a large percentage of the families. Therapist variables influencing the number of families receiving BFT included therapist location (community vs. inpatient unit) and the number of therapists trained in each service area. Little attitudinal change was observed, and did not predict the number of families seen. In a more recent study, primary care physicians (n = 25) trained in CBT (i.e., three- to four-hour workshop and four follow-up case discussions; Maunder et al., 2008) reported high satisfaction with the training, as well as an increase in the use of CBT techniques with patients. Declarative knowledge scores increased after training (Maunder et al., 2008) to proficiency levels.

Mental health therapists (n = 432) received a one-day workshop on CM with adolescent substance abusers (Henggeler, Chapman, et al., 2008). Although supervision was not provided, important organizational and therapist variables were identified. Fifty-eight percent of workshop attendees attempted to use the treatment post-training and therapist variables predicted adoption of the treatment: More educated and more experienced therapists who held favorable attitudes towards manualized therapy and lacked expertise in the treatment were more likely to implement the treatment. Self-reported adherence was also investigated. Therapist variables predicted self-reported adherence: Adherence was higher for younger therapists certified in addictions, with larger caseloads, higher numbers of youth on their caseloads, with and who held more positive views regarding treatment manuals. Organizational characteristics, specifically, organizational motivational readiness to change, and greater organizational training exposure and utilization were associated with increased adherence to the treatment modality.

More methodologically rigorous studies included independent ratings of skill / competence. Two studies investigated differing treatment modalities for substance abuse. One study conducted training (manual and four-day workshops) in three different treatment modalities—CT, DP, and DC (n = 62; Siqueland et al., 2000)—and included supervision focusing on feedback from taped sessions. General experience as a therapist and specific experience in treating patients in CT had a positive impact on change in competence post-training, whereas those having had more previous supervision showed less change in competence after training.

The most rigorous study included random assignment and a comparison condition, and investigated CBT training (~100 hours of didactics and supervision) on independently rated adherence and skill in substance abuse counselors (n = 29; Morganstern et al., 2001). Participants reported high levels of training satisfaction and also endorsed high ratings for the utility of CBT as a treatment. Attitudes towards the conceptualization of substance abuse changed in the intervention group. Most participants reached independently rated adherence levels of somewhat to extensively adherent and independently rated skill levels of poor to good.

Conclusions

One important conclusion is the lack of consensus on the role of therapist variables on competence and adherence in cognitive and / or behavioral approaches (i.e., BFT, CM, and CT). With regard to therapist competence post-training, it is unclear whether or not pretraining competence predicts post-training competence, and may differ by treatment modality (e.g., CT). Interestingly, multiple studies report that therapists with more previous supervision show less change in competence, perhaps due to more allegiance to their own conceptualizations. When considering therapist self-reported adherence to CM, several therapist and organizational predictors emerged, but only one study investigated this question, thus making it difficult to make any conclusions other than the importance of future replication and study.

Another conclusion emerges: When three levels of the SC model are addressed (i.e., training, organizational, and therapist levels), training in CBT may be more successful as evidenced by two studies (Maunder et al., 2008; Morganstern et al., 2001). After training in CBT, participant outcomes included high satisfaction, attitudinal change, utilization, increased knowledge, and rated proficiency in adherence and skill (Maunder et al., 2008; Morganstern et al., 2001).

Limitations

Study-specific limitations include lack of pretraining measurement of therapist behavior and inconsistency in survey administration (Fadden, 1997), low response rates (Maunder et al., 2008), and small sample sizes (Maunder et al., 2008; Morganstern et al., 2001; Siqueland et al., 2000). The only study that utilized a control condition did not compare the control group with the intervention group on rated skillfulness or adherence in delivery of CBT, thus reducing the internal validity of the study (Morganstern et al., 2001).

Training, Organizational Support, and Client Variables

Description of Studies

Four studies focused on training, organizational support, and client variables (see Table 7). The EBP training included (a) group CBT for adult substance abuse (Watkins, Osilla, Hepner, Sandres, & Thompson, 2008), (b) MI for substance abuse (Schoener, Madeja, Henderson, Ondersma, & Janisse, 2006), (c) TLDP for adults (Bein et al., 2000), and (d) MST for adolescent substance abuse (Henggeler, Sheidow, Cunningham, Donohue, & Ford, 2008). Three studies used a nonrandomized, convenience-sample, pre–post, quasi-experimental design (Bein et al., 2000; Schoener et al., 2006), one study did not include pretraining measures (included a comparison group for client outcomes; Watkins et al., 2008), and one study used a randomized design with a comparison group (Henggeler, Sheidow, et al., 2008). Sample sizes ranged from 5 to 30 participants. Educational level ranged from bachelor to postgraduate level (i.e., PhD or MD), and participants included community mental health therapists, psychologists, physicians, and substance abuse counselors.

Table 7.

Systems-contextual level: Training, organizational support, and client variables

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Bein et al. (2000)	16	TLDP	Adult psych.	MD / PhD	1. 50 weekly 2- hour seminars and supervision (focus on principles) 2. Manual 3. One training case	None	1. Client psych. 2. Therapist proficiency	Convenience sample; nonrandomized; 1-year f / u; pretraining comparison; SM	1. No differences between pre- and post-training client psych. at post- and 1-year f / u 2. Only 28% reached minimal level of proficiency
Henggeler, Sheidow, et al. (2008)	30	MST	Adol. subst. abuse	Master’s-level clinician	1. Manual 2. Workshop 3. Randomized to workshop only or intensive quality assurance (supervision and quarterly booster training)	Role-play with feedback and positive reinforcement	1. Youth- and caregiver- reported therapist A	Baseline, postworkshop then randomized; comparison group; 4-month f / u; SM	1. Significant adherence reported by youth at post and difference between adherence to MST and CBT at f / u 2. Significant difference between MST and CBT techniques reported by caregiver at post but not f / u 3. No change in use of MST youth or monitoring by caregiver report
Schoener et al. (2006)	10	MI	Adult subst. abuse and COD	Frontline clinicians	1. 2-day workshop 2. Eight supervisions	Supervised practice and feedback during workshop	1. Therapist S 2. Client change talk	Convenience sample; nonrandomized; pre- to postdesign; no f / u; no comparison group; SM	1. Therapist improved MI-consistent skills* 2. Therapist decreased MI-inconsistent skills 3. Observed client change talk
Watkins et al. (2008)	5	CBT	Adult subst. abuse	Subst. abuse addictions counselors	1. 2-day workshop 2. Weekly supervision 3. 1-day booster training	Not described	1. A 2. C 3. Client outcome	Convenience sample; nonrandomized; no pre-; no f / u; comparison group for tx; SM	1. 91% A 2. 4.3 / 6 C 3. Client depression improved

Note. TLDP, time-limited dynamic psychotherapy; psych., psychopathology; f / u, follow-up; SM, standardized measures; MST, multisystemic therapy; MI, motivational interviewing; subst., substance; COD, co-occurring disorders; S, skill; CBT, cognitive-behavioral therapy; A, adherence; C, competence; tx, treatment.

^*Still below proficiency levels.

Training Method and Content

All studies included passive learning (e.g., didactic presentation), whereas two included active learning strategies (e.g., practice with feedback; Henggeler, Sheidow, et al., 2008; Schoener et al., 2006). One study did not include any active training strategies (Bein et al., 2000), whereas another did not provide enough of a description to characterize method (Watkins et al., 2008). Training content included a focus on principles of the treatment (Bein et al., 2000), a focus on principles and skills acquisition (Schoener et al., 2006), or a focus on session-by-session instruction (Henggeler, Sheidow, et al., 2008). One study did not provide an adequate description of training content (Watkins et al., 2008). Training duration varied from a two-day workshop and eight supervisions (Schoener et al., 2006) to 100 hours of seminar and supervision (Bein et al., 2000). Two studies used a manual to supplement the didactic presentation (Bein et al., 2000; Henggeler, Sheidow, et al., 2008).

Organizational Support

All studies included ongoing supervision as part of the training intervention, ranging from eight supervision sessions (Schoener et al., 2006) to 100 hours of supervision (Bein et al., 2000). In one study, the level of organizational support was manipulated, where following a workshop, therapists received standard consultation or intensive quality assurance consisting of weekly consultation and quarterly booster training (Henggeler, Sheidow, et al., 2008).

Client Variables

Studies measured client in-session statements or treatment outcome to indicate whether or not therapist training effected client behavior change. In one study, client change talk was used as a proxy for treatment outcome (Schoener et al., 2006). One study included client ratings of therapist adherence as the primary outcome measure (Henggeler, Sheidow, et al., 2008).

Outcome Measures

The outcome measures for two studies included self-report and independently rated behavior (Bein et al., 2000; Watkins et al., 2008), whereas two studies included only independently rated behavior as the outcome measure (Henggeler, Sheidow, et al., 2008; Schoener et al., 2006). Self-reported outcomes included client psychopathology (Bein et al., 2000; Watkins et al., 2008). Independently rated behavior included adherence (Henggeler, Sheidow, et al., 2008), competence (Watkins et al., 2008), and skill (Bein et al., 2000; Schoener et al., 2006). All four studies used a standardized psychometrically sound measure (albeit adapted for each study; Bein et al., 2000; Henggeler, Sheidow, et al., 2008; Schoener et al., 2006; Watkins et al., 2008).

Summary of Studies

Clinicians (n = 30) were trained in MST through a manual and workshop and then were randomized to a workshop-only condition (access to materials and phone access to an MST expert as needed) or an intensive supervision condition (weekly supervision and consultation, booster training quarterly, improvement of skills incorporated into clinician development plans; Henggeler, Sheidow, et al., 2008). Supervisors were also evaluated to augment therapists’ use of the intervention. Significant differences between youth-reported clinician adherence to MST techniques emerged when comparing the two conditions after training. Both youth-reported adherence (75%) and caregiver-reported adherence (65%) were higher in the intensive supervision condition. However, the caregiver-reported adherence effect was not sustained at follow-up.

Schoener et al. (2006) investigated the effect of therapist behavior on client change talk following training (i.e., independently coded from actual therapy sessions). Community clinicians (n = 10) were trained in MI through a two-day workshop and eight supervision sessions for patients with co-occurring substance abuse and Axis I disorders. After training, independently rated behavior suggested that therapists showed improved MI skill (empathy, MI-spirit, and reflective listening) and decreased MI-inconsistent behavior (closed-ended questions and advising without permission). Additionally, change in therapist MI skill was accompanied by a change in client self-talk, a variable associated with subsequent client behavior change. However, therapists exhibited lower average MI proficiency in comparison with past MI training trials and did not reach the recommended proficiency level (5 / 7 Likert; Miller, 2000). Organizational and / or therapist variables influenced training response: Therapists who had been employed in the same agencies for long periods of time appeared less capable of benefitting from training for certain MI skills (i.e., open-ended questioning). Poststudy interviews provided contextual environmental information that may help explain these results: Many therapists reported high levels of anxiety and a number of systemic and structural challenges to the application of newly acquired MI skills.⁷

Two studies investigated therapist training effects on client treatment outcome in differing therapy modalities. Substance abuse counselors (n = 5) received two days of didactic training, ongoing weekly supervision, and a one-day booster training for group CBT for depression (Watkins et al., 2008). Counselors treated 61 clients and 35% of the sessions were coded for adherence and competence. Therapists demonstrated 91% adherence (two or three on a 0–3 scale) across coded sessions. Additionally, therapists delivered competent CBT with a mean competence rating of 4.3 (of 6). Client depression scores improved with treatment when compared with a comparison group. In another treatment modality, clinicians (n = 16) received 50 weekly two-hour seminars and supervision in TLDP for adult psychopathology. Training in TLDP did not improve patient outcome at post-training or at one-year follow-up. Follow-up analyses examined therapist proficiency at TLDP by independently rating two cases for each therapist. Only 9 of the 32 training cases (28%) were judged to have been conducted with a minimal level of skill (Bein et al., 2000).

Conclusions

If therapists reach proficient levels in adherence / competence, and have adequate consultation and organizational support, it is possible to elicit behavior change in client outcomes. In three of the four studies (Bein et al., 2000; Henggeler, Sheidow, et al., 2008; Schoener et al., 2006), therapists did not reach proficient levels in adherence and / or competence. This is concerning, particularly given that appropriate supervision and training seems to have been provided, particularly in the MST and TLDP studies. Given the striking amount of intervention, specifically targeting problem-solving barriers to MST implementation, these results are surprising. Similarly, in the study of TLDP, an adequate dosage of supervision was provided (one year of supervision—approximately 100 hours). However, this study only included one training case, which may not have been sufficient for gaining mastery over the skills needed to administer TLDP.

The importance of examining contextual variables at the level of therapist and organizational support becomes clear. What was it about these therapists that made it difficult for them to reach proficiency levels in MI, MST, or TLDP? One possibility may be their lack of experience. In previous studies of MI, therapist participants had prior experience in the treatment modality, suggesting that training further consolidated this experience (e.g., Miller & Mount, 2001; Miller et al., 2004). Similarly, in the study that reported positive client outcomes, therapists endorsed past experience with the treatment and high motivation to learn (Watkins et al., 2008). Perhaps therapists who are naïve to the fundamental principles of a treatment may require further training and follow-up supervision. Other therapist variables may also be important, such as allegiance to current therapeutic approaches and commitment to learning a new treatment modality. In one study, follow-up interviews suggested that systemic and contextual issues made it difficult to implement an EBP, highlighting the importance of organizational variables.

Limitations

Study-specific limitations include the small sample size of all the studies and the manner in which client outcome was operationalized—as a change in either self-talk (Schoener et al., 2006) or self-report rather than structured interviews (Bein et al., 2000; Watkins et al., 2008). Additionally, independently rated adherence from a youth and caregiver perspective only, without the inclusion of an expert perspective, was a weakness (Henggeler, Sheidow, et al., 2008).

Training, Organizational Support, Therapist, and Client Variables

Description of Studies

Few studies focused on all levels of the SC model (see Table 8). The EBP training included (a) TLDP for adults (Henry, Schacht, Strupp, Butler, & Binder, 1993), (b) MI for addictions (Miller et al., 2004), and (c) a comparison of supportive mental health counseling, MI, and group CBT for substance abuse and comorbid depression (Hunter, Watkins, Wenzel, Gilmore, Sheehe, & Griffin, 2005). One study used a nonrandomized, convenience-sample, pre–post, quasi-experimental design (Henry, Schacht, et al., 1993), whereas another study included a comparison group (Hunter et al., 2005). The most rigorous study included randomization to one of five groups and a comparison group (Miller et al., 2004). Study sample sizes ranged from 13 to 140 participants. Educational level ranged from bachelor to postgraduate level (i.e., PhD or MD), and participants included community mental health therapists, licensed substance abuse counselors, and medical professionals.

Table 8.

Systems-contextual level: Training, organizational, client, and therapist variables

References	n	EST	Patient	Therapist	Training	Active elements	Outcome variables	Design	Results
Henry, Schacht, et al. (1993)	16	TLDP	Adult psych.	MD and PhD	1. Manual 2. 50 weekly seminars and supervision 3. Training case	None	1. Therapist A 2. Therapist S	Pre- to postdesign; convenience sample; no f / u; no comparison group	1. Therapist variables influenced A 2. Client variables influenced S
Hunter et al. (2005)	13	MI, group CBT	Adult subst. + COD	Mental heath counselors	1. 2.5-hour 12-week training 2. Biweekly 3. Subset received depression training	Interactive educational meetings (not specified)	1. K 2. A 3. Job Sa 4. Job M	Pre- to postdesign; convenience sample; 1-year f / u; comparison group	1. K increased* 2. A increased* 3. Job Sa + M
Miller et al. (2004)	140	MI	Adult subst. abuse	Licensed subst. abuse profess	1. WS 2. WSF 3. WSC 4. WSC + F 5. SGT	Varied across training (WSF, WSC, and WSC + F)	1. C 2. Client change talk	Randomized; comparison group; 4-, 8-, 12-month f / u; SM	1. All four workshops showed increase in C pre–post 2. WS group reversed gains at 4-month f / u 3. Reduction in MI-inconst. behavior

Note. TLDP, time-limited dynamic psychotherapy; psych., psychopathology; A, adherence; S, Skill; f / u, follow-up; SM, standardized measures; MI, motivational interviewing; CBT, cognitive-behavioral therapy;subst., substance; COD, co-occurring disorders; K, knowledge; Sa, satisfaction; M, morale; WS, workshop only; WSF, workshop plus feedback; WSC, workshop plus coaching; WSC + F, workshop plus coaching and feedback; SGT, self-guided training; C, competence.

^*No statistical analyses completed between comparison and intervention group because of 50% staff attrition and small sample size.

Training Method and Content

All studies included both passive learning (e.g., didactic presentation) and two included active learning strategies (e.g., interactive educational meetings, Hunter et al., 2005; and feedback and coaching, Miller et al., 2004). Training content included a focus on principles of the treatment for all three studies. Training duration varied from two days (Miller et al., 2004) to 100 hours (Henry, Schacht, et al., 1993). Two studies used a manual to supplement the didactic presentation (Henry, Schacht, et al., 1993; Miller et al., 2004).

Organizational Support

Two studies (Henry, Schacht, et al., 1993; Hunter et al., 2005) included ongoing supervision as part of the training intervention, ranging from biweekly supervision (Hunter et al., 2005) to 100 hours of supervision and a training case (Henry, Schacht, et al., 1993). One study included two training enhancement procedures (ongoing feedback and coaching), which were considered to be most in line with organizational support given their similarity to supervision (Miller et al., 2004).

Therapist Variables

Hunter et al. (2005) surveyed participant attitudes towards mental illness, job satisfaction, and job morale pre- and post-training (Hunter et al., 2005); Miller et al. (2004) measured pretraining characteristics, history of substance abuse, self-esteem, and interpersonal style as predictors of post-training competence and client change outcome; and Henry, Schacht, et al. (1993) measured pretraining demographics and experience (e.g., years of experience and prior competence) as mediators of therapists’ response to training.³

Client Variables

Client characteristics were examined as mediators of therapist training response in two studies (Henry, Schacht, et al., 1993; Hunter et al., 2005), whereas in Miller et al. (2004), client change talk was used as a proxy for treatment outcome.

Outcome Measures

The outcome measures included self-report only (Hunter et al., 2005) or self-report and independently rated behavior (Henry, Schacht, et al., 1993; Miller et al., 2004). Self-reported outcomes included knowledge, attitudes about mental health, job satisfaction, and job morale. Independently rated behavior included interviewing style, adherence, interpersonal processes (Henry, Schacht, et al., 1993), and MI competence (e.g., MI-consistent behaviors, reflection:question ratio; Miller et al., 2004). One study used an investigator-created nonstandardized survey measure (Hunter et al., 2005), whereas two studies utilized standardized psychometrically sound measures (Henry, Schacht, et al., 1993; Miller et al., 2004).

Summary of Studies

One study used only self-report questionnaires to assess the effect of training on therapist knowledge, attitudes, and behavior. Mental health counselors (n = 13) trained in supportive mental health counseling and MI for addictions received 12 weeks of training (2.5 hours each) and biweekly supervision (Hunter et al., 2005). Note that a subset of counselors received 20 hours of training in group CBT for depression. Client characteristics were collected. Although the study included a comparison group, only descriptive analyses were completed due to high staff attrition (50%) in both the intervention and comparison site and the resulting very small sample. Declarative knowledge about substance abuse and co-occurring disorders increased in the intervention site after training (M = 83%) and was sustained over time (M = 81%) in comparison with the control site (post-training M = 68%; follow-up M = 74%). Attitudes became more positive at the intervention site at post-training and follow-up. Client characteristics were reported but not analyzed.

More methodologically rigorous studies included the effect of therapist training on independent ratings of adherence and competence. TLDP training consisted of 50 weekly two-hour in-person seminar and supervision sessions including didactic presentations of principles, a manual, and treatment of a training case with supervision (n = 16; Henry, Schacht, et al., 1993). Individual differences were quantified amongst the two trainers—the better trainer was directive and specific regarding the learning task, focused on therapist rather than patient dynamics, and provided positive reinforcement regarding specific therapist action. Therapist variables influenced training response—therapists with more prior supervision were less adherent, while therapists with certain interpersonal styles showed greatest technical adherence. Client variables also influenced training response—therapists showed greater improvements in skill when working with patients traditionally deemed less suitable for short-term techniques.

The gold standard for studies conducted in this literature includes random assignment and a comparison condition (Miller et al., 2004). Therapists were randomized to one of five training conditions (n = 140): (a) two-day workshop only, (b) two-day workshop plus feedback on tapes, (c) two-day workshop plus six individual coaching sessions, (d) two-day workshop plus coaching and feedback, and (e) a control condition—self-guided learning (therapist manual and training videotapes). Therapist variables did not predict therapist training response. All four intervention conditions (i.e., a–d) produced increased competence at post-training. At four-month follow-up, the workshop-only group lost their gains and returned near the levels of the comparison group. Only the groups receiving feedback and/or coaching reached proficiency levels in global MI-spirit (five of a seven-point Likert scale) and MI-consistent responses (95%) at post-training and follow-up. Client response only improved in the most intensive training (workshop, coaching, and feedback). The biggest effect in training was not an increase in therapist MI-consistent responses but a reduction in MI-inconsistent responses.

Conclusions

Intervening at all four levels of the SC model may be the key to producing effective outcomes. Therapist variables are influenced by training when measured as outcome variables. For example, attitudes and job satisfaction can be improved following training (Hunter et al., 2005). However, when examining therapist variables as predictor variables or mediating variables, contradictory evidence exists. One study found that therapist variables did influence response to training (Henry, Schacht, et al., 1993), while the most rigorous study found no effect of therapist variables (Miller et al., 2004).

Support for client variables as predictors of training and outcome variables emerges. When examined as a predictor of training response, it appears that therapist training improved when the client population was more difficult and more educated. There may be a complex relationship with supervision present here—perhaps more difficult clients call for more supervision time. In terms of outcomes, clients improved when therapists received the most intensive training intervention (Miller et al., 2004).

An interesting and important finding implicates the importance of training and organizational factors. Training style produced differential outcomes in therapist behavior in one study, suggesting that certain methods of training are preferential. Further, training method emerged as an important variable when manipulated—the only training that produced proficiency in MI at long-term follow-up included active learning strategies (i.e., coaching and / or feedback). The only training to elicit actual client change was the one that included the highest dosage of active learning strategies—feedback and coaching. This suggests the importance of studying training method (e.g., active versus passive) in future studies.

An issue that emerges from these studies is the identification of the barriers and challenges faced in community research. Hunter et al. (2005) published a follow-up article documenting barriers to DI research, including recruitment falling behind schedule, staff attrition, a comparison site instituting new practices that duplicated the intervention site, and the increase in patients already receiving mental health treatment when presenting to the intervention site (Wenzel, Ebener, Hunter, Watkins, & Gilmore, 2005). Follow-up client outcome research was not published due to difficulties in recruitment, although the intervention site continues to use the treatment in the community (S. Hunter, personal communication). These barriers make it difficult to disseminate and implement EBP in the community.

Limitations

Study-specific limitations include very high staff attrition that precluded the use of statistical analyses and difficulty in recruitment for client treatment outcome investigations (Hunter et al., 2005). None of these studies measured client outcome using the preferred technique of semi-structured interviews.

GENERAL DISCUSSION

The most definitive conclusion (see Table 9) is that training influences therapist knowledge, attitudes, and perceived behavior. Therapist behavior change (e.g., adherence and skill) can occur following training, but a number of conditions must be met. First, it is necessary for training to address most, if not all, levels of the SC model (the therapist is nested within a context and organizational, therapist, and client variables all interact transactionally with training to influence skillful implementation). Additionally, training must utilize active learning strategies to influence therapist behavior change.

Table 9.

General conclusions

Across treatment modalities and therapists, perceived and declarative knowledge increase following training in EBP.

Across treatment modalities and therapists, attitudes improve after training in EBP and this is maintained at follow-up.

Across treatment modalities and therapists, perceived (self-reported) therapist behavior change does not match actual behavior change.

Generally speaking, therapists trained in the current format (i.e., workshop, manual, and brief supervision) do not reach proficiency in treatment adherence, competence, and skill.

There is insufficient information about how therapist variables, client characteristics, and organizational variables influence therapist behavior following training (adherence, competence, and skill).

Generally speaking, therapist training in EBP does not currently engender improved client outcomes.

The quality of training is important to engender client change: Active learning during training is integral to influence both therapist and client change.

Provisional evidence suggests that when addressing all levels of the systems-contextual model, therapists reach proficiency levels in adherence, competence, and skill, particularly in CBT, and in turn influence client change.

Note. EBP, evidence-based practice.

Knowledge acquisition follows training. Both perceived and declarative knowledge increase after receiving training in EBP, and this finding appears to hold across treatment modalities and therapists. Therapist attitudes also change and this change is maintained over time following training. Importantly, training engenders self-reported behavior change that does not always match actual behavior change—pointing to the importance of including independently rated behavior in all future assessments of EBP training. The mismatch in perception and actual behavior is highly problematic and has important implications because clinicians may believe they are implementing a treatment with fidelity and skill (Miller & Mount, 2001). Additionally, this has deleterious consequences for client treatment outcome given the finding that better outcomes are observed in clients when a treatment is applied with fidelity (Elliot & Mihalic, 2004).

A most surprising finding is the lack of proficiency in treatment adherence, competence, and skill reached by therapists trained in the current gold standard (i.e., workshop, manual, and clinical supervision; Sholomskas et al., 2005). Multiple studies across differing treatments found that training (workshop, manual, and supervision) was not enough to produce proficient change in therapist adherence, competence, and skill. This is concerning, considering the importance of these constructs in administering a treatment with fidelity (Perpepletchikova & Kazdin, 2005). In turn, therapists were not necessarily able to engender client change. If therapists lack proficiency in a treatment, perhaps linked to inadequate training, then it is unlikely that clients will experience the benefits associated with a particular EBP.

Although proficiency in therapist behavior was not often achieved, when studies intervened at the levels of training, organizational, and therapist variables, therapists did reach proficiency levels in adherence, competence, and skill, particularly in CBT. Similarly, when all levels of the SC model (i.e., Miller et al., 2004) were addressed and active learning was used, both therapist and client change occurred. As posited by the SC model, therapist, client, and organizational variables interplay and influence the effects of training. To achieve client change, it is necessary for therapists to reach proficiency levels in knowledge, adherence, competence, and skill. This perspective speaks to the importance of training therapists to criteria prior to implementing treatments.

What occurs within training that may produce differential outcomes? An intriguing finding highlights the quality (i.e., content and method) of training. Active learning appears to be most effective—the only study to evidence client change included two active learning processes (i.e., coaching and feedback).

Most aspects of the SC model were understudied. An SC perspective dictates consideration of the organizational forces and factors that create the climate for successful DI efforts. Research that measures ways to understand organizations (see Organizational Social Context; Glisson et al., 2008) will inform needed organizational change for DI efforts. Zazzali et al. (2008) interviewed administrators of state organizations where an EBP had been implemented and found that organizational characteristics (e.g., interest in EBP and available resources) and organizational structures influenced the adoption and implementation of EBP. Similarly, low workplace support predicted less utilization of the Triple P program following training in service providers (Sanders, Prinz, & Shapiro, 2009). Further examination of the organizational context in the implementation of EBP is needed.

The incomplete literature leaves an absence of conclusions regarding therapist variables. Consistent measurement is a needed first step. One measure of therapist attitudes, the Evidence Based Practice Attitude Scale (EBPAS; see Aarons, 2004), provides a beginning and may help to identify those therapist variables associated with adherence to and competence with EBP.

Client characteristics were rarely investigated, despite the consensus that they are probably important. The influence of client variables cannot be answered adequately at this time. Future research needs to include measures of client risk, resiliency, and symptom severity. Importantly, future research should occur in tandem with RCTs as such a step would permit examining whether variations in therapist training engender client change.

Future Directions

Future DI research will benefit from addressing organizational, therapist, and client variables to demonstrate how these important contextual factors interact with training. RCTs with adequate samples and standardized measures, regardless of the EBP being studied, are encouraged. Investigations are needed to assess whether training on one EBP generalizes to training on other EBPs, and studies are needed to evaluate whether uniform guidelines can be created regarding best practices for training across treatment modalities.

The following questions regarding best practices for training require research:

1. What elements (e.g., manual, workshop, and supervision) should be included in training?

2. What is the optimal duration of training?

3. What is the optimal focus of training (i.e., general principles of a treatment or session by session)?

4. What type of active learning methods should be included?

5. What role does “training to criteria” play? Should trainees be certified in a treatment when they reach prespecified levels of competency (Sanders, Murphy-Brennan, et al., 2003; Sanders, Tully, et al., 2003)?

6. How much supervision is needed to achieve therapist behavior change and competent administration of an EBP?

Based on this review, we make the following recommendations:

1. Future training efforts focus on active learning and behavioral rehearsal with less emphasis on didactics.

2. Follow the SC model. Influencing one variable (e.g., therapist training) within a system is unlikely to result in effective implementation without addressing contextual factors. It is unlikely that training and DI will succeed without the understanding that therapists function within a context and that multiple variables (i.e., organizational support, client factors, and therapist factors) affect this context.

3. To assess that actual behavior change is occurring, future investigations of training must include direct measures of therapist behavior (e.g., adherence and skill) in addition to assessments of knowledge and perceived behavior change.

4. Standardized measurement is necessary to make cross-study comparisons. For example, a psychometrically reliable (Moyers, Martin, Manuel, Hendrickson, & Miller, 2005) system that is treatment specific includes the Motivational Interviewing Treatment Integrity scale (MITI; Moyers, Martin, Catley, Harris, & Ahluwalia, 2003). Studies of training in MI could use this measure rather than investigator-created measures. Additionally, developing measures that can be used across treatment programs would be useful.

5. The competencies for each treatment program should be identified so that it is clear what types of competencies are necessary for each EBP (see Roth & Pilling, 2008). For instance, for CBT for child anxiety, active treatment components could be identified (e.g., exposure and cognitive restructuring), and therapists trained in this treatment should achieve competencies in these domains.

6. Identifying the barriers to training and utilization of treatment is important and may be addressed early in training to problem-solve perceived barriers (Seng et al., 2006).

7. Supervision is crucial for skillful treatment delivery, and understanding the role of supervision and its key features are valuable areas for future research.

Despite the importance of EBP, we know less than preferred regarding how to best train therapists in EBP. Training efforts in EBP are underway, yet we are not informed by how best to proceed. If those who undertake EBP are not adequately trained in EBP implementation, will a subsequent evaluation of the benefits of EBP be unwittingly and unfairly compromised? Evaluations of training methods and variables, within contextual variables, will be informative. Without such an undertaking, one’s best EBP intentions may go unfulfilled.