Sustained Implementation of Cognitive-Behavioral Therapy for Youth Anxiety and Depression: Long-term Effects of Structured Training and Consultation on Therapist Practice in the Field

Abstract

Identifying factors that promote sustained implementation of evidence-based treatments (EBTs) after therapists receive training is critical for professional psychology. To address the field's minimal knowledge in this area, we interviewed community-based therapists (N = 23) who had completed intensive training in cognitive behavioral therapy (CBT) for either anxiety or depression as part of a randomized effectiveness trial (Southam-Gerow et al., 2010; Weisz et al., 2009). Therapists were interviewed three to five years after completion of the initial trial, representing one of the longest-term follow-ups of therapist practices after training. Therapists viewed each protocol and their individual CBT strategies as effective and appropriate for the majority of their current anxiety and depression caseloads. However, therapists used parts of each protocol much more frequently than the protocol as a whole (i.e., 78.5% used parts of the Coping Cat, and 7.5% used the whole protocol; 58.6% used parts of the PASCET, and 20% used the whole protocol). Therapists reported using problem-solving the most and exposure exercises the least for current anxious cases; they used cognitive restructuring the most and homework the least for current depression cases. Interventions that were more difficult to implement in usual care settings were less likely to be sustained. Future efforts should evaluate the characteristics and structure of EBTs that are most acceptable to therapists and should investigate which kinds of ongoing learning supports will maintain therapist skills in and continued use of EBTs.

What happens to evidence-based practice after intensive training and consultation supports dwindle? Despite a substantial knowledge base supporting the efficacy of psychological interventions in the treatment of child and adolescent (herein referred to as “youth”) emotional and behavioral disorders (David-Ferdon & Kaslow, 2008; Silverman et al., 2008), disseminating evidence-based treatments (EBTs) in a way that leads to sustained implementation in community settings remains a vexing challenge. Little is known about how well EBT training holds up over time or what factors influence their sustainability in local practice. Typical protocols for training therapists in evidence-based treatments usually include intensive didactic or experiential workshops and close supervision. However, relatively few efforts have been made to follow up with therapists to determine how well the skills have been sustained over time. And yet, therapist attitudes, particularly those who have had direct experience with implementing novel interventions, are highly influential in determining therapist interest and willingness in maintaining skills as intended (Aarons, 2004; Borntrager, Chorpita, Higa, & Weisz, 2009; Nakamura, Higa-McMillan, Okamura, & Shimabukuro, 2011). Several studies have demonstrated the relevance of attitudes on therapist use of EBTs (e.g., Beidas & Kendall, 2010; Henggeler et al., 2008; Jensen-Doss, Hawley, Lopez, & Osterberg, 2009). For example, Henggeler et al. (2008) found that implementation of contingency management interventions following a one-day workshop was predicted by therapist positive attitudes toward behavior therapy and beliefs about the efficacy of the intervention.

Therapist training programs have come to reflect a wide range of activities from bibliographic training (e.g., self-instruction using treatment manuals) to expert-led didactics (e.g., continuing education seminars) to ongoing consultation and supervision from experts. The most common method of training therapists new treatment approaches (i.e., reading written materials and attending single-event or limited workshops) tends to have little effect in producing desired outcomes, such as, increased skill or competence in EBT strategies (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010). Adding ongoing supervision, feedback, or extended coaching tends to enhance retention of skill after initial training workshops (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010). In one randomized trial of training approaches, community therapists' ability to implement CBT was higher at three-month follow-up when training workshops were augmented by ongoing supervisory supports than when no such supports were provided (Sholomskas et al., 2005). In another investigation of multiple training methods of Motivational Interviewing skills, Miller, Yahne, Moyers, Martinez, and Pirritano (2004) found no skill improvements through self-study and only modest gains in skills after clinicians received a two-day workshop; and therapist skills returned to baseline levels after 4 months without further supports. However, the addition of periodic feedback and/or coaching improved retention of proficiency up to 12 months after initial training. Similarly, Beidas, Edmunds, Marcus, and Kendall (2012) found that the number of consultation hours after training significantly predicted higher therapist adherence and skill three-months after training. Such results suggest that ongoing supervisory supports are necessary after initial training to promote successful short-term skill attainment, a necessary precursor to long-term treatment implementation.

Little is known about the long-term effects of these implementation supports, because few implementation studies include follow-up assessments. In two reviews of EBT training methods (Beidas & Kendall, 2010; Herschell et al., 2010), the majority of studies only evaluated immediate post-training effects of the training program, including outcomes such as, therapist knowledge, skill, and attitudes. In Beidas and Kendall's review, only 40.6% of training evaluations conducted longer-term follow-ups, wherein most follow-ups (61.5%) were six months or shorter, and all were shorter than a year after training. In Herschell et al.'s review, using different study inclusion criteria, 32.7% of studies conducted longer-term follow-ups, of which, 61.1% were six months or shorter. In one of the few longer-term follow-up studies, Saitz, Sullivan, and Samet (2000) completed a naturalistic follow-up of trainees who had attended a half-day continuing education course on substance abuse screening and brief intervention. Trainees were phone interviewed up to five years after attending the workshop. Most (96%) reported the workshop made a difference in their clinical practice, and the large majority of professionals (86% - 96%) reported they were more likely to use formal screening and counseling procedures taught in the workshop. The study lacked a control group and participants self-selected themselves to attend the workshop. Nevertheless, longer-term follow-up studies like this provide an indication that relatively brief trainings can impact professional report of services.

The Beidas and Kendall (2009) and Herschell et al. (2010) reviews also identified few training studies of youth protocols with long-term follow-up. No studies conducted follow-up assessments beyond one year, and most were between two and six months. For example, Henggeler et al. (2008) conducted six months of naturalistic follow-ups after public sector practitioners attended a 1-day workshop. At six months, 58% of workshop attendees had attempted to use the targeted intervention (contingency management) in substance-abusing adolescents they were treating. In a controlled trial of a school-based suicide-risk identification training program, Wyman et al. (2008) demonstrated that a 1.5-hour training plus 30-minute refresher course could produce significant changes in increased knowledge and provider appraisals a year after training. However, the training program did not increase targeted behaviors (e.g., asking students about suicide; referral behaviors) compared to the control condition, and targeted behaviors only increased for the 14% of school staff who had already been communicating with students about suicide before the training.

In another comparison trial, Beidas et al. (2012) compared three types of brief training approaches (six-hour standard workshop, computer-based workshop, and an augmented, experiential workshop) plus three months of weekly phone consultation in training providers in CBT for anxious youth. The brief training programs had the most immediate impact on knowledge, but the majority of participants also achieved “training criterion” in adherence (61%) and CBT skills (85%) by the end of the three-month phone consultation period. Together, the child training literature provides evidence for both encouragement and concern. Wyman et al.'s (2008) findings suggest that brief exposure to training materials is insufficient to sustain targeted behaviors over the longer-term, and Beidas et al.'s findings suggest that continued consultation is essential for improving poor rates of sustained skills. At the same time, Henggeler et al. (2008) suggests that some therapist behaviors can be impacted by relatively small doses of training. Each study is limited by its relatively brief follow-up period. We have less information about how effects would be sustained beyond six to 12 months after the intensive training period ends.

The current study examines community therapists' use of two CBT protocols and their attitudes towards using EBTs in two clinical contexts: treating anxious and depressed youth as part of a community research study and treating youth in their current practice setting. Structured interviews were completed with clinicians three to five years after they had served as study therapists in a community-based randomized effectiveness trial for which they had received intensive workshop training and supervisory supports (Southam-Gerow et al., 2010; Weisz et al., 2009). Such training and consultation represented a higher intensity of training than has been offered in past investigations of therapist training. The goal was (a) to determine whether community therapists continued to implement the EBTs on their own after the intensive supports of the research study ended, and (b) to assess therapist attitudes toward the EBT (appropriateness, difficulty implementing, effectiveness) after having the opportunity to implement the protocols in both a clinical trial and in their current practice. It was expected that attitudes toward EBT and specific treatment protocols would remain positive, given past research that suggests positive attitudes can develop even after brief training. It was also expected that therapists would report using CBT techniques in their practice, particularly the specific interventions in which they received training. However, it was expected that some strategies would be easier to sustain than others. Use of didactic components, such as emotion education, problem solving, and cognitive restructuring, was expected to be sustained relatively equally over the follow-up period. However, we expected in vivo exposures to be more difficult to sustain, as this intervention may require sustained practice and external supports. In vivo exposures tend to require greater availability of resources (e.g., exposure confederates, a team of like-minded therapists) and flexibility in practice setting (e.g., ability to conduct sessions outside) that might challenge their implementation.

Methods

Participants and Procedures

The Youth Anxiety and Depression Study (YADS; Southam-Gerow et al., 2010; Weisz et al., 2009) was the first fully randomized effectiveness study examining CBT for youth anxiety and depression in clinically representative treatment settings. As part of the YADS study, community clinicians from outpatient public mental health clinics were trained to deliver one of two manual-based CBT treatments for anxiety or depression in children (Coping Cat; Kendall & Hedtke, 2006; Primary and Secondary Control Enhancement Therapy [PASCET]; Weisz, Thurber, Sweeney, Proffitt, & LeGagnoux, 1997). A double randomization procedure was used wherein therapists were randomly assigned to be trained in CBT or to conduct treatment-as-usual, and participating youth were randomly assigned to CBT or treatment-as-usual. Training and supervisory supports were consistent with the highest standards of the training literature (Beidas et al., 2010; 2012; Herschell et al., 2010). Therapists attended a 6-hour training workshop that included didactics and group role plays. When treating a YADS case, therapists received weekly supervision that included feedback on video or audiotaped sessions.

All therapists who had received training in CBT (Coping Cat or PASCET) as a study therapist on YADS were eligible and invited to participate via phone and/or email. The YADS trial was completed in 2004 and follow-up interviews were conducted from 2007 – 2009 (considerable time was required to locate therapists and to find acceptable times for interviews to be conducted). For therapists no longer at their original work sites, contact information was obtained from searches of state licensing boards and/or referral from co-workers. Therapists were provided with study information and then provided oral consent to participate. All procedures were approved by the Virginia Commonwealth University Institutional Review Board. Eleven of 18 (61.1%) therapists trained in the Coping Cat, and 12 of 26 (46.2%) therapists trained in PASCET agreed to complete current study procedures. The remaining therapists were not able to be contacted (e.g., out-of- date contact information; n=12) or did not respond to recruitment efforts (n=6). Web-based versions of the survey were also developed to accommodate therapist schedules. Table 1 summarizes participant characteristics.

Table 1. Participants Demographics and Professional Setting.

	Coping Cat Therapists (N = 11)	PASCET Therapists (N = 12)
Age: M (SD)	45.22 (13.85) years	37.64 (5.64) years
	Range: 31 – 76	Range: 32 – 51
Gender	9 (82%) female	8 (67%) female
Ethnicity	5 (45%) Caucasian	7 (58%) Caucasian
	3 (27%) Not reported	5 (42%) Latino/Latina
	1 (9%) Latino/Latina
	1 (9%) Asian
	1 (9%) Multi-Ethnic
Highest degree obtained	6 (55%) Psy.D.	6 (50%) Ph.D.
	3 (27%) MSW/LCSW	3 (25%) Psy.D.
	2 (18%) Master's in Psychology	2 (16%) MSW/LCSW
		1 (8%) Master's in Psychology
Years of Clinical Practice	13.09 (11.53) years	5.50 (5.05) years
	Range: 4 – 45	Range: 2 – 20
Current employment setting	4 (36%) Private Practice	4 (33%) Private Practice
	3 (27%) Community Mental Health	3 (25%) Community Mental Health
	2 (18%) School	2 (17%) University setting
	1 (9%) Hospital	2 (17%) Other
	1 (9%) Other	1 (8%) School
Method of data collection	6 (55%) via phone interview	6 (50%) via phone interview
	5 (45%) via online survey	6 (50%) via online survey

Measures

Therapist Follow-up Coping Cat Interview (CCI; available from authors)

The CCI is a structured interview designed to assess a clinician's perception of the Coping Cat's utility across multiple settings. It can be administered by phone, mail, or web-based survey. The CCI contains two sections. First, therapists are asked to rate the effectiveness of and difficulty in implementing the Coping Cat in the anxiety cases they saw as part of the YADS trial. Therapists are asked to consider each aspect for the Coping Cat protocol as a whole and then to consider each aspect for the Coping Cat's individual treatment components: Emotion Education, Relaxation, Cognitive Restructuring, Problem Solving, Self-Evaluation and Rewards, Role Play, In vivo Exposure, and Homework. Items are rated on a 10-point Likert-type scale from 0 “Not at all [effective/difficult]” to 4 “Moderately,” to 9 “Extremely [effective/difficult].” For example, therapists are asked, “Thinking about your time in the YADS study, how effective were the following Coping Cat components for your project cases? How effective was [Relaxation]?”

Second, therapists are asked to evaluate the Coping Cat for its use in cases seen in the therapist's current clinical setting. Therapists are asked to rate the Coping Cat's effectiveness, difficulty in implementing, and appropriateness for current or recent cases where anxiety was a clinical focus. Each item is again rated on a 0 to 9 scale and respondents rate both the overall treatment protocol and individual treatment components. They then estimate the percentage of their current anxiety cases in which they have actually used the Coping Cat. Finally, therapists estimate the percentage of cases they used the Coping Cat (in part or in whole) across a broad range of non-targeted clinical problems (ADHD, defiance, family relationship problems, other) to examine perceived generalizability of the protocol.

Therapist Follow-up PASCET Interview (PI; available from authors)

The PI parallels the CCI structurally and assesses the same domains of therapist perceptions across YADS and current depressed cases: effectiveness and difficulty in implementing the PASCET as a whole and for its individual components, appropriateness for depressed youth and others, and percentage of current cases for whom they have actually used the PASCET. Individual PACET components included, Problem Solving, Activity Selection, Relaxation, Presenting a Positive Self, Talent Development, Cognitive Restructuring, Social Support, Sequential Coping, and Homework. Items are rated on a 10-point Likert-type scale from 0 “Not at all [effective/difficult/appropriate]” to 4 “Moderately,” to 9 “Extremely [effective/difficult/appropriate].”

Results

Initial Coping Cat (CC) Adherence

Initial therapist adherence was strong for therapists trained in the CC protocol (Southam-Gerow et al., 2010) with integrity ratings indicating that 98.9% (SD = 3.77, range: 87.5-100.0%) of CC sessions contained the expected procedures.

Perceived effectiveness and difficulty in implementing CC strategies

Table 2 summarizes therapist attitudes toward the CC and its strategies. For cases seen in the YADS trial, all CBT techniques were seen as effective for anxiety cases. On a 0-9 scale, perceived effectiveness scores ranged from 5.3 to 7.1 (M = 6.5, SD = .56), placing the mean at a level greater than “moderately” effective (based on CCI anchors given to respondents). Homework was seen as the least effective and role plays were seen as the most effective. Mean effectiveness for the whole CC protocol was 6.7 (SD=1.9) for YADS cases. Perceived difficulty in implementing specific CBT techniques with YADS cases was low, with scores ranging from 3.2 to 4.5 (M = 3.7, SD = .46), where a score of 4 equated to “moderately” difficult. The least difficult technique to implement was rewards, and the most difficult was exposure. Overall, the CC protocol as a whole was not seen as difficult to implement in the context of the YADS trial (M=3.8, SD=1.4).

Table 2. Clinician Attitudes toward and Sustained Use of Coping Cat (CC) Interventions with Anxiety Cases (N = 11).

	Emotions	Relaxation	Cognitive Restructuring	Problem Solving	Rewards	Role Play	Exposure	Homework	Whole CC	Parts of CC
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
Attitudes toward YADS cases
Effective (0-9)	6.7 (1.42)	6.1 (1.30)	6.3 (1.56)	6.8 (1.54)	6.8 (1.99)	7.1 (2.12)	7.0 (1.67)	5.3 (1.79)	6.7 (1.95)
Difficult to Implement (0-9)	3.2 (1.94)	3.6 (1.80)	3.8 (1.83)	3.5 (1.44)	3.1 (1.87)	3.5 (2.07)	4.5 (1.51)	4.3 (2.49)	3.8 (1.40)
Attitudes toward current anxiety cases
Effective (0-9)	7.2 (1.48)	6.6 (1.69)	6.4 (1.58)	7.0 (1.63)	6.8 (2.04)	6.6 (1.58)*	4.4 (1.8)*	4.3 (2.56)	4.4 (3.6)	7.7 (1.1)
Difficult to implement (0-9)	2.7 (1.79)	3.3 (1.95)	3.8 (2.18)	3.1 (1.76)	2.9 (1.81)	3.0 (2.05)	5.0 (2.4)	4.4 (1.91)	4.8 (1.66)	1.7 (2.0)
Percentage of cases for whom this strategy is “appropriate”	84.5% (19.16)	74.5% (19.92)	66.5% (21.35)	87.3% (15.55)	75.9% (20.59)	63.2% (20.65)	45.9% (21.54)	52.5% (22.51)	44.5% (19.5)	89.5% (12.57)
Modal response (%) for “appropriate”	100%	50%	50%	90%	90%	50%	50%	40%	30%	100%
Percentage of cases where therapist “actually used” this strategy	74.5% (28.13)	60% (34.24)	73.5% (13.44)	85% (15.09)	65.7% (23.56)	64% (20.11)	34.5% (22.91)	43.5% (26.88)	7.5% (16.2)	78.5% (27.1)
Modal response (%) for “used”	100%	50%	50%	85%	65%	70%	40%	30%	0%	100%

^*paired t-test, p < .05

Greater variation was evident in therapist attitudes regarding current casework with anxious youth. Most CBT techniques were still seen as effective (M = 6.1, SD = 1.3, range: 4.3 – 7.2). However, perceived effectiveness of exposures was 37% lower for current cases than for YADS cases (paired t-test = 3.24, p = .02), and 7% lower for role plays (paired t-test = 2.75, p = .02). Perceived effectiveness for other strategies were statistically nonsignificantly different (all paired t-tests, p > .05). The overall perceived effectiveness of the CC protocol was 34% lower for current cases compared to YADS cases but this difference was nonsignificant, paired t = 1.45, p = .22. Perceived difficulty implementing CC techniques with current anxious cases was relatively similar to YADS cases (M = 3.5, SD = 1.0, range: 2.7 – 5.0), and exposures were again perceived as the most difficult technique to implement. Nonsignificant differences were found in paired t-tests across strategies comparing YADS and current cases (all paired t-tests, p > .05). Overall, perceived difficulty in implementing the entire protocol was 26% greater for current cases but this difference was statistically nonsignficant (paired t-test = -1.62, p = .14).

Perceived utility and use of CC strategies with current anxiety cases

Therapists were asked for what percentage of their current anxiety caseload each CC strategy was appropriate, and for what percentage of their current anxiety caseload the CC was actually used (see Table 2). A wide range of responses was evident with exposures seen as appropriate for the smallest percentage of current anxiety cases (M = 45.9%, SD = 21.5%) and problem solving seen as appropriate for the highest percentage (M = 87.3%, SD = 15.6%). The entire CC protocol was seen as appropriate for 44.5% (SD =19.5%) of anxiety cases, and parts of the CC protocol were seen as appropriate for 89.5% (SD = 12.6%).

Therapist use of CC strategies with current anxiety cases ranged from exposures (M = 34.5%, SD = 22.9%) to problem solving (M = 85.0%, SD = 15.1%). Therapists reported using the entire CC protocol with 7.5% of anxiety cases (SD = 16.2%) and parts of the CC protocol with 78.5% (SD = 27.1%). Figure 1 illustrates the distribution of practitioner use of the CC.

Figure 1.

Practitioner use of CBT for youth anxiety and depression three-to-five years after completing training and consultation as part of a randomized effectiveness trial. Data represent the proportion of current cases with whom therapists reported “actually using” none of, parts of, or the entirety of the Coping Cat or PASCET protocols.

Generalizability of CC protocol

Therapists were asked to what extent the CC or its parts were effective for, and used with, clinical problems that are not specifically targeted by the CC (Table 3). Therapists reported the CC was broadly effective for much of their non-anxious caseloads (M = 5.7, SD = .52, range: 5.2 – 6.5). Therapists reported using at least parts of CC in 64.5% of cases with depression, 44.5% of cases with ADHD, 37.5% of cases with defiance or behavioral problems, 47.5% of cases with family relationship problems, and 27.1% of cases with “other” problems.

Table 3. Applicability of the Coping Cat (CC) and PASCET to non-targeted clinical problems.

	Whole or Parts of CC	Whole or Parts of PASCET
	M (SD)	M (SD)
Depression
Effective for this problem? (0-9)	6.5 (1.51)
Percentage of Cases Protocol was used for Depression	64.5% (24.99)
Modal Response (% of cases used)	50%
Anxiety
Effective for this problem? (0-9)		5.8 (2.91)
Percentage of Cases Protocol was used for Anxiety		37.3% (41.74)
Modal Response (% of cases used)		0%
ADHD
Effective for this problem? (0-9)	5.2 (2.25)	5.5 (3.79)
Percentage of Cases Protocol was used for ADHD	44.5% (30.95)	23.0% (35.29)
Modal Response (% of cases used)	20%	0%
Defiance/Behavioral Problems
Effective for this problem? (0-9)	5.3 (1.73)	5.4 (2.94)
Percentage of Cases Protocol was used for Defiance	37.5% (27.61)	27.0% (32.68)
Modal Response (% of cases used)	20%	0%
Family Relationship Problems
Effective for this problem? (0-9)	5.9 (1.29)	4.3 (2.71)
Percentage of Cases Protocol was used for Family Relationship Problems	47.5% (26.6)	27.7% (31.5)
Modal Response (% of cases used)	40%	0%
Other
Effective for this problem? (0-9)	5.6 (1.52)	6.2 (3.63)
Percentage of Cases Protocol was used for other problems	27.1% (36.84)	22.73% (39.07)
Modal Response (% of cases used)	0%	0%

Initial Adherence of PASCET

Initial therapist adherence was strong for therapists trained in the PASCET protocol (Weisz et al., 2009) with integrity ratings indicating that 98.0% (SD = 0.01, range: 88.0 - 100.0%) of PASCET sessions contained a required element.

Perceived effectiveness and difficulty in implementing PASCET strategies

Table 4 summarizes therapist attitudes toward the PASCET and its strategies. For depression cases seen in the YADS trial, all CBT techniques were seen as effective. On a 0-9 scale, perceived effectiveness scores ranged from 5.3 to 7.0 (M = 6.3, SD = .60), where a score of six equated to a rating higher than “moderately” effective, based on PI anchors given to respondents. Relaxation and sequential coping (pre-planning multiple solutions to a problem and using continuous assessment to determine which one works) were seen as the least effective, and problem solving was seen as the most effective. Mean effectiveness for the whole PASCET protocol was 6.4 (SD=2.4) for YADS cases. Perceived difficulty in implementing specific CBT techniques with YADS cases was low, with scores ranging from 2.9 to 4.5 (M = 3.5, SD = .67), where a score of four equated to “moderately” difficult. The least difficult techniques to implement were problem solving and cognitive restructuring (identifying and challenging negative thoughts), and the most difficult was assigning homework. The PASCET protocol as a whole was not seen as difficult to implement in the context of the YADS trial (M=2.2, SD=1.9).

Table 4. Clinician Attitudes toward and Sustained Use of PASCET Interventions with Depression Cases (N = 12).

	Problem Solving	Activities	Relaxatio n	Self- presentat ion	Talents	Cognitive Restruct uring	Social support	Sequenti al coping	Homewo rk	Whole PASCET	Parts of PASCET
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
Attitudes toward YADS cases
Effective (0-9)	7.0 (1.76)	6.5 (1.51)	5.3 (2.57)	6.4 (1.78)	6.5 (1.62)	6.9 (1.85)	6.7 (1.25)	5.3 (0.82)	5.9 (1.81)	6.4 (2.35)
Difficult to Implement (0-9)	2.9 (2.75)	3.3 (2.70)	3.5 (2.68)	3.6 (2.19)	4.0 (2.22)	2.9 (2.60)	3.6 (1.96)	4.2 (1.81)	4.45 (2.46)	2.2 (1.95)
Attitudes toward current depressed cases
Effective (0-9)	7.1 (1.1)	6.3 (2.36)	7.0 (1.41)*	5.6 (2.46)	5.8 (1.93)	6.9 (1.76)	6.7 (1.0)	6.1 (1.35)	5.3 (2.33)	7.7 (1.15)	6.8 (1.75)
Difficult to implement (0-9)	2.2 (2.21)	1.8 (1.7)	2.3 (2.15)*	2.5 (1.78)*	2.7 (2.23)	2.6 (2.02)	2.0 (1.61)*	2.7 (1.35)*	3.6 (2.37)	4.9 (3.06)*	1.4 (1.31)
Percentage of cases for whom this strategy is “appropriate”	87.9% (16.71)	89.6% (15.24)	82.8% (17.6)	77.9% (15.29)	78.3% (21.36)	90.4% (12.14)	79.5% (23.82)	78.0% (13.78)	58.2% (25.42)	44.6% (35.26)	85.4% (19.94)
Modal response (%) for “appropriate”	100%	100%	100%	75%	100%	100%	100%	70%	25%	0%	100%
Percentage of cases where therapist “actually used” this strategy	55.9% (38.26)	56.82% (40.45)	55.9% (38.46)	52.7% (38.3)	50.9% (39.36)	70.5% (38.18)	48.0% (39.1)	38.5% (36.97)	23.9% (33.89)	20% (39.08)	58.6% (40.63)
Modal response (%) for “used”	100%	100%	75%	100%	100%	100%	0%	0%	0%	0%	100%

^*paired t-test, p < .05

Similar attitudes were reported for current depression cases. Most techniques were viewed as effective for current depression cases (M = 6.5, SD = .73, range: 5.3 – 7.1), and perceived effectiveness ratings of relaxation were 32% greater for current compared to YADS cases (paired t = -2.35, p = .05). All other paired t-tests were statistically nonsignificantly different, p > .05. Perceived effectiveness of the whole PASCET was 20.3% greater for current depression cases, but this difference was nonsignificant, t = -2.00, p = .18. Perceived difficulty implementing PASCET appeared to be lower across techniques (M = 2.6, SD = .95, range: 1.8 – 3.6) when comparing current depression cases to YADS cases. Homework continued to be the most difficult strategy to implement and assigning activities the easiest. Implementation was rated as relatively easier in current depression cases, compared to YADS cases, in relaxation, t = 2.16, p = .05, self-presentation, t = 2.49, p = .03, social support, t = 4.00, p = .003, and sequential coping, t = 4.67, p = .001. However, perceived difficulty in implementing the entire PASCET protocol was 122.7% greater in current depression casework (paired t = -3.22, p = .01).

Perceived utility and use of PASCET strategies with current depression cases

Therapists were asked for what percentage of their current depression caseload that each PASCET strategy was appropriate, and in what percentage of their current depression caseload they used PASCET strategies (see Table 4). Homework was seen as appropriate for the smallest percentage of current depression cases (M = 58.2%, SD = 25.4%) and cognitive restructuring was seen as appropriate for the highest percentage (M = 90.4%, SD = 12.1%). The entire PASCET protocol was seen as appropriate for 44.6% (SD =35.3%) of depression cases, and parts of the PASCET protocol were seen as appropriate for 85.4% (SD=19.9%).

Therapist use of PASCET strategies with current depression cases ranged from assigning homework (M = 23.9%, SD = 33.9%) to cognitive restructuring (M = 70.5%, SD = 38.2%). Therapists reported using the entire PASCET protocol with 20% of depression cases (SD = 39.1%) and parts of the PASCET protocol with 58.6% (SD = 40.6%). Figure 1 illustrates the distribution of practitioner use of the PASCET.

Generalizability of the PASCET protocol

Therapists were asked to what extent PASCET or its parts were effective for, and used with, clinical problems that are not specifically targeted by PASCET (Table 3). Therapists reported that PASCET was moderately effective for much of their non-depression caseloads (M = 5.4, SD = .71, range: 4.3 – 6.2). Therapists reported using at least parts of PASCET in 37.3% of cases with anxiety, 23.0% of cases with ADHD, 27.0% of cases with defiance or behavioral problems, and 27.7% of cases with family relationship problems.

Discussion

As the field turns greater attention from evidence-based treatment development to evidence-based provider training, it is becoming increasingly important to incorporate feedback from all potential consumers. In this case, we endeavored to learn from a unique set of professionals – mental health providers who had the opportunity to implement a novel evidence-based protocol in two settings, a structured randomized trial and in a therapist's own naturalistic practice. Three to five years after completing intensive training in two CBT protocols that included weekly consultation, therapists continued to report relatively positive attitudes and continued use of both protocols across a range of clinical settings. However, in current casework, therapists tended to use parts of each protocol rather than implementing the entire program, and some specific strategies (e.g., exposure and homework) were less used than others.

Therapists reported that they used the majority of CBT techniques from the CC and PASCET protocols with their current anxiety and depression cases. Therapists considered most techniques generally effective and appropriate for the majority of their anxious and depressed clients. Therapists perceived the difficulty of implementing these practices in their current contexts as low. Therapists reported “actually using” at least parts of the CC in 78.5% of current anxiety cases and parts of the PASCET in 58.6% of current depression cases. More specifically, therapists reported using six of eight CC techniques in over 60% of their current anxiety cases. Therapists reported using six out of nine PASCET techniques in over 50% of their depression cases. Given the significant investment required to provide even initial training in novel therapy skills, it is heartening to see therapists appreciating and using EBT techniques after initial training. By comparison, Jensen-Doss et al. (2009) conducted a review of child mental health practice in Texas after provision of state-mandated evidence-based treatment training. There, 72% of child mental health providers reported using a mandated evidence-based protocol in 75% of their cases or more. Sustainability rates in the current study were somewhat lower, but the fact that YADS therapists reported using CBT techniques in 50-60% of their anxiety and depression cases on a volunteer basis is encouraging news.

Perceived generalizability of the two protocols provided another sign of promise. The CC is a “single target” treatment protocol, but therapists endorsed that at least parts of the CC were useful for a wide range of youth problems, including depression (appropriate for 64.5% of cases), ADHD (44.5%), defiance (37.5%) and family relationship problems (47.5%). The PASCET was also seen as broadly applicable to a diversity of problems, including anxiety (37.3% of cases), ADHD (23%), defiance (27%), and family relationship problems (27.7%). The perception of broad applicability of these protocols to multiple target problems bodes well for their long-term sustainability in community settings where case complexity may be seen as a barrier to EBT implementation. The ability to treat many problems with a single set of techniques and treatment aids (manual, workbook) simplifies the task of choosing and implementing interventions.

Important lessons were also learned from therapist feedback. Not all CBT techniques were viewed as equally positive. By a large margin, therapists preferred to use parts of the two protocols over the entire protocol. Therapists used parts of the CC in 78.5% of current anxiety cases compared to the 7.5% of cases in which they used the entire protocol. Parts of the PASCET were used in 58.6% of depression cases compared to 20.0% of cases for the entire PASCET. This is consistent with past research where clinicians expressed preference for modular-based formats and self-selecting specific interventions based on client need (Borntrager et al., 2009; Palinkas et al. 2008). Past research has shown that therapists do not harbor negative attitudes toward evidence-based practice so much as negative impressions of treatment manuals and their implied rigidity. Our findings confirm that therapists made independent decisions about treatment selection. Whether these decisions are based on pre-existing therapist attitudes, prior experiences with evidence-based treatments, or in the nature of the original training remains to be seen. All of these factors may play a role (Jensen-Doss et al., 2009; Nakamura et al., 2011).

Whether it is a positive or negative outcome that therapists used parts, rather than the entirety, of EBT manuals is the subject of ongoing debate. On one hand, one study found that modular-based treatments may perform better than standard protocols focused on single disorders or homogeneous clusters (Weisz et al. 2012). In this study, the modular condition even promoted the use of more evidence-based content than usual care treatment and more non-specific therapy content than standard sessions. This suggests that modular formats might strike the balance of structure and flexibility. On the other hand, research exists to suggest that permitting “too much” flexibility in clinical decision making can lead to “straying” from recommended EBT interventions (Craske et al., 2007). The current study remains neutral in this debate as the design permits purely descriptive interpretations. Our goal was to learn from therapists which practices were naturally seen as most effective and easy to implement and which might require greater supports should increasing their use be a goal.

Differing attitudes emerged regarding specific CBT strategies when comparing use in a clinical trial or current practice. Exposures and role plays in the CC were seen as less effective for therapists' current anxiety caseloads, and exposures continued to be rated as the most difficult intervention to implement. Indeed, therapists reported using exposures in only 34.5% of current anxiety cases while using all other CC strategies in over 60% of cases. Homework, perhaps surprisingly, was only used in 43.5% of current anxiety cases. None of the individual PASCET techniques were seen as less effective or more difficult to implement in current clinical practice. In fact, several strategies were seen as easier to implement in their current depression cases than during the YADS trial, including, relaxation, self-presentation skills, seeking social support, and training sequential coping (i.e., the use of contingency coping strategies). Relaxation was also perceived as more effective with current depression cases. However, perceived difficulty in implementing the entire PASCET protocol was greater in current depression cases compared to YADS cases. Further, most individual PASCET techniques were actually used in under 60% of current depression cases, with homework used in under 25% of depression cases. The one notable exception was cognitive restructuring which therapists reported using in 70% of their current depression cases.

Several themes emerge from these reports. The techniques therapists reported using most appear to be those that fit well within traditional clinical care settings. Cognitive restructuring was amongst the most frequently used interventions (73% of CC; 70% of PASCET). Therapists reported using emotion education, relaxation, problem-solving, and rewards assignment in at least 75% of anxiety cases. Each of these interventions fits easily into traditional one-on-one, office-based talk therapy and blends a combination of didactics, discussion, and mild behavioral rehearsal.

Action-oriented, behavioral interventions, such as exposure and homework were less frequently implemented, a worrisome finding, given the evidence-base that supports in vivo exposure and extra-therapy practice (homework) as potent components of treatment (David-Ferdon & Kaslow, 2008; Silverman et al., 2008). Exposure and other experiential interventions pose several challenges for therapists in traditional clinic settings. In vivo exposures can require substantial resources to maximize their benefit, including supplies, confederate actors, an ability to travel to naturalistic sites, and sufficient time for set-up, enactment, and post-processing. Such demands pose challenges to common clinic structures where session time may be restricted to 50-minute sessions and reimbursement schedules do not account for prolonged sessions or pay for supplies. Therapists working in relative isolation may find it difficult to find role playing confederates – either fellow professionals who will be prepared to handle youth conducting exposures throughout the clinic or same-aged peers to serve as mutual exposure participants. Clinic policies and liability concerns may limit a therapist to conducting therapy in the therapy room or clinic, limiting the ability to create naturalistic exposures. Each of these barriers, among others, need to be considered and addressed as we attempt to implement exposure exercises in local practice.

Homework was a second technique significantly under-utilized in current practice. It was considered less appropriate, less effective, and more difficult to implement with anxious and depressed youth than other strategies. Previous research suggests that clinicians tend to implement techniques that they believe are effective and enjoyable (Cook, Schnurr, Biyanova, & Coyne, 2009). Thus, therapist attitudes toward homework may reflect the challenges in getting youth to complete homework more than beliefs about the strategy, itself. This may be unfortunate given the apparent importance of homework in extending CBT lessons outside the therapy room (Kazantzis, Whittington, & Dattillio, 2010).

These results suggest that treatment approaches that do not fit smoothly into real-world conditions of clinical practice may be less likely to be sustained over time than interventions that do not clash with everyday practice conditions or constraints. This is consistent with the recent analysis by Weisz, Ugueto, Cheron, & Herren (2013) suggesting that everyday practice exists within a kind of “mental health ecosystem,” and that the match or mismatch between treatment characteristics and facets of the ecosystem will strongly influence whether the practices can be implemented and sustained. In cases where certain treatment components that are important to successful outcomes clash with the ecosystem, it may be critical to structure and deliver those components to fit more smoothly into the conditions of everyday clinical practice.

A second direction for future work involves enhancing ongoing support for implementation efforts. A number of models have been offered to extend training to local practice settings (Southam-Gerow, Rodriguez, Chorpita, & Daleiden, 2012). These include pyramid training (train-the-trainer, cascading), academic detailing (pairing novice and expert mentors), and pairing training efforts to therapist characteristics (prior training background, theoretical orientation, familiarity with EBTs). Of course, these are intensive (and expensive) training approaches. Costs may be reduced through the use of technology (Beidas, Koerner, Weingardt, & Kendall, 2011), such as, supplying printable worksheets online or making video-based demonstrations available at the time of implementation can help reduce barriers at the point of clinical contact. Expert consultation can also be made more cost-effective through technology. Fostering chat-rooms and wiki sites that allow experts to consult with a larger group of providers can maintain therapist interest and confidence. A newer movement entails the development of “peer learning communities” that leverage the interests and resources amongst local groups of practitioners to facilitate self- and collaborative-learning (Darling-Hammond & Richardson, 2009; Tosey, 1999). Experimentation with this model has primarily occurred in education and corporate settings and is only now being considered as a viable method for sustaining therapist learning in therapy practice (Chu, Carpenter, Wyszynski, Conklin, & Comer, 2015).

The current study has limitations. The sample of participating therapists was small and represented only a portion of the original group of YADS therapists. Nonparticipating therapists may have had different opinions or may have been less inclined to maintain EBTs. Therapists were trained in either CC or PASCET but not both, so comparisons across the two treatments were limited. In the original YADS study, therapists were randomly assigned to receive training in either PASCET, CC, or to a usual care condition, so we would expect that each group of therapists reflects typical and equivalent groups of professionals. Further, we did not maintain control over therapists after the YADS study ended; therapists could have self-studied or discussed opinions about CC or PASCET with other study therapists. This lack of control may have led to some contamination, but it does represent the natural course of learning following any form of training. The results are based solely on therapist report, some of which were retrospective in nature. Therapist perspective carries significant weight in this line of research, but future research would be bolstered by including objective data (e.g., chart review; fidelity ratings of video recorded sessions). Interviews were conducted over a long period of time (3-5 years after study termination), but the small sample prevented analysis to examine relation between therapist report and latency between original study and follow-up interview. It should be noted that therapists had participated in more intensive training and consultation than is typically available; effects may not generalize to lower dose efforts. We also encourage the study of therapist training and consultation programs across theoretical orientations. Such comparative work might illuminate common and specific training elements that support evidence-based practice.

The methodological strengths of the study are that community practitioners were participants rather than research employees or graduate student trainees. Therapists were employees of community mental health clinics who were seeing cases as part of their natural caseloads. Therapists then had the unique experience of implementing CBT protocols in the context of a randomized effectiveness trial (with ongoing supervisory supports) and then later in their current clinical context (which represented a range of clinical settings). Our longer term follow-up (3-5 years) is the longest we could identify in the youth training literature. This provides a particularly useful test of the holding power of EBTs over time in everyday practice.

The results of this study encourage both optimism and caution for those engaged in dissemination and implementation efforts. Community therapists can be trained to implement evidence-based protocols in the context of support and expert consultation. Therapists appear to retain some interest in and commitment to maintaining these practices even after the external supports have been removed, but therapists appear to be selective in their maintenance of evidence-based practices. Future research and training curriculum development might benefit from identifying how best to support ongoing learning and maintenance of skills under typical practice conditions.