Training and Consultation to Promote Implementation of an Empirically Supported Treatment: A Randomized Trial

Abstract

Objective

The study evaluated the efficacy of three training modalities and the impact of ongoing consultation after training. Cognitive-behavioral therapy (CBT) for anxiety among youths, an empirically supported treatment, was used as the exemplar. Participants were randomly assigned to one of three one-day workshops to examine the efficacy of training modality: routine training (training as usual), computer training (computerized version of training as usual), and augmented training (training that emphasized active learning). After training, all participants received three months of ongoing consultation that included case consultation, didactics, and problem solving.

Methods

Participants were 115 community therapists (mean age of 35.9 years; 90% were women). Outcome measures included the Adherence and Skill Checklist, used to rate a performance-based role-play; a knowledge test; and the Training Satisfaction Rating Scale.

Results

All three training modalities resulted in limited gains in therapist adherence, skill, and knowledge. There was no significant effect of modality on adherence, skill, or knowledge from pretraining to posttraining. Participants were more satisfied with augmented and routine training than with computer training. Most important, number of consultation hours after training significantly predicted higher therapist adherence and skill at the three-month follow-up.

Conclusions

The findings suggest that training alone did not result in therapist behavior change. The inclusion of ongoing consultation was critical to influencing therapist adherence and skill. Implications for implementation science and mental health services research are discussed.

Implementation science is an emerging field and is the “scientific study of methods to promote the systematic uptake of clinical research findings and other evidence-based practices into routine practice” (1). Implementation science addresses challenging questions concerning how to most effectively transport empirically supported treatments into community settings. Implementation of empirically supported treatments in the children’s mental health system has only recently gained prominence and holds promise (2). Barriers to implementation of clinical innovation in the children’s mental health system include poor attitudes toward innovation, difficulty embedding innovation into existing systems, and untrained providers (3). Perhaps the most daunting challenge to implementation of empirically supported treatments for youths in community settings is training clinicians to administer these complex treatments with fidelity (3). Thus one frequently used implementation strategy is training providers (2).

The most commonly used training strategy is a brief workshop (4). Workshops improve therapist knowledge and attitudes but are not effective in changing therapist behavior in providing patient care (5,6). This outcome is due to the one-time nature of training, or “training and hoping” (7), which does not allow for integration into existing practices. Two methods hypothesized to improve brief workshops are to construct workshops that are informed by the adult learning literature (8) and to follow workshops with ongoing consultation that allows providers to integrate clinical innovation into existing practices (6).

The creation of brief workshops informed by adult learning principles may result in improved therapist behavior outcomes (8). Most workshops are passively delivered despite research demonstrating that active learning, a process in which an individual acquires knowledge and skills through interaction and reflection, is preferable (9). Furthermore, it has been recommended that training focus on principles of a treatment rather than on one protocol because principles provide a foundational understanding and reduce the burden of learning multiple protocols (10). Finally, computer-assisted training holds promise (11,12) because it is well suited for limited-resource settings, such as community mental health clinics.

Improving the efficacy of brief workshops is desirable but may not be sufficient to change therapist behavior. Ongoing consultation after training may be the critical element in increasing effectiveness of training (6). For example, one study demonstrated that coaching and performance feedback on cases posttraining changed therapist behavior (13). The mechanism by which consultation works is unknown, but consultation likely provides therapists with a venue for clarification, and practice of concepts, learning concepts and practicing over time, case consultation, and using problem solving to overcome implementation barriers. The impact of adding consultation to brief training has not been examined as an implementation strategy in child mental health services and holds promise for effective provider behavior change.

This study evaluated the efficacy of three modalities of brief training and the impact of including ongoing consultation after this training. Cognitive-behavioral therapy (CBT) for treating anxiety among youths, an empirically supported treatment, was used as the exemplar. Participants were randomly assigned to a one-day workshop to examine routine training (training as usual), computer training (computerized version of training as usual), or augmented training (training that emphasized active learning and principles of CBT). After training, all participants received three months of ongoing consultation that included case consultation, didactics, and problem solving of implementation barriers specific to the local context. Outcomes included therapist adherence to CBT for child anxiety, skill in CBT for child anxiety, knowledge about CBT for child anxiety, and satisfaction with training. The study design was prospective and experimental.

Methods

Recruitment and screening

All procedures were approved by the Temple University Institutional Review Board. Participants were recruited from the community via professional Listservs, training programs, and word of mouth. After complete description of the study to participants, written informed consent was obtained. Recruitment began in May 2009 and was completed by September 2009. Assessments, training, and consultation began in June 2009 and were completed by January 2010.

Inclusion criteria

Criteria for inclusion in the study were working in the community with children ages eight to 17 with anxiety disorders, having training in mental health, reading and speaking English, having access to a computer or telephone for consultation, and participating voluntarily. Exclusion criteria were having eight or more hours of prior training in CBT for child anxiety (13).

Power analysis

Before recruitment, we conducted a power analysis, which indicated that a sample size of 38 participants in each group (N=114) would result in 80% power to detect a moderate effect (Cohen’s d=.6) at an alpha level of .05.

Randomization

We recruited 115 participants, who were randomly assigned to one of three training modalities. Using equal allocation concealment, random assignment occurred at the level of training date. An unpredictable randomization sequence was generated that determined random assignment of training dates to condition. Participants were not informed of which training condition they received.

Training methods

Study design

Participants were randomly assigned to one of the three training conditions, all of which were provided in person. After training, all participants received weekly consultation by telephone or over the Internet for three months.

Routine training

This condition, involving 41 participants, included a six-hour workshop that focused on session-by-session administration of the Coping Cat program (14). The workshop was designed to follow the format of continuing-education workshops and did not emphasize active learning. Didactic instruction included a PowerPoint presentation and viewing of videotapes of representative youths receiving treatment.

Computer training

This condition included a six-hour computer-based, self-guided training for 34 participants (15). It closely resembled routine training, with the exception that it was delivered via computer and was therefore self-paced. The training provided step-by-step instructions for each session of the Coping Cat program and examples and materials for the therapist to peruse (14). Active learning in this condition included videos of treatment sessions, therapist tips, and links to research articles. Although training was via computer, participants received training in a computer lab to ensure that they completed an amount of training equivalent to that provided in the other two modalities.

Augmented training

Forty participants attended a six-hour experiential workshop focusing on core CBT principles: coping modeling, identifying somatic cues and relaxation, identifying and challenging anxious self-talk, problem solving, and behavioral exposure (16). This training emphasized active learning through the use of behavioral role-play (17) and interteaching (18). Behavioral role-play is a technique where therapists role-play implementing a treatment and receive feedback on their performance (11), whereas in-terteaching is a behaviorally based instructional technique in which individuals engage in small-group discussion with guided questions (18). Core principles were introduced didactically, modeled by the instructor, and then followed by small-group activities.

Consultation

Participants from all three conditions were provided weekly consultation via the WebEx virtual conferencing platform for three months after training. Participants could call in via telephone or computer to attend the one-hour weekly virtual group meeting. Those who opted to consult via computer were able to view a whiteboard and the consultant via Web camera. The consultation curriculum was designed with participant input and included case consultation, didactic topics (such as treating a client with comorbid depression), practice with concepts (such as relaxation), and assistance in implementation of the treatment within context (in a psychiatric clinic or school, for example).

Measures

Clinician Demographics and Attitudes Questionnaire

This 15-item questionnaire assesses demographic characteristics and prior experience with CBT for youth anxiety (19).

Training Satisfaction Rating Scale

This 12-item questionnaire assesses training satisfaction on a Likert scale from 1 to 5 (20). The score was the total of the ratings on the 12 items. Possible scores range from 12 to 60, with higher scores indicating higher satisfaction.

Knowledge test

This 20-item test assesses knowledge of CBT for youth anxiety and was previously used in CBT training (19,21). The total score was the total of correct items.

Adherence and Skill Checklist

This checklist was developed to measure therapist fidelity to CBT for child anxiety (19). Adherence, defined as the utilization of the procedures of a protocol in the treatment of a client (22), was assessed by coding the presence of six CBT competencies: identification of somatic symptoms, identification of anxious cognitions, relaxation, coping thoughts, problem solving, and positive reinforcement. The score could range from 0 to 6, with higher scores indicating better adherence. Skill, the level of competence shown by the therapist in the delivery of treatment (22), was evaluated on a Likert scale that ranged from 1 to 7. Coders were blind to hypotheses, training condition, and time point of the assessment. Rated adherence (intra-class correlation=.98) and skill (intra-class correlation=.92) demonstrated outstanding interrater reliability.

Performance-based role-play

A structured role-play was used to assess participants’ adherence and skill in a simulated clinical setting. Research assistants blind to condition were trained to portray anxious youths. Therapists were asked to prepare the client for a treatment session that included an exposure task, a competency central to CBT. The role-plays were digitally recorded and later independently coded with the Adherence and Skill Checklist. Role-plays were conducted pre- and posttraining and at three-month follow-up.

Data analysis

All analyses were conducted with SPSS, version 17.0. Four outcomes were examined: adherence, skill, knowledge, and satisfaction. Analyses were conducted to isolate the impact of training modality from pre- to post-training on the outcome variables and the impact of consultation at the three-month follow-up on the outcome variables. To test the main effects (training and time) and the interaction (training × time) on the outcome variables (adherence, skill, and knowledge), repeated-measures analysis of variance was used. Satisfaction was analyzed with one-way analysis of variance. To test the impact of consultation on outcome variables, linear regression was used to model the effect of number of consultation hours on adherence, skill, and knowledge.

Adherence, skill, and knowledge were analyzed with reference to a clinically significant criterion (12). A 70% cutoff score was used to indicate being trained to criterion for adherence and knowledge. This value was selected to be consistent with the 80% training criterion used in other evaluations of child anxiety CBT (21) but is less conservative because the participants were community clinicians naïve to CBT as a treatment for child anxiety. A skill score of 3.5 was used as the cutoff for being acceptably trained to criterion and was consistent with past trials evaluating the efficacy of CBT (5,12,23). Chi square analyses were used to investigate differences in training to criterion by condition.

Results

Demographic characteristics and retention of participants

Participants were primarily female (90% of sample), and the mean±SD age was 35.9±11.4 years. Table 1 provides additional information about demographic characteristics and professional background.

Table 1.

Characteristics of 115 community therapists

Characteristic	N	%
Demographic
Age (M±SD)	35.93±11.36
Female	104	90
Race or ethnicitya	110	96
Caucasian	77	67
African American	15	13
Hispanic or Latino	6	5
Asian	5	4
Native American	1	1
Other	6	5
Academic degree or background	115	100
M.A. or M.F.T.	47	41
M.S.W. or L.C.S.W.	28	24
Graduate student	18	16
M.D. or D.O.	8	7
Ph.D.	6	5
Psy.D.	5	4
Ed.D.	2	2
Other degree	1	1
Clinical	112	97
Clinical experience (M±SD months)	65.46±82.38
Identification with CBT (M±SD)b	4.86±1.68
Prior experience with treating anxious youths	58	50

^aData were missing for five participants.

^bPossible cognitive-behavioral therapy (CBT) scores, as indicated with one item from the Clinician Demographics and Attitudes Questionnaire, range from 1, not at all, to 7, extremely.

Individuals who completed the pretraining assessment composed the intent-to-train sample. Of the 115 intent-to-train participants, 113 (98%) completed the posttraining assessment and 100 (87%) completed the follow-up assessment.

Adherence

Mean adherence scores significantly improved over time from pretraining (1.6±1.2) to posttraining (3.0±1.4) (F= 83.21, df=1 and 111, p<.001), but there was no significant effect of training or interaction of time and training (Table 2).

Table 2.

Outcomes of augmented, routine, and computer training of 115 community therapists

Measure	Augmented training		Computer training		Routine training		Na	df	pb
	M	SD	M	SD	M	SD
Adherencec
Baseline	1.51	1.21	1.71	1.31	1.63	1.09	114d	2, 111	.78
Posttraining	3.36	1.22	2.71	1.36	2.93	1.51	115	2, 112	.20
Skille
Baseline	2.67	1.22	2.94	1.30	3.02	1.13	114d	2, 111	.40
Posttraining	3.68	1.47	3.67	1.47	4.13	1.45	115	2, 112	.09
Knowledgef
Baseline	15.25	1.92	14.85	2.41	15.27	2.32	115	2, 112	.67
Posttraining	16.83	1.93	17.42	1.81	17.45	1.83	115	2, 112	.26

^aFor participants who dropped out of the study, their last observation was carried forward.

^bFor the main effect of condition

^cPossible scores range from 0 to 6, with higher scores indicating better adherence to protocol. F=83.21, df=1 and 111, p<.001

^dPretraining adherence and skill data were missing for one participant as a result of recording error.

^ePossible scores range from 1 to 7, with higher scores indicating higher competence. F=69.06, df=1 and 111, p<.001

^fPossible scores range from 0 to 20, with higher scores indicating greater knowledge. F=96.74, df=1 and 112, p<.001

Skill

Participant skill significantly improved over time from pretraining (2.9±1.2) to posttraining (3.8±1.5) (F=69.06, df=1 and 111, p<.001), but there was no significant effect of training. A significant interaction of time and training was identified (F=4.55, df=2 and 111, p< .01) such that participants in the augmented training (pretraining, 2.7±1.2; posttraining, 3.7±1.5) improved their skill more from pre- to posttraining compared with participants in the computer training (pretraining, 2.9± 1.3; posttraining, 3.7±1.5) (Table 2).

Knowledge

Participant knowledge scores significantly improved over time from pre-training (15.1±2.2) to posttraining (17.2±1.9) (F=96.74, df=1 and 112, p<.001), but there was no significant effect of training or interaction of time and training (Table 2).

Training to criterion

There were no significant differences across training modalities in percentage of participants trained to criterion (Table 3). Note that these analyses were based on data from only the participants who completed the study, not the intent-to-train sample. Overall, from pretraining to posttraining, the number of participants trained to criterion increased on adherence (32%), skill (37%), and knowledge (15%). From posttraining to follow-up, the number of participants trained to criterion increased on adherence (23%), skill (20%), and knowledge (1%). On average, participants received 13 hours of training (six hours of training and seven hours of consultation). After receiving that dosage, approximately 96% of participants were trained to criterion in knowledge, 61% were trained to criterion in adherence to CBT for child anxiety, and 85% of participants were trained to criterion in skill.

Table 3.

Community therapists trained to criterion after a one-day workshop and three months of follow-up consultation, by training condition

Variable	Overall			Augmented training			Computer training			Routine training			χ2a	p
	Total			Total			Total			Total
	N	N	%	N	N	%	N	N	%	N	N	%
Adherence
Baseline	114	7	6	39	3	8	34	3	9	41	1	2	1.56	.46
Posttraining	113	43	38	39	16	41	34	10	29	40	17	43	1.56	.46
Postconsultation	102	62	61	36	24	67	30	17	57	36	21	58	.83	.66
Skill
Baseline	114	32	28	39	8	21	34	11	32	41	13	32	1.68	.43
Posttraining	113	73	65	39	30	77	34	18	53	40	25	63	4.69	.10
Postconsultation	102	87	85	36	34	94	30	24	80	36	29	81	3.72	.16
Knowledge
Baseline	115	92	80	40	35	88	34	25	74	41	32	78	2.39	.30
Posttraining	115	109	95	40	38	95	34	33	97	41	38	93	.73	.70
Postconsultation	92	88	96	34	33	97	26	24	92	32	31	97	1.94	.75

^adf=2

Satisfaction

Differences in satisfaction across training modalities were observed (F=7.22, df=2 and 112, p<.001). Planned comparisons identified that participants in routine training (53.7±5.4, t=−3.78, df=112, p<.001) and augmented training (55.5±4.7, t=2.40, df=112, p<.05) were more satisfied than participants in computer training (50.8±5.9).

Consultation

On average, the 108 consultation sessions lasted 52.6±10.8 minutes and had 7.8±4.5 participants per session. The mean number of cases discussed per call was 2.7±1.9. Each participant used a mean of 7.2±3.2 consultation hours. Number of consultation hours significantly predicted participant adherence (F=20.67, df=1 and 113, p< .001, β=.39, R²=.16) and skill (F =11.46, df=1 and 113, p<.001, β=.30, R²=.09) at the three-month follow-up but did not significantly predict participant knowledge.

Discussion

Two main findings relevant to the implementation of empirically supported treatments in child mental health services emerged from this study. First, a brief workshop was not effective in changing therapist behavior to implement CBT into practice. All three training modalities (routine, computer, and augmented training) resulted in limited gains in therapist adherence and skill—even brief training emphasizing active learning. Second, ongoing consultation after a brief workshop was effective; it significantly improved therapist adherence and skill and allowed therapists to solidify concepts and skills learned in the brief workshop for embedding CBT within their clinical practices.

There was no effect of training modality on therapist adherence, skill, or knowledge, although satisfaction varied across modalities. The null finding with regard to therapist adherence and skill is consistent with previous research showing that, even at its best, training by itself is not a sufficient implementation strategy (12,13). Consistent with previous findings, knowledge of CBT for youth anxiety increased for all participants from pretraining to posttraining (11,24). Inconsistent with previous studies (11,12), participants were more satisfied with the face-to-face training conditions than with computer training. However, satisfaction was high across training modalities. Given limited resources in community mental health settings and high rates of therapist turnover, computer training may be the most viable training strategy given that it is as effective as instructor-led training.

Ongoing consultation after a brief workshop significantly improved therapist fidelity to an empirically supported treatment. This important finding corroborates theory (6) and suggests that consultation should be included as a core element in implementation. Each hour of consultation after training improved therapist adherence by .4 point and skill by .3 point. These increases suggest a sizable return on investment. The virtual delivery of consultation was especially innovative because a diverse set of clinicians could interact at times that suited their schedules. Virtual consultation seems ideal for limited-resource settings, such as community mental health clinics (25). Furthermore, consultation provided an experiential laboratory where therapists could learn from peers, receive feedback from an expert consultant on actual child cases, practice skills, tailor the treatment for context, and resolve implementation difficulties. Ongoing support made it possible for therapists to embed the new clinical practice into their existing repertoire of skills.

With regard to clinical significance, after a six-hour workshop, approximately 38% of participants were trained to criterion in adherence and 65% were trained to criterion in skill. Ongoing consultation of approximately seven hours over three months brought those numbers up further, to 61% and 85%, respectively. This finding is particularly noteworthy given that other studies have demonstrated only minimal to moderate gains in adherence and skill after training and often a decline at follow-up when training was not followed by consultation (11–13,17). These findings suggest that a relatively limited dosage of training, approximately 13 hours, can result in meaningful behavior change for therapists when consultation is included.

Despite several strengths of this study, there were limitations. The focus on an empirically supported treatment for child anxiety may not be generalizable to training in other treatments. However, a number of child mental health service settings were represented (including schools, outpatient clinics, hospitals, and community mental health clinics) and resulted in high generalizability across settings. The sample was self-referred, which may not be generalizable to therapists mandated to receive training. In addition, the therapists identified highly with CBT, had moderate knowledge of CBT, were primarily female and Caucasian, and had a master’s-level education, which may not be generalizable to other samples. Finally, because our study endeavored to control the number of training hours in each workshop, it was necessary for participants to receive the computer training in a computer lab. This was a limitation because in a real-world setting, computer training would occur in the home or office. Further, the computer training was not highly interactive; thus these findings may not generalize to computer training that includes more interactive design.

Conclusions

Although a one-day workshop can be effective in changing provider knowledge, it is not effective in changing provider behavior. We recommend that training no longer be used as a stand-alone implementation strategy. However, with the combined implementation strategies of brief training and ongoing consultation, community clinicians can be effectively trained in clinical innovations that are complex and multistep (26). Brief training may be best provided through computer delivery given that it is a permanent product that an agency can purchase that is effective, accessible, and cost-efficient and can be used in situations where there is high therapist turnover (8). Most important, consultation merits as much emphasis as brief training, and we suggest that whatever limited resources are available be used to provide consultation. Unfortunately, little empirical work has been completed on consultation as an implementation strategy. Research on supervision (27) and coaching teachers (28) can inform this emerging and exciting area. Investigation of how to train consultants, what type of consultation is most effective (individual versus group), and the merit of internal versus external consultation is needed.