Randomized Trial of Web-based Training to Promote Counselor Use of CBT Skills in Client Sessions

Abstract

With funding from the National Institute on Drug Abuse, we delivered a Web training program on cognitive behavioral therapy (CBT) to addiction counselors and supervisors in 54 U.S. addiction units and conducted a randomized controlled trial with 127 counselors in 2006–07. Adequate adherence to CBT practice at pre- and post-training was judged from audiotapes of client sessions using an adequacy rating guide of counseling skills. A web-administered questionnaire assessed demographics, prior training, attitudes, and self-report counseling practices. Logistic regression model findings are described with discussion of dissemination of evidence-based practices, study limitations, and future research needs for empirically-supported training programs.

1. INTRODUCTION

The gap between science and standard practice in substance abuse treatment is widely acknowledged.^1–3 Many empirically supported treatments (ESTs) do not find their way into practice. Cognitive behavioral therapy (CBT) is a specific EST with ample evidence supporting its use with addiction.^4–7 Although there are high quality ESTs, outcomes of training programs to disseminate ESTs have been disappointing,^2,8–10 with considerable variability in clinician proficiency even with extensive training and feedback.^2,8,10–13 One review of 17 evaluations concluded that workshop training produced some immediate improvements but did not result in maintenance of gains over time,¹⁴ and on-the-job workshop formats have not resulted in skill acquisition.¹⁵ While intensive training and supervision has been found to result in skill mastery, it is too costly and time-prohibitive.^16–18

Use of interactive Web technologies may be promising for dissemination of CBT skills in addiction treatment.^11,19 While web-technologies and distance education are increasing in use by more organizations,²⁰ including universities²¹ to Addiction Technology Transfer Centers, the evaluations remain focused primarily on acceptability to learners, change in knowledge, and participant self-ratings.²² The present study was conducted to empirically test the skill acquisition of addiction counselors exposed to an intensive Web course in the workplace.

2. STUDY DESIGN

With funding from the National Institute on Drug Abuse (NIDA), we designed and delivered an innovative, asynchronous Web course for addiction counselors and conducted a randomized trial of its effectiveness. The primary goal of TEACH-CBT (Technology to Enhance Addiction Counselor Helping) was to increase use of specific CBT skills in sessions with clients. We evaluated the effectiveness in a randomized trial of clinical teams at community-based programs, seeking a critical mass of trainees at a program who could reinforce each other’s learning.^23–25 We measured the transfer of learned skills to the counselor’s typical clients, a key dissemination challenge.^26–29

Training content was similar across intervention and control groups; it was derived from the NIDA training manual distributed to the control group:³⁰ definition of CBT session structure, collaborative session agendas, and out-of-session assignments; motivational interviewing topics; functional analysis and the cognitive behavioral model of addiction; behavioral skills practice; cognitive skills practice; applying CBT to HIV risk behaviors; and other life problems that interfere with recovery.

This paper presents analysis of whether Web counselors were more likely to pass a pre-specified criterion of “adequate adherence to CBT practice” after training and whether they gained significantly more skills than control group participants.

Participants

TEACH-CBT recruited 63 addiction units and 176 counselors and screened counselors at 61 addiction units as two units were lost because of local IRB delays. Addiction unit eligibility criteria were director’s support letter; eligible team of supervisor and 2 or more counselors; and a ‘study liaison’ to consent clients for audio-taping. Counselor eligibility criteria were: Master’s counseling degree or Bachelor’s and addiction certification; two years of experience; 6 or more clients on caseload; six or more sessions with clients was permitted; and submission of baseline audio-tapes. Counselors with prior CBT training of eight or more hours and supervised practice were excluded.

Web course completers earned 16 continuing education units and other nominal financial incentives were offered for attending orientation and submitting audiotapes. The Institutional Review Board at New England Research Institutes approved all study recruitment, consent, and data collection procedures and a data safety monitoring plan was submitted to NIDA.

Randomized units included 36 standard outpatient, 12 residential, eight methadone maintenance, two acute inpatient and three combined modalities, with 127 counselor participants (Figure 1).

FIGURE 1.

Teach-CBT study flow

Study Materials

Supervisors attended two webinar ‘orientation’ sessions and were encouraged to take CBT training concurrently with the counselors (Web or manual training based on assigned group).

A faculty member hosted monthly supervisor phone calls. The maximum attendance was seven and average attendance was three or four.³¹ Supervisors in both conditions were encouraged to discuss with counselors the self-monitoring of CBT skill application using the on-line tool (Web Course) or written forms (NIDA manual), to participate in trainee assignments, and to learn how counselors were implementing CBT in client sessions.

The Web training had discrete knowledge and skill learning objectives and 30 short screens, one-half providing optional content, on-line exercise, or single questions with feedback. Each module had audio-vignette dialogue of a role-played treatment session or expert commentary, and an on-line assignment to complete between modules. A pilot study of one module informed final module format.³²

Measurement Protocol

Data were collected at two time points after baseline. Post-training data were collected at 8 weeks (or Web completion) and follow-up data were collected 3 months later with assistance of an innovative data management system described in Muroff et al.³³

Each counselor submitted audiotapes of a single session with two clients at baseline. At both post-training and follow-up, counselors submitted audiotapes of 3 sessions for a total of 8 audiotapes except for 5 agencies ineligible for follow-up data collection as it closed One randomly selected tape for each counselor at each time point was selected for analysis and transcribed.

Participants and supervisors completed questionnaires at each time point with questions on personal demographics, prior training, exposure to and adoption of new techniques, attitudes towards ESTs, barriers to using CBT, CBT strategies used in client sessions, and a set of CBT knowledge items.

We developed an Audiotape Rating Guide (ARG) that reflected application of the skills taught and defined core criteria for eight counseling skills taught in both the Web course and covered in the NIDA manual (available upon request). To enhance validity of the ARG and rating system, the investigators consulted with nationally-recognized CBT experts (see acknowledgements). Each core counseling skill was the composite of multiple expected behaviors (i.e., sub-items) that would be present if the intervention were delivered adequately. The core skills comprised 3 broad domains: a client-centered motivational stance, generic skills informed by CBT theory, and specific CBT interventions (Figure 2).

FIGURE 2.

Five raters received training on the ARG, applied it to the same five audiotape sessions, and were debriefed on concordance in ratings. Raters evaluated the presence/absence of the expected behavior and how thoroughly it was performed on a 4-point scale (once/barely, somewhat, thoroughly, very thoroughly). Raters then evaluated the overall degree and adequacy of the clinician’s demonstrated multi-dimensional skill on a 7-point scale (1=not performed, to 7= considerable use and very good application). A 4 was equivalent to low pass or “sufficient use & minimally-adequate application”, and a 5 was equivalent to pass or “sufficient use and adequate rater training and at the end of rating. Six randomly selected audiotapes were rated by two raters and scores compared. Percent agreement on pairs ranged from 40 to 100 percent and averaged 57 percent for only moderate agreement, which unfortunately reduced the probability that we would detect a real difference between training groups.

Data Analysis

The primary outcome is adequate adherence to CBT delivery, a Pass/Fail outcome predefined as meeting two criteria: low pass or greater on at least one of three CBT-generic skills, and low pass or greater on at least one of three CBT-specific skills. Change in average skill score also was measured for CBT overall skill, for CBT-generic skills, and for CBT-specific skills.

A generalized linear model of the pass/fail outcome variable as a function of addiction unit and counselors nested within addiction units was used to assess the need to model at the unit level. The lack of a significant p-value (p=.10) for the test indicated no need to perform the analysis at the unit-level nor use a multi-level model.

For each binary outcome, the number of counselors passing by training group and measurement wave were reported. This was broken down further by gender, education, certification, older age, in recovery, large agency (top third in number of clients), prior motivational interviewing training, active supervisor, agency modality (stratification variable) and race.

Logistic models were used for binary outcomes. Passing at the post-training audio-taping was modeled as a function of training group, with a control variable for passing at baseline and a term for each rater. Results of additional exploratory models examining the impact of key covariates are not presented as no covariates were significantly associated with the outcomes. Similar models were constructed using generalized linear models for the continuous measures of average skill scores, with baseline continuous scores included in the model.

3. STUDY RESULTS

The non-randomized agency units differed from randomized units on several characteristics: more likely to be residential modality, located in the southeastern U.S., smaller number of clients, and the proposed study supervisors did not have advanced degrees (data not shown). The counselors who were not randomized did not differ on demographic characteristics, practice characteristics, or recovery status from the counselor participants at randomized units (Table 1). Randomization of agency units created balance on modality and state among the Web and control units although the Web-course participants were more likely to be at large-sized agencies, measured as the average weekly caseload. There was a marginally-significant trend for fewer Web-course participants to be white non-Hispanic.

Table 1.

Characteristics of Counselors by Randomization Group

Characteristic	Intervention Group N=62	Control Group N=65	Comparison of randomized groups	At All Randomized Units N=127	At Units Not Randomized N=51
	N (%) of group		Statistic p value	N (%) of group
US Region   Northeast   South   West   Midwest	32 (51.6) 11 (17.7) 14 (22.6) 5(8.1)	35 (53.9) 6 (9.2) 9 (13.6) 15 (23.1)	Χ2=7.6 p=0.054	67 (52.8) 17 (13.4) 23 (18.1) 20 (15.8)	30 (58.8) 12 (23.5) 5 (9.8) 4 (7.8)
Predominant Modality   Methadone   Residential (or inpatient)   Outpatient drug-free	10 (16.1) 11 (17.7) 41 (66.1)	6 (9.2) 18 (27.7) 41 (63.1)	Χ2=2.6 p=0.27	16 (12.6) 29 (22.8) 82 (64.6)	4 (7.8) 18 (35.3) 29 (56.9)
Counselor weekly addiction caseload   Mean (SD), clients   Median, range, clients	4.0 (8.5) 12, 3 – 38	11.4 (8.1) 9.5, 0 – 40	T = −1.77 p =0.08	12.8 (8.4) 10, 0–40	14.6 (9.1) 10, 1–40
Years of counseling experience   Mean (SD)   Median, range	6.7 (4.9) 5, 0.8 – 21.3	7.5 (6.0) 5.5, 0.6 – 25.0	T = 0.82 p=0.41	7.1 (5.5) 5.1, 0.6–25	9.3 (8.2) 5.6, 1 – 30
Certification/registration/licensure   Certified addiction counselor   Registered   Licensed   None of above	19 (31.2) 9 (14.8) 1 (1.6) 32 (52.5)	19 (29.2) 11 (16.9) 1 (1.5) 34 (52.3)	Χ2=0.13 p=0.97	38 (71.7) 20 (15.9) 2 (1.6) 66 (52.4)	7 (25.0) 4 (14.3) 1 (3.6) 16 (57.1)
Highest degree   Master’s or more   BA   Associate or less	38 (61.3) 17 (27.4) 7 (11.3)	39 (60.0) 18 (27.7) 8 (12.3)	Χ2=0.04 p=0.98	77 (60.6) 35 (27.6) 15 (11.8)	17 (60.7) 8 (28.6) 3 (10.7)
Prior advanced/specialized MI training	23 (37.7)	27 (41.5)	Χ2=0.19 p=0.66	50 (39.7%)	12 (42.9%)
Prior on-line professional training course	17 (27.9)	15 (23.1)	Χ2=0.38 p=0.54	32 (25.4%)	10 (35.7%)
Female	37 (59.7)	47 (72.3)	Χ2=2.26 p=0.13	84 (66.1%)	18 (66.7%)
Average age (n=123)   Percent 44 or older   Mean (SD), years   Median, range, years	30 (50.0) 43.1 (11.2) 43.5, 25–64	35 (55.6) 44.6 (12.0) 45, 22–65	T=0.70 p=0.48	65 (52.9) 43.9 (11.6) 44, 25–65	20 (71.4) 47.8 (11.1) 48 (25–65)
Self-identified, in recovery (n=120)	37 (61.7)	41 (68.3)	Χ2=0.59 p=0.44	78 (65.0)	9 (33.3)
Agency weekly clients   In large agency (100 or more) N, (%)   Mean (SD), clients   Median, range, clients	29 (48.3) 187.6 (217.0) 80, 11 – 900	18 (28.6) 98.6 (123.3) 60, 2 -- 550	Χ2=9.8 p=0.002	47 (38.2) 142.0 (180.2) 70, 2 – 900	15 (29.4) 116.6 (135.3) 60, 12 – 560
Race/ethnicity   Hispanic   Black non-Hispanic   White non-Hispanic	5 (8.1) 15 (24.2) 42 (67.7)	4 (6.2) 6 (9.2) 55 (84.6)	Χ2=5.6 p=0.06	9 (7.1) 21 (16.5) 97 (76.4)	4 (14.3) 5 (17.9) 19 (67.9)

Follow-up Rate

The number of audiotapes at baseline, post-training and 3-month follow-up were 251, 313, and 255, respectively. All control group agencies and 92.8 percent (26 of 28) of intervention agencies submitted post-training audiotapes. Further, post-training audiotapes were received from 85.8 percent (109 of 127) of participating counselors; 88.6 percent of Web-group counselors and 100 percent of control-group counselors. For the counselors that lacked pairs of data, the statistician substituted an audiotape from another counselor in the same training condition.

Outcomes

Fifty-seven assigned counselors started the Web course (92 percent) and of those, 82 percent completed 6 or more of the 8 modules. Of 28 Web supervisors, 26 started the course and 19 completed 6 or more modules.

The percentage of counselors from each group with passing scores is presented in Table 2. A comparison of the unadjusted post-test pass rates of the Web and control groups was not statistically significant on overall CBT skill adequacy. Table 2 also shows the unadjusted pass rates of both groups on CBT-specific skills and CBT-generic skills.

Table 2.

Percent of Counselors Passing on 3 Core Skill Outcomes by Time, Unadjusted

Session Audiotape Core Skill Outcome	Web Course Group (n=47)		Control Group (n=52)		Difference between groups p value
	Percent Pass	Chi- square p value by time	Percent Pass	Chi-square p value by time
Overall CBT Skills (All 6 CBT domains)   Baseline   Post-training	17.0% 28.3%	Χ2=1.94 p=0.16	21.2% 40.4%	Χ2=4.51 p=0.034	Χ2=0.30 p=0.59 Χ2=1.70 p=0.19
Generic CBT Skills(3 domains)   Baseline   Post-training	45.3% 49.1%	Χ2=0.15 p=0.70	34.6% 57.7%	Χ2=5.57 p=0.018	Χ2=1.24 p=0.26 Χ2=0.79 p=0.38
CBT-Specific Interventions (3 domains)   Baseline   Post-training	26.4% 34.0%	Χ2=0.72 p=0.40	28.8% 46.2%	Χ2=3.32 p=0.068	Χ2=0.08 p=0.78 Χ2=1.63 p=0.20

Percent pass on a core skill domain was based on a Low Pass benchmark which was 4 or more on a 7-point scale.

Table 3 presents unadjusted findings on the secondary outcome of continuous CBT skill adequacy scores. The post-training gains in overall CBT skill adequacy were small (about 10 percent gain) and not significant for the Web-course group and marginally significant for the control group. The post-test scores did not differ statistically between the Web course and control groups.

Table 3.

Average Rating of Counselors on Core Skill Outcomes by Time, Unadjusted

Session Audiotape Core Skill Outcome	Web Course Group (n=47)		Control Group (n=52)		Difference between groups p value
	Mean Score	p value score by time	Mean Score	p value core by time
Overall CBT Skills (All 6 CBT domains)   Baseline   Post-training	2.4 2.6	p=0.21	2.3 2.8	p=0.058	p=0.99 p=0.50
Generic CBT Skills (3 domains)   Baseline   Post-training	2.5 2.8	p=0.21	2.3 2.8	p=0.032	p=0.38 p=0.99
CBT-Specific Interventions (3 domains)   Baseline   Post-training	2.0 2.2	p=0.22	2.2 2.4	p=0.32	p=0.29 p=0.37

Mean score based on a 0–7 point scale. Mean differences evaluated with a t-test.

The logistic regression models (data not shown) confirmed that there were no statistically significant differences in pass rates and adequacy scores between training groups at post-test. One additional model was estimated which included an interaction term for training group and addiction unit size (largest vs. other). The coefficient for this interaction term was marginally significant (3.54, p=0.06). In small units (under 100 clients per week), the control group had a 47.2 percent pass rate while the Web-course group had a 23.1 percent pass rate. In larger units, the control group had a 21.4 percent pass rate while the Web course group had a 32.0 percent pass rate.

4. DISCUSSION

Study Innovations and Achievements

This project implemented an ambitious study with a large, community-based sample of counselors recruited and trained within their own work environments. The research design accounted for several factors not addressed in previous training studies. We incorporated several features that were intended to enhance the transfer of learning into addiction program practice. We developed an innovative, comprehensive Web-based training course which taught and provided practice on evidence-based treatment content for three CBT-specific skill areas (functional analysis, behavior skills practice, cognitive skills practice), three CBT-generic skill areas for structuring client sessions, and additional client-centered intervention methods (e.g., motivational interviewing, collaborative treatment). The Web course provided a comprehensive set of training materials but did not provide an accompanying printed manual.

The study design acknowledged the role of the environment for training uptake and the translation of skills; we recruited small teams rather than individual counselors to develop a critical mass of committed individuals within an addiction unit.^34–36 While we adhered to the recruitment of agency teams, this approach led to additional research challenges, most importantly recruitment delays in accruing teams rather than individuals and loss to study enrollment of interested counselors who worked in units that could not meet full criteria. We adopted this approach because of our commitment to a training model that stressed supervisor and colleague support when learning new skills. Nevertheless, addiction unit recruitment for a training program was a cumbersome process, and we did not obtain evidence that supervisor support and encouragement increased training outcomes.

One research achievement was the acquisition of up to 8 audiotapes from each counselor of ‘in vivo’ sessions at these community-based settings. High follow-up rates on post-training questionnaires were also achieved. We analyzed what actually occurred in the clinicians’ counseling sessions through review of transcripts and scoring of specific counselor behaviors. Thus, our effectiveness study of training was based on measures of actual skill transfer. Further, training was conducted within the counselor’s work environment rather than an outside workshop or training program.

To our knowledge this is the first randomized trial of skill transfer in own work place among community-based substance abuse clinicians. At the time our study began (2003), only a small number of published studies had reported any efforts to measure counselor skill attainment for community based counselors.^18,21 Only a few training studies have captured demonstration of skill acquisition through role-play,¹⁹ simulated-patient actors ³⁷ or client “training cases” and these scenarios are not necessarily typical of the counselor’s caseload.^12,38 Our emphasis on transfer of skills to ongoing client treatment in counselors’ own agency setting is a possible model for NIDA/SAMHSA Blending Initiatives which aim to support adoption of ESTs in everyday practice. Since this study’s onset, additional researchers have now assessed outcomes through skills demonstrated with the counselor’s actual clients via audiotaping.^39–41

Unexpectedly, we found an interaction of training effects with agency unit size. In larger addiction units, Web clinicians reached higher pass rates than clinicians in the control group. In smaller agencies, clinicians in the control group reached higher pass rates than the Web group. This finding should be replicated in future studies as there may be real training effect differences associated with agency size.

The study’s results provided no evidence of a difference in skills transfer associated with Web-course participation relative to training with a treatment manual. There may be several factors that contributed to this null finding. First, the study created similarities in training conditions for both groups that might not have existed otherwise. Both groups were encouraged to engage in active exposure to training practice using the training materials with supervisor support. Further, we did not provide a printed manual of training materials to the Web-course participants and the control group may have had some advantages in practicing with supervisors or trainees using the printed training manual. While both groups were encouraged to self-monitor skill application, the Web course on-line self-monitoring materials were rarely used and the Web discussion board did not lead to ongoing discussion between participants. Thus, elements of the Web course that were intended to create a shared learning community did not develop. It is possible that technology advances since completion of this study will make full use of Web courses more readily accessible to addiction counselors.

It is unclear whether the Web-group counselors failed to gain more skills than the control group or failed to put them into actual practice. We have proxy measures for future analyses such as self-report of skills used in counseling sessions and knowledge improvement, but are not encouraged as prior research shows that self-rating of skills is not necessarily correlated with objective skill assessments.^42–44 A good number of speculative variables may have overshadowed the single effects of training method including therapist-related (prior training and experience, time devoted to training, out-of-session skill practice; rapport with clients), supervision-related (type and amount, adherence monitoring), client-related (level of impairment), and agency-related (administrative support).

Our study differs from others in measuring actual skills in actual practice, but also our training approach relative to other Web courses may have contributed to null findings. We covered very broad and comprehensive materials and attempted to provide a flexible set of CBT principles and practices. However, this scope and flexibility probably increased the complexity of the learning experience for participants and also hindered evaluation of the value of the Web training. The Web course did not propose specific session topics (“manualized” approach) but rather described empirically-supported techniques appropriate for different kinds of problems that incorporate into a session. This flexible approach may contribute to counselor uncertainty about how and when to apply specific concepts, certainly a threat to implementation of skills. More recent Web course trials have limited the scope of content and measured only one or several skills. For example, in the study by Sholomskas and colleagues,¹⁹ the outcome measure was demonstration of three key CBT interventions which included explaining the CBT rationale for treatment and conducting a functional analysis of drug use, coping with craving, and examining seemingly irrelevant decisions (automatic thoughts). Weingardt and colleagues ⁴⁵ tested 8 CBT modules with three unified outcome measures: CBT knowledge, self-efficacy, and job burnout. With the lengthy exposure (8 or more weeks) and varied material of the current Web course, we may have reduced the counselors’ opportunity to demonstrate superior performance for any given, narrow, content domain.

Nevertheless, despite the promise of Web-course learning as a dissemination mechanism, there may be real limits when it is not accompanied by other training support. There likely is no substitute for individual feedback to counselors-in-training via expert coaching or supervision.^37,46,47 Even then, research is showing that the gains from a single training program may be modest.^14,19 Efforts to improve proficiency of counseling in practice may take ongoing support and repeated efforts. In sum, even if a Web course contributes to CBT knowledge and skill acquisition, there is still a need for structured, personalized ongoing expert coaching for skill transfer.

Study limitations

Particular study limitations should be noted. The marginally improved scores of the control group is an unexpected finding. Preliminary analysis indicates that the control group “achievement” is not limited to the audiotape data but also includes knowledge gains⁴⁸ and self-report of skill application.⁴⁹ These findings suggest that control group members engaged in many study-related activities that were not part of their usual clinical functioning (perhaps a variation of the proverbial Hawthorne effect).^50,51 Further, agencies were more CBT-savvy than anticipated, perhaps because six Web course and five control group agencies were also a part of NIDA’s Clinical Trials Network and hence involved in other training opportunities.

Two other issues may have diminished the study’s ability to distinguish between well-trained and less well-trained clinicians. Regarding the audiotape ratings, the sessions were in vivo and not scripted by the study in terms of session goal or topic. Without such topic prompting, the well-trained counselor in either group may not have demonstrated skills he or she had acquired in a specific CBT strategy. Second, improved inter-rater reliability would be a goal in future studies. While two raters had average pass rates at baseline that conformed closely to our a priori assumptions (less than 20 percent of counselors passing), the others had higher average pass rates. Moderate inter-rater agreement implies lack of precision in what was being rated and would reduce the chance of detecting a significant improvement associated with training. We note that many studies we reviewed did not report inter-rater reliability which may contribute overall to null findings in the literature.⁴⁰ Miller, Moyers, and colleagues found that exact rater agreement was 55.9 percent on the 7-point Combined Behavioral Intervention (CBI) scale. ² On balance, however, our procedures did not reach the standard set by other studies^52,53 Finally, we report only the post-training audiotape findings. Our original hypotheses were that the two groups would differ on skills acquired from training and that these differences would be maintained over a follow-up period. Given what we learned in these analyses, we do not believe that we have the power to detect the small differences between training groups that may have occurred at follow-up, if any. Additionally, we know the 3-month follow-up audio-tape data are biased in that the last five agencies randomized (primarily from one state; all with IRB issues that delayed data collection) were not asked to complete the follow-up audio-tape batch because of study time constraints.

Conclusion

We conclude that Web-based training is feasible and attractive to clinicians providing substance abuse services. It should be pursued at this time with continued evaluation of its effectiveness relative to other training methods. Given that the majority of counselors in this study did not reach adequate levels of translating CBT into practice in either condition, training programs should be accompanied by coaching and supervision rather than be stand-alone.

There remains a need for new technologies to assist clinicians in translating what is learned in training into actual practice. We also agree with McHugh and Barlow’s review⁵⁴ of evidence-based psychological treatments which concluded that, when evaluating the efficacy of dissemination and implementation of ESTs, clinical outcomes should be accompanied by training outcomes that measure successful acquisition and adoption of competent skills. While Web courses address a clear need to expose more clinicians to ESTs, the challenge remains to develop low-cost programs with demonstrated effectiveness in skill acquisition during and after training exposure.