Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Oct 1.
Published in final edited form as: J Subst Abuse Treat. 2013 Jun 27;45(4):10.1016/j.jsat.2013.05.005. doi: 10.1016/j.jsat.2013.05.005

The RoadMAP Relapse Prevention Group Counseling Toolkit™: Counselor Adherence & Competence Outcomes

Adam C Brooks 1, Carolyn M Carpenedo 1, Jaymes Fairfax-Columbo 1, Nicolle T Clements 2, Lois A Benishek 1,3, Dan Knoblach 1, Deni Carise 3,4, Kimberly C Kirby 1,3
PMCID: PMC3867814  NIHMSID: NIHMS501665  PMID: 23810230

Abstract

Training counselors in empirically-supported treatments (ESTs) far exceeds the ever-decreasing resources of community-based treatment agencies. The purpose of this study was to examine outpatient substance abuse group counselors' (n=19) adherence and competence in communicating and utilizing concepts associated with empirically-supported relapse prevention treatment following a brief multimedia toolkit (RoadMAP Toolkit™) training. Moderate or large baseline to post-training effect sizes for counselor adherence to toolkit content were identified for 13 of 21 targeted behaviors (overall d range=.06-2.85) with the largest gains on items measuring active skill practice. Post-training adherence gains were largely maintained at the 6-month follow-up, although no statistically significant improvements were identified over time for counselor competence. This study provides important preliminary support for using a multi-media curriculum approach to increase empirically-supported relapse prevention skills among group counselors. Future research should focus on finding ways to improve counselor skill level and to determine the impact of the Toolkit on client outcomes.

Keywords: group counseling, curriculum, counselor training, relapse prevention, cognitive-behavioral, substance abuse

1. Introduction

Considerable investment has been made in the development of empirically-supported psychosocial treatments (ESTs) for the treatment of substance use disorders (Miller & Willbourne, 2002), but the adoption and implementation rates of ESTs in community based treatment are low (Garner, 2009; Massatti, Sweeney, Panzano, & Roth, 2008; Miller Sorenson, Selzer, & Brigham, 2006). Providers of addiction treatment face numerous workforce challenges (e.g., varying quality in the workforce, counselor turnover) that hamper training, supervision, and quality assurance (Eby, Burk, & Maher, 2010; Kerwin, Walker-Smith, & Kirby, 2006; McLellan, Carise, & Kleber, 2003). Consequently, EST penetration in community-based substance abuse treatment has been less than desired among counselors who provide individual (Santa Ana, Martino, Ball, Nich, Frankforter, & Carroll, 2008) and group counseling (Knoblach, Brooks, Nick, Carpenedo et al., in review). Training in ESTs continues to be provided through manuals and didactic continuing education workshops, despite a decade of research demonstrating that while self-study and workshop approaches may result in improvements in knowledge about and acceptance of ESTs, they do not typically improve clinical adherence or competence (Beidas & Kendall, 2010; Miller & Mount, 2001; Walters, Baer, Matson, Ziedonis, 2005). Workshops supplemented by ongoing feedback or supervision does result in sustained improvements in counselor practice (Baer, Rosengren, Dunn, Wells, Ogle, & Hartzler, 2004; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004; Sholomskas, Syracuse-Siewert, Rounsavill, Ball, Nuro, & Carroll, 2005; Walters et al., 2005). However, these approaches are costly and potentially unwieldy, and require the ongoing involvement of trainers and supervisors.

Our team has been exploring the use of a multimedia group counseling curriculum toolkit to supplement training (Brooks, DiGuiseppi, Laudet, Rosenwasser, et al., 2012; Carise, Brooks, Alterman, McLellan, Hoover & Foreman, 2009). Treatment curricula focused on the delivery of group counseling are not new (Hoffman, Landry, & Caudill, 2003; Rawson et al., 1995; McGovern, Drake, Merrens, Mueser, & Brunette, 2008), but they have received significantly less research attention than corresponding individual-focused psychosocial treatments. This is unfortunate, since the majority of community-based treatment is provided in a group format (Price et al., 1991; Weiss, Jaffe, de Menil, & Cogley, 2004).

As previously reported and described in this journal, our team developed a brief multimedia RoadMAP Relapse Prevention Toolkit™ (Brooks et al., 2012). This Toolkit (which consists of video vignettes, colorful posters, worksheets, and teaching aids) is designed to assist counselors in increasing the amount of evidence-based relapse prevention content (Marlatt & Gordon, 1985) provided in their group counseling sessions with minimal training. It introduces counselors to key relapse prevention content and simple, repetitive core strategies that can be easily taught to a diverse group of clients. The Toolkit serves as both a mode of information transfer to clients, as well as a powerful real-time behavioral prompt and teaching tool for counselors (see Brooks et al., 2012 for details about the underlying theoretical foundations, development, content, and initial Toolkit evaluation).

Our team found that counselors reported high levels of satisfaction with both a one-session Toolkit prototype (Carise et al., 2009) and the RoadMAP Toolkit™ (Brooks et al., 2012). Additionally, we found that counselor relapse prevention content adherence demonstrated a very large effect in improvement after completing a brief Toolkit training and receiving no additional supervision or feedback (Brooks et al., 2012); counselor competence, already at adequate-to-average levels at baseline, did not change.

Our previously reported work demonstrated gross changes in pre-post counselor adherence. However, the Toolkit was designed to change counselor practice by promoting increased active skill practice in group treatment; we aimed to accomplish this by 1) embedding active skill practice opportunities in each module 2) training counselors in two key cognitive behavioral therapy (CBT) skills (functional analysis, homework assignment/review) and 3) encouraging their repetitive use. While it is true that the acquisition of coping skills has not typically been shown to mediate the effects of CBT treatment on outcomes (Morgenstern &Longabaugh, 2000), CBT manuals continue to stress the importance of counselors teaching coping skills to their clients and encouraging active practice in and out of the treatment session (Carroll, 1998; Monti, Kadden, Rohsenow, Cooney, & Abrams, 2002). In addition, increased homework engagement is associated not only with the development of more frequent and competent use of coping skills but also with decreases in drug use (Carroll, Nich, & Ball, 2005; Gonzalez, Schmitz, & DeLaune, 2006; Kiluk, Nich, Babuscio, & Carroll, 2010). However, despite the importance attributed to active skill practice in manualized CBT approaches, our preliminary work shows that it is rare in community practice. Counselors enrolled in our RoadMAP Toolkit™ study reported weekly on the types of groups and activities implemented with their clients over a 12-week period prior to being exposed to the Toolkit (Knoblach et al., under review). Their “treatment as usual” data indicated that 25% of their groups were self-reported as CBT in nature. Interestingly, 31% of the group time was spent discussing non-CBT topics, and very little time was spent implementing key practice skill behaviors. For example, only 5% of their group time was spent analyzing clients' drug use patterns or goal setting, only 3% of the time helped clients identify and make plans for engaging in non-drug using activities, and only 1% of session time was used to conduct in-session role-plays, practice activities, or review homework. These findings indicate that counselors spent much more time discussing content rather than engaging in actual skills practice that could play a central role in preventing relapse. The extent to which the use of a CBT RP toolkit can increase group counselors' actual implementation of skill practice behaviors with their clients is not known.

This current report expands on our initial findings in several relevant ways. In this report we examine the extent to which counselors utilize specific skills acquisition behaviors associated with Coping with Craving (CwC) and Drug Refusal Skills (DRS) shortly after they have been provided with (and minimally trained to use) our RoadMAP Toolkit™. Second, changes in counselor adherence and competence associated with these two content areas at baseline, post-toolkit training, and six months post- training are examined. Third, we report on post–training and 6-month follow-up rates of counselors' use of three core RP training strategies (i.e., homework assignment and review, functional analysis of drug use); additionally, we compared adherence and competence scores at these time points for toolkit module 1 (for which we provided direct counselor training; i.e., My Addiction Pattern) to two other modules (for which we did not provide training; i.e., CwC, DRS).

2. Methods

2.1 Participants

Counselors who were employed at one of three mid-Atlantic community outpatient substance abuse treatment programs were invited to participate in this study. They were eligible to participate in the study if they conducted at least two adult counseling group sessions per week and were willing to have their group sessions observed by research staff. Twenty-six of 28 eligible counselors (93%) consented to participate in the study; of these, seven were reassigned or changed positions before baseline observations were completed. Nineteen counselors provided complete observation data at the baseline period, 17 (89%) of which provided full post-training data and 14 (74%) of which provided full observation data at all three time points.

2.2 Design and Procedures

2.2.1 Design Overview

This study consists of a single-arm, repeated measures design intended to examine the impact that the RoadMAP Toolkit™ had on counselor adherence and competence at implementing RP content in their group counseling sessions. Human subjects approval was obtained from the Treatment Research Institute and the Philadelphia Department of Health IRBs.

2.2.2 Recruitment

Research staff attended a scheduled meeting with treatment program staff, provided a brief overview of the study (including eligibility criteria), and invited counselor participation. Research staff then met individually with interested counselors to answer questions and (when appropriate) complete the consent and HIPAA processes as well as a baseline assessment.

2.2.2.1 Baseline Observations of Group Counseling Sessions

Counselor participation in directly observed groups occurred in three phases: 1) Baseline, occurring three months prior to Toolkit training; 2) Post-Training, occurring after the three-month baseline period was completed and within 2-4 weeks of the Toolkit training; and 3) Follow-up, occurring six months after the Toolkit training. In each phase, counselors allowed coders to observe four group counseling sessions (with concurrent client consent). At Baseline, the first two were “general” groups in which they were free to focus on a topic of their choice. For their third and fourth groups, counselors were asked to conduct the best group possible, using whatever materials they wished, on two common RP topics: Coping with Cravings (CwC; i.e. “how to deal with cravings and urges”), and Drug Refusal Skills (DRS; i.e., “how to handle unwanted offers of alcohol or other drugs”). These two content areas were chosen because they are central to most cognitive-behavioral relapse prevention approaches, and we did not want to unduly increase the counselors' work load by asking them to conduct six (rather than two) baseline sessions. A trained coder observed and coded counselor behaviors at each group session. See our previously published report (Brooks et al., 2012) for additional information.

2.2.2.2 RoadMAP Toolkit ™ and Toolkit Training

Following the baseline period, counselors received their copies of the RoadMAP Toolkit™ and participated in a three-hour training in its use. Briefly, the Toolkit consists of six RP-related modules: My Addiction Patterns/functional analysis of drug use behaviors, CwC, DRS, Managing Difficult Emotions,Seeminly Harmless Decisions, Finding Meaning in Recovery). It contains simple clinical guides for each module that contain key teaching points and supplemental materials (e.g., posters, worksheets, recovery cards, videos; see Brooks et al., 2012 for a detailed description of the Toolkit and training).

After reviewing the benefits of RP as an EST and showing the components of the Toolkit, counselors were walked through Toolkit Module 1 (drug use patterns and functional analysis), with emphasis on how each aspect of the Toolkit module could be used to teach relevant RP concepts and engage clients in various relevant recovery activities both within and outside of the group counseling session. Counselors were given an opportunity to conduct a functional analysis and taught how to use the recurring homework assignment (scheduling positive activities). Counselors received no direct training or instruction on how to conduct the remaining five Toolkit modules, and were simply informed that once they understood how one session worked they would be able to teach themselves the rest of the curriculum as they prepared for each session. In addition, counselors were not provided with any additional training, supervision, or tips throughout the rest of the study and were given two weeks to familiarize themselves with the Toolkit.

2.2.2.3 Post-Training and 6-Month Follow-Up Observations

After this two-week period, we conducted four weekly group observations of the first four Toolkit sessions (i.e., My Addiction Patterns, CwC, DRS, Managing Difficult Emotions). All CwC and DRS sessions were observed in order to compare baseline and post-Toolkit adherence and competence ratings on these topics. Counselors were free to use the Toolkit after the second set of observations and before the third and final set of observations (of modules 1-4) which occurred approximately six months after the training.

2.3 Measures

2.3.1 Counselor Background Form

This measure collects descriptive information about counselors such as demographics, education, licensures/certifications, recovery status and exposure to RP content.

2.3.2 Group Observation Checklist

The Group Observation Checklist (GOC) is based substantially on the Yale Adherence and Competence Scale (Carroll et al., 2000; YACS) designed for use with ESTs (e.g., cognitive behavioral treatments). The GOC contains 10-12 specific content items for each of the six modules. The GOC assessed counselor adherence (i.e., frequency and extensiveness) to and competence when implementing each of the observed counseling behaviors. Both were scored on 7-point Likert scales, with 1=not at all and 7=extensively for adherence and 1=poor and 7=excellent for competence.

Eight coders were trained to live-code groups using a detailed GOC coding manual. Inter-coder agreement was monitored bi-monthly throughout the course of the study. Twenty-five percent of the sessions were randomly selected to be observed by two coders in order to monitor inter-coder agreement. The first author and all coders met to discuss dual observations and discuss coders' reasoning for score differentials on items. Inter-coder agreement ratings did not drop below the targeted 80% agreement rate during the study. See Brooks et al. (2012) for additional information on the GOC and coder training.

2.4 Analysis

(1) We conducted reliability analysis of group coders by calculating kappa values between primary and secondary coders. (2) We compared baseline and post-training mean item differences on the CwC and DRS modules using t-tests; due to the large number of related tests, we performed Holms' adjustments for multiple testing (Holm, 1979). Effect sizes were calculated using Cohen's d. (3) To assess CwC and DRS adherence and competence gains across all three time-points, we employed repeated measures ANOVAs with Bonferroni adjusted pairwise comparisons to determine where between time period differences were significant. (4) Counselor use of core relapse prevention skills (functional analysis, assigning and reviewing homework) across successive sessions was presented descriptively; weighted least squares analyses based on repeated measurements were conducted to identify differences in skill adherence and competence over time. (5) In order to determine if there were higher adherence and competence scores on the module on which counselors received direct training (i.e., My Addiction Pattern) in comparison to two on which counselors did not receive direct training (CwC, DRS), we conducted ANOVAs at Time 2 and Time 3.

3. Results

3.1 Participants

Seventeen counselors provided full baseline and post-training data. Just under half were female (47%); 29% were African-American, 59% were Caucasian, and none were of Hispanic origin. Seventy-one percent had earned a bachelor's degree, and 18% had a master's degree. Only one counselor (6%) was a certified addictions counselor. Thirty-one percent of the sample self-identified as being in recovery. Nearly half (44%) reported attending a continuing education RP workshop in the past two years, and the majority (77%) reported reading an RP manual in the past two years. With the exception of gender (X2(2)=8.04, p=.018), no statistically significant site by counselor characteristic differences were found.

3.2 Coder Reliability

Coder reliability was calculated on the double-coded groups for each of the 42 Coping with Craving (CwC) and Drug Refusal Skills (DRS) items (i.e., 21 frequency/adherence items; 21 competence items). A weighted kappa was calculated for the for the 7-point Likert scale frequency/adherence items (e.g., ordinal scale). An unweighted kappa was used for the competence items since they consisted of a mix of ordinal and nominal responses (i.e., a 7-point Likert scale with a “Not Done” response option). Also, the sample size of coder pairs was small, thus some items had little to no variation in the responses, with high levels of negative agreement between coders (i.e., both coders regularly agreeing that no target counselor behavior was present). When there is no variation (i.e., when all coders respond with identical values) the kappa value cannot be computed. Therefore, our kappa results include only the valid kappa values where variation is present in the responses. Kappa values less than .40 are conceptualized as poor, .40-.59 as fair, .60-.74 as good, and greater than .75 as excellent (Cicchetti & Sparrow, 1981). On average, we obtained good inter-coder reliability ratings for adherence [Mn kappa=.642; range= 1.0 (min=0, max=1); 3 of 21 items <.40] and fair reliability for competence ratings [Mn kappa=.475; range= 1.0 (min=0, max=1); 7 of 21 items <.40].

3.3 Within Session Baseline to Post-Training Content Changes

We examined which CwC and DRS content adherence areas showed the greatest improvements from baseline to post-training on an item-by-item basis (including mean, standard deviation, t-test results, and effect size estimates; Table 1). Specifically, an item level pre-post comparison of the 11 CwC items shows a statistically significant increase on four items: 1) teach/discuss/model the coping skill “play through the tape” (t=6.08, p<0.0001), 2) teach/discuss/model the coping skill “seeking support” (t=4.33, p=0.001), 3) teach/discuss/model the coping skill “practicing personal positive self-statements” (t=4.66, p<0.0001), and 4) teach/discuss/model the coping skill “delay using” (t=6.25, p<0.0001). An item level pre-post comparison of the 11 DRS items shows a statistically significant increase on three items: 1) define/describe difference between passive, aggressive, and assertive communication styles (t=6.45, p<0.0001), 2) teach/discuss/model the use of direct eye contact but also remain polite when doing so (t=4.66, p<0.0001), and 3) problem solving hypothetical risky situations (t=4.71, p<0.0001). Importantly, numerous other items demonstrated moderate to very large effect sizes, but the small sample size (n=17) and correction for multiple tests may have limited our ability to detect effects.

Table 1. Holms Corrected Comparisons of Item-Level Adherence Values for Coping with Craving and Drug Refusal Skills Modules.

Pre-Toolkit Mean (SD) Post-Toolkit Mean (SD) t p-value d
Coping w/ Craving Adherence
 Elicit Client Experiences 2.81 (1.4) 3.69 (1.3) 1.60 0.130 0.65
 Teach Cravings Normal, Predictable 2.19 (1.4) 2.88 (1.5) 1.74 0.102 0.48
 Teach Craving Types 2.62 (1.5) 2.88 (1.5) 0.57 0.580 0.17
 Teach Recognize Triggers 2.19 (1.2) 2.62 (1.6) 1.10 0.289 0.31
 Teach Avoid Triggers 1.69 (1.1) 2.19 (1.2) 1.26 0.228 0.43
 Identify Coping Strategies 1.50 (0.9) 2.19 (1.1) 2.20 0.044 0.70
 Apply Play through the Tape* 2.00 (1.2) 4.00 (0.8) 6.08 <.001 2.00
 Apply Seek Support** 2.12 (1.5) 3.75 (1.3) 4.33 0.001 1.16
 Apply Distraction 2.94 (1.8) 3.88 (0.8) 2.08 0.055 0.94
 Apply Coping Statement* 1.50 (0.8) 3.56 (1.4) 4.66 <.001 1.87
 Apply Delay Using* 1.31 (0.8) 3.44 (0.9) 6.25 <.001 2.85
Drug Refusal Skills Adherence
 Identify Risky Situations 3.00 (1.5) 2.31 (1.5) 1.43 0.173 0.46
 Avoid Risky Situations 1.94 (1.3) 1.69 (1.0) 0.50 0.621 0.22
 Refusing takes Practice 1.69 (1.1) 1.62 (1.3) 0.14 0.894 0.06
 Teach Refusal Styles* 1.62 (1.4) 4.12 (1.2) 6.46 <.001 1.92
 Eye Contact/Polite* 1.81 (1.1) 3.50 (1.3) 4.66 <.001 1.41
 Firm No/ “Shut the Door” 2.38 (1.4) 3.44 (1.3) 2.87 0.012 0.78
 Leave/Suggest Alternatives 1.81 (1.2) 2.00 (0.9) 0.44 0.669 0.18
 Hypothetical Risky Scenarios* 1.69 (1.4) 3.44 (1.1) 4.72 <.001 1.63
 Outcomes of Assertive Communication 1.06 (0.3) 1.75 (0.9) 2.91 0.011 1.15
 Role-Play 2.00 (1.3) 4.12 (1.8) 4.19 0.001 1.37
Open in a new tab

Note.

*

indicates p<.001;

**

indicates p=.001.

3.4 6-Month Follow-Up Counselor Adherence and Competence Ratings

Due to job changes and promotions, three of the counselors who completed post-training observations left their positions before their 6-month follow-up rating could be conducted. The following analysis includes only the 14 counselors for whom direct observation ratings at all three time points could be obtained. ANOVA models, with means and standard deviations for CwC and DRS adherence and competence ratings, are presented in Table 2.

Table 2. Repeated Measures Analysis of Variance for Coping with Craving and Drug Refusal Skills Adherence and Competence.

Description of Variable Baseline N=19 Post-training N=17 6-Month Follow-up N=14 F p-value
Coping with Craving
 Adherence 2.01 (0.73) 3.13 (0.64) 2.92 (0.53) 10.94 .002
 Competence 3.72 (0.62) 3.83 (0.64) 3.77 (0.53) .024 .977
Drug Refusal Skills
 Adherence 1.87 (0.61) 2.73 (0.60) 2.65 (0.62) 8.40 .005
 Competence 3.54 (0.90) 3.75 (0.53) 3.89 (0.67) 1.70 .228
Open in a new tab

Note. This table presents baseline, post-training, and 6-month follow-up values for all counselors who allowed group observations at each of the three time points. Also presented are the Repeated Measures Analysis of Variance tests for all counselors who completed group observations at all three time points (N=14 for Coping with Craving Adherence and Competence and Drug Refusal Skill Adherence; N=13 for Drug Refusal Skill Competence).

The ANOVA model for CwC adherence scores was significant (F(2,12)=10.94, p=0.002); Bonferroni adjusted pairwise comparisons showed significant differences between baseline and post-training (p<0.001; d=1.49) and between baseline and 6-month follow-up (p=0.007; d=1.43), but not between post-training and follow-up (p=1.00; d=−.36). The ANOVA model for CwC competence scores was not significant (F(2,12)=0.024, p=0.977).

The ANOVA models for DRS adherence and competence scores demonstrated a similar pattern. The ANOVA model for DRS adherence scores was significant (F(2,12)=8.40, p=0.005); Bonferroni comparisons showed significant differences between baseline and post-training (p=0.003; d=1.34) and between baseline and 6-month follow-up (p=0.026; d=1.27), but not between post-training and follow-up (p=0.735; d=−.13). The ANOVA model for DRS competence scores was not statistically significant (F(2,12)=1.70, p =0.228).

3.5 Post-training Toolkit Use

Eighty percent (12 of15) counselors reported that they voluntarily used Toolkit materials during the time between the post-training group observations and the 6-month follow-up observations. Of those counselors who used Toolkit materials, on average they were used in 6.58 sessions (SD=11.55, range=1-46). However, this wide range is largely due to one outlier counselor who used Toolkit materials in 46 groups between Post-training and Follow-up observations. When this counselor was excluded, the average number of sessions in which Toolkit materials was used was 3.00 sessions (SD=2.61, range=1-9).

3.6 Across Session Engagement of Core Counselor Skills

Counselors' use of three counseling skills (i.e., functional analysis, homework assignment, homework review) was assessed during the four group observations associated with the post-training and 6-month follow-up time periods (Table 3). These clinical behaviors were not coded at baseline, as the items were Toolkit-specific. We used a weighted least squares approach to model repeated measurements of these skills across modules for each time period using SAS's PROC CATMOD. For significant effects of session, we then used specific contrasts to determine which modules were significantly different from one another. Counselors conducted functional analyses during Module 1 at post-training (Time 2) and 6-month follow-up (Time 3) at relatively high rates (100% and 86%, respectively); however, these rates decreased across sessions, as only 14%-29% of the counselors continued to practice functional analyses repeatedly, with rates as low as 14% at the time of the 6-month follow-up, a statistically significant difference at each time point (Time 2, X2(3)=74.66, p<.0001; Time 3, X2(3)=39.45, p<.0001). Specific contrasts show that functional analysis was conducted more frequently in Module 1 than in each other module in Time 2 (X2(1)=34.99, p<.0001 for each module test), and in Time 3 (vs Module 2 X2(1)=14.00, p=.0002; vs Module 3 X2(1)=11.79, p=.0006; vs Module 4 (X2(1)=35.00, p<.0001). Counselor engagement with assigning and reviewing homework showed more frequent compliance across modules during both assessment time points, with generally half to two thirds of counselors at least attempting these behaviors in each module. During Time 2, there were no significant differences in rates of assigning homework across modules (X2(3)=2.90, p=.41); however there were significant differences during Time 3 (X2(3)=14.95, p=.0019). Specific contrasts show that homework assignment was done more frequently during Module 2 than any other module (vs Modules 1 and 3 X2(1)=5.60, p=.0180; vs Module 4 X2(1)=10.50, p=.0012). Finally, during both time periods, there were significant differences across modules for reviewing homework (Time 2 X2(2)=8.94, p=.0115; Time 3 X2(3)=35.13, p<.0001). Specific contrasts for Time 2 show that counselors reviewed homework more often in Module 2 than in Module 4 (X2(1)=7.78, p<.0053), while contrasts for Time 3 show that counselors reviewed homework less often in Module 1 than in any other module (vs Module 2 X2(1)=10.50, p=.0012; vs Module 3 X2(1)=25.20, p<.0001; vs Module 4 (X2(1)=18.67, p<.0001).

Table 3. Adherence (F/E) and Competence (CM) for the Core Counseling Skills of Functional Analysis, Homework Assignment, and Homework Review Across Successive Toolkit Modules at Post-Training and 6-Month Follow-up.

Session
1 2 3 4
My Addiction Pattern Coping w/ Craving Drug Refusal Skills Managing Difficult Emotions p-value
Conduct Functional Analysis
 Post-Training Frequency 100% 29% 29% 29% < .0001
Mean F/E (SD) 5.36 (.842) 2.07 (1.82) 1.71 (1.27) 1.86 (1.56) ---
Mean CM (SD) 3.86 (1.41) 3.50 (1.29) 3.75 (.957) 3.75 (2.22) ---
 6-Month Follow-up Frequency 86% 36% 29% 14% < .0001
Mean F/E (SD) 4.43 (1.91) 1.93 (1.54) 1.71 (1.33) 1.43 (1.09) ---
Mean CM (SD) 4.17 (1.75) 4.00 (1.00) 4.50 (1.29) 4.00 (.000) ---
Assign Homework
 Post-Training Frequency 79% 86% 64% 79% .41
Mean F/E (SD) 2.86 (1.23) 2.71 (.994) 2.57 (1.45) 2.79 (1.25) ---
Mean CM (SD) 3.91 (1.3) 2.83 (.937) 2.78 (.972) 3.18 (1.17) ---
 6-Month Follow-up Frequency 57% 86% 57% 43% .0019
Mean F/E (SD) 2.64 (1.60) 3.36 (1.28) 2.00 (1.18) 2.07 (1.39) ---
Mean CM (SD) 4.38 (.518) 3.50 (.905) 3.13 (1.36) 3.17 (.408) ---
Review Homework
 Post-Training Frequency N/A 86% 64% 50% .0115
Mean F/E (SD) N/A 3.07 (1.39) 2.36 (1.28) 2.21 (1.53) ---
Mean CM (SD) N/A 3.08 (.900) 3.00 (.866) 3.71 (1.50) ---
 6-Month Follow-up Frequency 7.0%* 50% 71% 64% < .0001
Mean F/E (SD) 1.07 (.267) 2.00 (1.24) 2.43 (1.16) 2.43 (1.40) ---
Mean CM (SD) 2.00 (N/A) 3.29 (1.25) 3.60 (.843) 3.44 (1.13) ---
Open in a new tab
*

This homework score of 7.0% represents one counselor out of 14 who was very active in using the Toolkit between the Post-Training and 6-Month Follow-up; this counselor had been regularly using the Scheduling homework assignment, and had assigned it in the session prior to Session 1 of the 6-Month Follow-up.

3.7 Comparative Adherence Module Effects due to Amount of Training Exposure

Because we trained counselors using only the first Toolkit module (My Addiction Pattern/MAP), we assessed counselor adherence and competence across modules to determine if there were differences between the training module and other observed modules. We conducted separate ANOVAs for post-training (Time 2) and 6-month follow-up (Time 3) on adherence and competence scores for the MAP, CwC, and DRS modules. There were no significant differences of module type for adherence (Time 2 F(2,48)=1.67, p=.198; Time 3 F(2,38)=1.58, p=.219) or competence (Time 2 F(2,48)=.022, p=.978; Time 3 F(2,38)=.586, p=.562).

4. Discussion

The results associated with this study suggest that after completing a brief training in the use of the RoadMAP Toolkit™, counselor content adherence on both Coping with Craving (CwC) and Drug Refusal (DRS) modules improved from baseline to post-training and these gains, while showing some attrition, were largely maintained at the 6-month follow-up. Counselors demonstrated consistent adherence gains on nearly all topic areas covered within the CwC and DRS modules and made their largest, statistically significant gains on active skill practice domains (item effect sizes ranging from 0.06 to 2.85, mean d = 0.99). There were no significant differences in post-training or follow-up adherence on modules which received direct training versus modules which counselors self-taught.

Our previous work (Knoblach et al., in review) demonstrated that counselors consistently self-reported that during CBT groups they rarely engaged in clinical techniques geared towards active skill practice (e.g., role play, homework assignment), and this self-report was consistent with our direct observation. Analysis of counselor engagement with two core Toolkit skills (functional analysis and assigning/reviewing homework) across repeated sessions of the Toolkit showed that counselors did incorporate these skills into their groups, with the majority attempting them at least once and a solid minority adopting them consistently across sessions, at much greater levels than they were self-reporting without the Toolkit (Knoblach et al., in review). This is notable given that coping-related homework completion by is associated not only with the development of more frequent and competent use of coping skills but also with decreases in drug use (Carroll et al., 2005; Gonzalez et al., 2006; Kiluk et al., 2010).

Studies which directly observe substance abuse counselor clinical EST proficiency at pre-training, post-training, and again in a follow-up from two to four months in length typically show some deterioration in adherence between post-training and follow-up. For example, average effect sizes of level of adherence deterioration between post-training and follow-up was moderate in size after self-study instruction (mean d = −0.57; Miller et al., 2004; Moyers et al., 2008; Martino et al., 2011; Sholomskas et al., 2005) and after didactic workshop training (mean d = −0.55; Baer et al., 2004; Miller et al., 2004; Moyers et al., 2008). Studies of counselor adherence featuring ongoing supervision, coaching, or feedback after workshop resulted in markedly decreased levels of deterioration between post-training and follow-up (mean d = −0.12; Miller et al., 2004; Moyers et al., 2008; Martino et al., 2011; Sholomskas et al., 2005). After attending a very brief training, and with no ongoing supervision or coaching, the guidance and structure provided by the RoadMAP Toolkit™ demonstrated similar advantages in reducing post-training deterioration (mean d = −0.24), demonstrating the same type of enduring gains typically achieved with ongoing training. This finding is important given that the Toolkit was designed to require minimal counselor preparation time. Counselors delivered very consistent performance on a series of groups that were conducted approximately five to six months apart. In addition, it appears that minimal training on the use of the Toolkit and a single relapse prevention module adequately prepares counselors to implement other modules with their group clients.

We detected no improvements in counselor competence, which was in the adequate-to-average range at baseline. As previously discussed (Brooks et al., 2012), we have concluded that while very brief training on the use of a multimedia toolkit is sufficient to significantly improve the amount of evidence-based content a counselor covers in group, more intensive training and supervision may be needed to improve counselor competence. The fact that counselors demonstrated significantly improved content adherence with only three direct hours of training and with no direct training on the modules on which they were assessed (CwC and DRS) may indicate that the modeling components featured in the Toolkit video helped them to deliver RP groups featuring more evidence-based content and active coping skills practice.

4.1 Strengths and Limitations

While this study has several strengths, including being conducted in the context of community treatment groups facilitated by counselors with real world clients and featuring reliable direct observation of counselor performance, the study also has several limitations. Chiefly, this initial pilot study does not include a control group, without which it is impossible to separate the effects of the Toolkit from practice effects and the effects on counselor performance from being observed. Additionally, we used direct observation, putting a research assistant directly in the room with counselor and clients, which may have pressured the counselor to over-perform (although this pressure also would have occurred during the baseline period). Somewhat related to this, coding recorded sessions rather than live sessions is likely to have provided a more accurate picture of counselor behaviors both in terms of extent to which the skills were implemented and the quality of those skills. Our inability to identify differences in counselor competence over time may be due to an actual lack of change or to limitations associated with the group measure, as suggested by the relatively poor coder reliability estimates for the competence items. Finally, this study demonstrates the effects of an adapted EST on counselor adherence, but without clinical outcomes, which will be needed to demonstrate that the toolkit approach brings actual differential benefits to clients over and above treatment-as-usual.

4.2 Conclusions

The treatment research field has demonstrated that intensive workshop training followed by ongoing supervision is sufficient for improving counselor performance. However, this approach is costly and out of reach for the majority of clinical providers. Carroll and colleagues called for focusing on particular, standardized frontline treatments in which all counselors should be trained (Carroll & Rounsaville, 2007), but also calibrating intensive training efforts based on counselor aptitude and ability (Carroll et al., 2010). A curriculum-based approach may be one of the more efficient strategies for training large numbers of counselors to deliver a standardized EST. Our work with the RoadMAP Toolkit™ may indicate that careful attention to curriculum development and presentation strategy can efficiently improve counselor adherence to an EST, with minimal drift at a 6-month follow-up. Investment in curriculum-based approaches that can demonstrate fidelity with minimal training and are easy for local supervisors to maintain is critically needed.

Acknowledgments

This research was made possible by an American Recovery and Reinvestment Act grant awarded by NIDA (5 R01 DA025034-02) and by a research grant awarded by NIAAA (5R01AA17867-3). The authors wish to acknowledge the research assistants who attended and coded numerous treatment groups: Deanna Ryder, Claire Nick, Graham DiGuiseppi, Tyler Case, and Brian Versek. The authors also acknowledge the contribution of Richard Rawson, Ph.D., and Kenneth M. Carpenter, Ph.D., who served as content experts for the RoadMAP Relapse Prevention Toolkit™.

This research has resulted in marketable group counseling curriculum Toolkits owned by the Treatment Research Institute (TRI), a not-for-profit research organization. TRI intends to market these Toolkits. The developers of the Toolkits, and the authors of this report, have no financial interest in or ownership of these products. However, many authors are employed by TRI.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Baer JS, Rosengren DB, Dunn CW, Wells EA, Ogle RL, Hartzler B. An evaluation of workshop training in motivational interviewing for addiction and mental health clinicians. Drug and Alcohol Dependence. 2004;73(1):99–106. doi: 10.1016/j.drugalcdep.2003.10.001. [DOI] [PubMed] [Google Scholar]
  2. Bandura A. Social learning theory. Englewood Cliffs, NJ: Prentice Hall; 1977. [Google Scholar]
  3. Beidas RS, Kendall PC. Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice. 2010;17(1):1–30. doi: 10.1111/j.1468-2850.2009.01187.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brooks AC, DiGuiseppi G, Laudet A, Rosenwasser B, Knoblach D, Carpenedo CM, Carise D, Kirby KC. Developing an evidence-based, multimedia group counseling curriculum toolkit. Journal of Substance Abuse Treatment. 2012;43(2):178–189. doi: 10.1016/j.jsat.2011.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Carise D, Brooks A, Alterman A, McLellan AT, Hoover V, Forman R. Implementing evidence-based practices in community treatment programs: Initial feasibility of a counselor “toolkit”. Substance Abuse. 2009;30(3):239–243. doi: 10.1080/08897070903041194. [DOI] [PubMed] [Google Scholar]
  6. Carroll KM. A cognitive-behavioral approach: Treating cocaine addiction. Vol. 1. Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration, Department of Health and Human Services; 1998. [Google Scholar]
  7. Carroll KM, Nich C, Ball SA. Practice makes progress? Homework assignments and outcome in treatment of cocaine dependence. Journal of Consulting and Clinical Psychology. 2005;73(4):749–755. doi: 10.1037/0022-006X.73.4.749. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Carroll KM, Nich C, Sifry RL, Nuro KF, Frankforter TL, Ball SA, Rounsaville BJ. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence. 2000;57(3):225–238. doi: 10.1016/s0376-8716(99)00049-6. [DOI] [PubMed] [Google Scholar]
  9. Carroll KM, Farentinos C, Ball S, Crits-Christoph P, Libby B, Morgenstein J, Woody G. MET meets the real world: design issues and clinical strategies in the Clinical Trials Network. Journal of Substance Abuse Treatment. 2002;23:73–80. doi: 10.1016/s0740-5472(02)00255-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Carroll KM, Rounsaville BJ. A vision of the next generation of behavioral therapies research in the addictions. Addiction. 2007;102(6):850–862. doi: 10.1111/j.1360-0443.2007.01798.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Carroll KM, Martino S, Rounsaville BJ. No train, no gain? Clinical Psychology: Science and Practice. 2010;17:36–40. [Google Scholar]
  12. Center for Substance Abuse Treatment. Counselor's Treatment Manual: Matrix Intensive Outpatient Treatment for People with Stimulant Disorders. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2006. DHHS Publication No (SMA)06-4152. [Google Scholar]
  13. Cicchetti DV, Sparrow SA. Developing criteria for establishing inter-rater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency. 1981;86(2):127–137. [PubMed] [Google Scholar]
  14. Eby LT, Burk H, Maher CP. How serious of a problem is staff turnover in substance abuse treatment? A longitudinal study of actual turnover rates. Journal of Substance Abuse Treatment. 2010;39:264–271. doi: 10.1016/j.jsat.2010.06.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Garner BR. Research on the diffusion of evidence-based treatments within substance abuse treatment: A systematic review. Journal of Substance Abuse Treatment. 2009;36:376–399. doi: 10.1016/j.jsat.2008.08.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gonzalez VM, Schmitz JM, DeLaune KA. The role of homework in cognitive-behavioral therapy for cocaine dependence. Journal of Consulting and Clinical Psychology. 2006;74(3):633–637. doi: 10.1037/0022-006X.74.3.633. [DOI] [PubMed] [Google Scholar]
  17. Hoffman JA, Landry MJ, Caudill BD. Living in Balance: Moving from a life of addiction to a life of recovery. Hazelden; Center City, MN: 2003. [Google Scholar]
  18. Holm S. A simple sequential rejective multiple test procedure. Scandinavian Journal of Statistics. 1979;6:65–70. [Google Scholar]
  19. Kerwin ME, Walker-Smith K, Kirby KC. Comparative analysis of state requirements for the training of substance abuse and mental health counselors. Journal of Substance Abuse Treatment. 2006;30(3):173–181. doi: 10.1016/j.jsat.2005.11.004. [DOI] [PubMed] [Google Scholar]
  20. Kiluk BD, Nich C, Babuscio T, Carroll KM. Quality versus quantity: Acquisition of coping skills following computerized cognitive-behavioral therapy for substance use disorders. Addiction. 2010;105:2120–2127. doi: 10.1111/j.1360-0443.2010.03076.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Knoblach DJ, Brooks AC, Nick C, Carpenedo CM, Benishek LA, Clements NT, Carise D, Kirby KC. What really happens in group CBT treatment for substance abuse? Self-report and direct observation of community-based group counselors In review. [Google Scholar]
  22. Morgenstern J, Longabaugh R. Cognitive-behavioral treatment for alcohol dependence: A review of evidence for its hypothesized mechanisms of action. Addiction. 2000;95:1475–1490. doi: 10.1046/j.1360-0443.2000.951014753.x. [DOI] [PubMed] [Google Scholar]
  23. Marlatt GA, Gordon JR. Relapse prevention: Maintenance strategies in the treatment of addictive behaviors. New York: Guilford Press; 1985. [Google Scholar]
  24. Martino S, Ball SA, Nich C, Canning-Ball M, Rounsaville BJ, Carroll KM. Teaching community program clinicians motivational interviewing using expert and train-the-trainer strategies. Addiction. 2010;106:428–441. doi: 10.1111/j.1360-0443.2010.03135.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Massatti RR, Sweeney HA, Panzano PC, Roth D. The de-adoption of innovative mental health practices (IMHP): Why organizations choose not to sustain an IMHP. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1-2):50–65. doi: 10.1007/s10488-007-0141-z. [DOI] [PubMed] [Google Scholar]
  26. Mayer RE. Multimedia learning. New York: Cambridge University Press; 2001. [Google Scholar]
  27. McGovern M, Drake RE, Merrens MR, Mueser KT, Brunette MF. Hazelden Co-Occurring Disorders Program Integrating Combined Therapies. Hazelden; 2008. [Google Scholar]
  28. McLellan AT, Carise D, Kleber HD. Can the national addiction treatment infrastructure support the public's demand for quality care? Journal of Substance Abuse Treatment. 2003;25(2):117–121. [PubMed] [Google Scholar]
  29. Miller WR, Mount KA. A small study of training in motivational interviewing: Does one workshop change clinician and client behavior? Behavioural and Cognitive Psychotherapy. 2001;29(4):457–471. [Google Scholar]
  30. Miller WR, Wilbourne PL. Mesa grande: A methodological analysis of clinical trials of treatment for alcohol use disorders. Addiction. 2002;97(3):265–277. doi: 10.1046/j.1360-0443.2002.00019.x. [DOI] [PubMed] [Google Scholar]
  31. Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A Randomized Trial of Methods to Help Clinicians Learn Motivational Interviewing. Journal of Consulting and Clinical Psychology. 2004;72(6):1050–1062. doi: 10.1037/0022-006X.72.6.1050. [DOI] [PubMed] [Google Scholar]
  32. Miller WR, Sorenson JL, Selzer JA, Brigham GS. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment. 2006;31:25–39. doi: 10.1016/j.jsat.2006.03.005. [DOI] [PubMed] [Google Scholar]
  33. Morgenstern J, Morgan TJ, McCrady BS, Keller DS, Carroll KM. Manual-guided cognitive-behavioral therapy training: A promising method for disseminating empirically supported substance abuse treatments to the practice community. Psychology of Addictive Behaviors. 2001;15(2):83–88. [PubMed] [Google Scholar]
  34. Monti PM, Kadden RM, Rohsenow DJ, Cooney NL, Abrams DB. Treating alcohol dependence: A coping skills training guide. 2nd. Vol. 196. New York: Guildford Press; 2002. [Google Scholar]
  35. Moyers TB, Manuel JK, Wilson PG, Hendrickson SML, Talcott W, Durand P. A randomized trial investigating training in motivational interviewing for behavioral health providers. Behavioural and Cognitive Psychotherapy. 2008;36:149–162. [Google Scholar]
  36. O'Donnell CL. Defining, Conceptualizing, and Measuring Fidelity of Implementation and Its Relationship to Outcomes in K-12 Curriculum Intervention Research. Review of Educational Research. 2008;78:33–84. [Google Scholar]
  37. Price RH, Burke AC, D'Aunno TA, Klingel DM, McCaughrin WC, Rafferty JA, Vaughn TE. Outpatient drug abuse treatment services, 1988: Results of a national survey. In: Pickens RW, Leukefield CG, Schuster CR, editors. Improving drug abuse treatment. Rockville, MD: National Institute on Drug Abuse; 1991. pp. 63–92. [PubMed] [Google Scholar]
  38. Rawson RA, Shoptaw SJ, Obert JL, McCann MJ, Hasson AL, Marinelli-Casey PJ, Brethren PR, Ling W. An intensive outpatient approach for cocaine abuse treatment: The Matrix model. Journal of Substance Abuse Treatment. 1995;12:117–127. doi: 10.1016/0740-5472(94)00080-b. [DOI] [PubMed] [Google Scholar]
  39. Santa Ana EJ, Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. What is usual about “treatment-as-usual”? Data from two multisite effectiveness trials. Journal of Substance Abuse Treatment. 2008;35(4):369–379. doi: 10.1016/j.jsat.2008.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Schoener EP, Madeja CL, Henderson MJ, Ondersma SJ, Janisse JJ. Effects of motivational interviewing training on mental health therapist behavior. Drug and Alcohol Dependence. 2006;82(3):269–275. doi: 10.1016/j.drugalcdep.2005.10.003. [DOI] [PubMed] [Google Scholar]
  41. Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don't train in vain: A dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 2005;73(1):106–115. doi: 10.1037/0022-006X.73.1.106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Smith JL, Amrhein PC, Brooks AC, Carpenter KM, Levin D, Schreiber EA, Nunes EV. Providing live supervision via teleconferencing improves acquisition of motivational interviewing skills after workshop attendance. The American Journal of Drug and Alcohol Abuse. 2007;33(1):163–168. doi: 10.1080/00952990601091150. [DOI] [PubMed] [Google Scholar]
  43. Walters ST, Matson SA, Baer JS, Ziedonis DM. Effectiveness of workshop training for psychosocial addiction treatments: A systematic review. Journal of Substance Abuse Treatment. 2005;29(4):283–293. doi: 10.1016/j.jsat.2005.08.006. [DOI] [PubMed] [Google Scholar]
  44. Weiss RD, Jaffe WB, de Menil VP, Cogley CB. Group therapy for substance use disorders: What do we know? Harvard Review of Psychiatry. 2004;12:339–350. doi: 10.1080/10673220490905723. [DOI] [PubMed] [Google Scholar]

RESOURCES