Developing an evidence-based, multimedia group counseling curriculum toolkit

Abstract

Training community-based addiction counselors in empirically supported treatments (ESTs) far exceeds the ever-decreasing resources of publicly funded treatment agencies. This feasibility study describes the development and pilot testing of a group counseling toolkit (an approach adapted from the education field) focused on relapse prevention (RP). When counselors (N = 17) used the RP toolkit after 3 hours of training, their content adherence scores on “coping with craving” and “drug refusal skills” showed significant improvement, as indicated by very large effect sizes (Cohen’s d = 1.49 and 1.34, respectively). Counselor skillfulness, in the “adequate-to-average” range at baseline, did not change. Although this feasibility study indicates some benefit to counselor EST acquisition, it is important to note that the impact of the curriculum on client outcomes is unknown. Because a majority of addiction treatment is delivered in group format, a multimedia curriculum approach may assist counselors in applying ESTs in the context of actual service delivery.

1. Introduction

Over the past few decades, a number of replicable, empirically supported treatment (ESTs) approaches have been identified to address substance use issues (Miller & Wilbourne, 2002). Unfortunately, most providers still do not use these ESTs (Garner, 2009; Massatti, Sweeney, Panzano & Roth, 2008; Miller, Sorenson, Selzer, & Brigham, 2006) because of current treatment provider budget restrictions, intervention complexity, and training burden, which is magnified by counselor and supervisor turnover (Gallon, Gabriel, & Knudsen, 2003; Knudsen, Ducharme, Roman, & Link, 2005; McLellan, Carise, & Kleber, 2003; Roman & Blum, 1997). Much counselor training consists of on-the-job experience under supervision of varying intensity and quality. Although the field is moving toward professionalization (Kerwin, Walker-Smith, & Kirby, 2006; Mulvey, Hubbard, & Hayashi, 2003), substance abuse counselors will continue for the foreseeable future to be educationally heterogeneous recovering and nonrecovering individuals with a broad range of training experiences and competencies (Kerwin et al., 2006). A study of counselor practice in treatment-as-usual (TAU) conducted in the context of the National Institute on Drug Abuse’s (NIDA’s) Clinical Trials Network demonstrated that EST components such as behavioral skills training occurred in only 3% of sessions, whereas discussion of problematic cognitions occurred in only 8% of sessions (Santa Ana et al., 2008). The use of 12-step facilitation (12SF) techniques was also correspondingly low. This paucity of EST practice in sessions is disturbing, given that “chatting” about unrelated topics occurred in more than half of the sessions (Martino, Ball, Nich, Frankforter, & Carroll, 2009; Santa Ana et al., 2008).

Rigorous studies using direct assessment of clinicians’ abilities indicate that the most widely used method of training counselors (continuing education workshops lasting from a few hours to a few days) is insufficient and results in counselor misperception that they are practicing an EST when they are not (Beidas & Kendall, 2010; Miller & Mount, 2001). On the other hand, workshops followed by coaching, feedback, or ongoing supervision result in significant changes in counselor practice (Baer et al., 2004; Carroll et al., 2002; Miller & Mount, 2001; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004; Morgenstern, Morgan, McCrady, Keller, & Carroll, 2001; Schoener et al., 2006; Sholomskas et al., 2005; Smith et al., 2007; Walters, Matson, Baer, & Ziedonis, 2005). Unfortunately, such intensive training can be expensive, and several large trials of dissemination in the community have yielded mixed results regarding the degree to which intensive training efforts actually result in improved client outcomes (Ball et al., 2007; Carroll et al., 2006; Morgenstern, Blanchard, Morgan, Labouvie, & Hayaki, 2001).

1.1. Searching for cost-effective training strategies

Reductions in resources for training and high counselor and supervisor turnover necessitating frequent retraining have left the field searching for more cost-effective training strategies. Some research groups have attempted to reduce training burden via Web-based and computerized training protocols for cognitive–behavioral therapy (CBT; Sholomskas et al., 2005; Weingardt, Cucciare, Bellotti, & Lai, 2009) and 12SF (Nowinski & Baker, 1998; Sholomskas & Carroll, 2006). Web-based approaches have typically outperformed the results attained from having counselors simply read a manual, although training results are generally not as strong as those derived from in-person workshops with supervision (Sholomskas & Carroll, 2006; Sholomskas et al., 2005; Weingardt et al., 2009). A second strategy is to use a “train-the-trainer” approach, whereby one or two local supervisory-level personnel are trained extensively in an approach, then provide ongoing supervision to bring local colleagues up to an appropriate level of proficiency (Carroll & Rounsaville, 2007; Martino et al., 2011). A third strategy, and one that our group has been exploring, has been to develop “toolkits” consisting of multimedia tools (videos, colorful posters) and brief, flexible counselor guides to facilitate effective use of evidence-based therapies. These materials allow counselors to learn and practice adapted ESTs with consumers without extensive prior training on the EST. The toolkit strategy enables trainers to introduce an EST to counselors, provides concrete tools to maintain its use, promotes active learning on the part of the counselor and client (Beidas & Kendall, 2010), and assists in ongoing supervision. Because the majority of clinical services are currently provided in a group format (Price et al., 1991; Stinchfield, Owen, & Winters, 1994; Weiss, Jaffe, de Menil, & Cogley, 2004), we have been exploring this approach for group counseling.

Our approach is informed by training strategies developed in the education field to foster the effective transfer of complex information and conceptual skills. Educators also contend with limited resources for training and supervision in the use of new materials or subjects. Teachers instructing students at numerous educational levels use flexible, interactive, multimedia lesson plans to assist students in acquiring and applying complex concepts through a series of carefully constructed sessions (curricula). Although these prepackaged curricula are common at the K-12 levels (O’Donnell, 2008), examples can also be found in medical and professional education (Wilkie, Judge, Wells, & Berkley, 2001). Our training strategy draws upon the learning-by-teaching literature, which is centered on the idea that teaching others is a powerful way to learn (Bargh & Schul, 1980; Biswas, Schwartz, Leelawong, Vye, & TAG-V, 2005; Chi, Siler, Jeong, Yamauchi, & Hausmann, 2001). Just as teachers and peer tutors have been shown to learn material more effectively while they prepare and then subsequently teach new material (Ketamo & Suominen, 2010; Leelawong & Biswas, 2008; Tsien & Tsui, 2007), we anticipate that counselors may learn ESTs more efficiently while preparing to teach them to clients. Learning-by-teaching strategies have already been built into curriculum approaches (Splett, Coleman, Maras, Gibson, & Ball, 2011).

Our development efforts also capitalize on the cognitive theory of multimedia learning (Mayer, 2001). Mayer’s theory postulates (a) that human learners process visual and auditory stimuli in separate channels, (b) that each channel has limited capacity at any given time, and (c) that human learners learn by actively engaging and selecting content across these different channels to assemble working conceptual models (Mayer, 2001; Mayer & Anderson, 1991). Consequently, presentation of material in both a visual and auditory format provides significant advantages to learning new material, as does offering this learning in stages (Mayer, Mathias, & Wetzell, 2002). Substance abuse interventions that largely rely on the transfer of didactic verbal information could benefit from similar multimedia presentations (e.g., videos, worksheets) to facilitate the transfer of EST content accurately and reliably from clinician to patients. Our group has developed two of these multimedia curriculum packages for group counselors.

Prior to developing our two toolkits, we conducted a feasibility trial using a one-session toolkit prototype on decisional balance, an activity often used in the context of motivational interviewing (MI; Miller & Rollnick, 2002); we field tested it among 26 experienced substance abuse counselors (all had more than 10 years working in the field) and 210 of their patients recruited in six community-based treatment programs (Carise et al., 2009). This prototype toolkit consisted of a DVD (televised during group to illustrate the clinical concept), a counselor guide (to provide guidance and talking points for the counselor during group), and one worksheet and wallet card for patient use. Every counselor (100%) and most patients (97%) were satisfied with the toolkit session, and 84% of patients said they would like to attend additional groups like the toolkit session. Almost all counselors (96%) were still using one or more toolkit components 3 months after their initial exposure without having received any further encouragement from the investigators. Based on these promising early results, we have followed a curriculum development approach to build toolkits for other ESTs, starting with relapse prevention (RP) and 12SF. The goals of this article are to (a) describe our process and final product in adapting RP strategies to a “curriculum toolkit” and (b) present preliminary data on the effectiveness of the toolkit for improving counselor adherence and skillfulness at delivering groups centered on RP concepts.

2. Methods

2.1. Selecting curricula approach and content

We based our teaching/training approach chiefly on Bandura’s (1977) theory of modeling and observed learning. Modeling is a driving strategy in our toolkit design. Because we were striving to facilitate adherent counselor practice with only minimal training time, we realized that we would have to fold in clear, easily accessed models of how to conduct the toolkit sessions and that it would be advantageous to have counselors access these models repetitively. We also felt that participating clients could benefit from these types of models. We had already decided to include brief video introductions into each toolkit session to communicate important didactic information, so we decided to have these video clips perform a dual purpose. The narrator and characters in the video, in addition to communicating relevant EST information, also serve as counselor and client models. These models, which are viewed right at the point of service delivery, contain numerous vignettes and examples of interaction around the therapy concepts in and out of the treatment setting.

We also centered our training approach on the use of learning-by-teaching to build mastery (Artzt & Armour-Thomas, 1999; Biswas et al., 2005; Chi et al., 2001); it has been demonstrated that students learn more and retain learned information longer when they are tasked with preparing to teach that material (Biswas et al., 2005; Biswas, Schwartz, & Bransford, 2001). Theoretically, a learning-by-teaching strategy (a) helps develop meaningful networks of knowledge, (b) increases the assumption of responsibility for future learning, and (c) fosters the development of reflection on teaching performance (Biswas et al., 2005).

Although our original toolkit prototype on decisional balance (Carise et al., 2009) consisted of a single session, we felt the approach would be better suited to assist counselors in psychosocial interventions requiring the communication of didactic content (CBT, 12SF) rather than trying to build toolkits for an intervention that depends more on the manner in which a clinician responds to varying levels of client-expressed ambivalence, such as MI. Based on this rationale and informed by our initial toolkit development, we set out to develop a curriculum of toolkits that would be instructive and clinically helpful to clients while remaining simple enough to continually draw counselors back to the series so that the toolkit would become a staple of their programming.

We selected cognitive–behavioral RP therapy (Carroll, 1998; Marlatt & Gordon, 1985) as one of the two evidence-based strategies to build into our first two curricula. RP approaches have been shown to be superior to no-treatment and attention controls in numerous trials (see review by Irvin, Bowers, Dunn, & Wang, 1999), and it has been tested in treating numerous substance use disorders: alcohol (Monti, Kadden, Rohsenow, Cooney, & Abrams, 2002), cocaine (Carroll, Nich, Ball, et al., 2000; Carroll, Nich, Sifry, et al., 2000; Maude-Griffin et al., 1998), marijuana (Carroll et al., 2006), and methamphetamine (Rawson et al., 2004). RP is also effective in group settings (Maude-Griffin et al., 1998; McKay et al., 1997; Monti et al., 2002), which is important because most treatments in the United States currently is delivered in group form. Cognitive–behavioral RP approaches are already currently available in well-designed group therapy manuals (Monti et al., 2002; Sobell & Sobell, 2011; Velasquez, Maurer, Crouch, & DiClemente, 2001) and in a patient-friendly curriculum manual (Matrix; CSAT, 2006), all of which have been tested in randomized controlled trials. Although these are excellent presentations of CBT, we had noted that many programs did not seem to be using these manualized approaches, although they were aware of the manuals and claimed that their programs delivered CBT. Therefore, we felt that a counselor-friendly introductory curriculum could lead to greater use of RP/CBT in practice and hopefully further enhance counselors’ use of the other existing manuals.

The usefulness of a toolkit depends in part on content selection: key elements of a given EST must be abstracted and presented clearly, concisely, and in a way that will be engaging for both the counselor and clients. To identify content, we reviewed treatment manuals in the public domain that presented RP/cognitive–behavioral approaches (Matrix; Carroll, 1998; CSAT, 2006). We selected module topics that (a) convey the core element of the practice, (b) center on behavioral activation on the part of clients to foster engagement in the curricula outside of group sessions (e.g., homework assignments), (c) involve similar processes that could be used across clinical situations (e.g., the application of the same set of coping skills to craving, emotions, seemingly harmless decisions), and (d) could be conveyed across multiple group sessions using a limited set of teaching and discussion techniques (e.g., video-based discussion questions, educational posters).

For an example of how the module topics were selected to meet these four principles, consider the RoadMAP Toolkit. We concluded that the core elements that clients need to be exposed to are (1) addiction is learned and coping skills can be learned, and therefore, (2) important behavioral activation elements include learning to master scheduling positive, drug-free activities. Consistent with principle (3), four of the six modules flexibly apply the same coping strategies to different types of situations. Finally, (4) all of the content in the six RoadMAP modules can be conveyed with the same types of materials and follow the same pattern and order of delivery.

These criteria required some difficult choices about content that could be omitted for the sake of simplicity while retaining core features of the evidence-based approach. We consulted with researchers with clinical backgrounds and familiarity with RP. Their critiques led to streamlining the material further than initially planned (down to five to six sessions), maximizing the use of concept repetition, and streamlining the curriculum for counselors, particularly by capitalizing on one homework assignment that could be repeated to build mastery for both counselors and clients. In addition, our consultants felt that more training than we initially intended to employ (3 hours) would be needed to prepare counselors to use the toolkit. Table 1 presents the teaching content and core clinical activities of the RoadMAP Toolkit.

Table 1.

Specific session-by-session content and core clinical skills of the RoadMAP RP toolkit

RoadMAP Toolkit

1) My addiction pattern—my alternative plan

• Analyze or “MAP” past using episodes

• Recognize basic triggers

• Schedule and follow through with drug-free activities

2) Coping with craving

• Learn basic characteristics of craving

• Apply five evidence-based coping skills to craving situations

3) Drug refusal skills

• Learn characteristics of effective refusal

• Discuss effective DR responses for hypothetical situations

• Practice DR skills through role play with constructive feedback

4) Managing difficult emotions

• Recognize and monitor emotions and states that can lead to using

• Apply Session 2 coping skills to managing difficult emotions

5) Seemingly harmless decisions

• Recognize decisions that seem harmless but may lead to high-risk    situations

• Create an “all-purpose coping plan” to anticipate consequences

6) Making recovery matter

• Discuss short- and long-term recovery goals

• Set a goal plan for recovery in motion

Central clinical activities

1) Teach patients to analyze typical using patterns

2) Teach patients to schedule their time and follow-through

3) Reinforce core theme of recognize, avoid, cope

4) Mediate group activities such as role-play and discussion

5) Teach and discuss coping strategies for high-risk situations

6) Elicit commitment to apply skills and reinforce practice outside of group

2.2. Consulting with toolkit end users

We conducted two focus groups, one consisting of practicing outpatient counselors and one of current clients and alumni. Focus group counselors were recruited by referral from treatment programs where we did not intend to test the toolkit curriculum. Focus group clients were recruited from area treatment programs in the Delaware Valley. We encouraged the clinical directors to invite clients who were currently abstinent and regularly attending treatment. Alumni and persons in long-term recovery were recruited through word of mouth. Each institutional review board-approved focus group was conducted by the lead author (A. C. B.) and lasted approximately 4 hours, including two scheduled 15-minute breaks. All focus group participants provided verbal consent and were paid $100 each. Both focus groups were audio recorded and transcribed by research assistants who omitted any personally identifying information.

During the counselor focus group, we elicited participants’ feedback on the structured curricula they may have used in the past (e.g., SAMHSA TIP series, MATRIX Model, Gorski workbooks), as well as counselor suggestions for the development of our toolkit materials. Counselors agreed that the video characters and situations were relatable for “high-functioning clients” but that the counselor character was too talkative and that clients would lose interest. Counselors also expressed concerns that too many concepts were being taught at once and suggested slowing the pace to avoid information overload. After reviewing our prototype materials, the participants in the client focus groups reacted favorably to the toolkit materials but suggested that the situations and characters in the instructional video be more “gritty” and “harsh” (showing clients at rock bottom) to relate to the typical client.

2.3. Content presentation strategies: curricula series tools

Our chief goals in designing the toolkit materials for counselors to use during sessions were to (a) present as much of the core content as possible in redundant ways that did not depend on counselor adherence or delivery, (b) provide flexibility for counselors, (c) present concepts attractively using professional quality visuals and formatting, and (d) assume little to no training (the “out-of-the-box” test). We designed the toolkit curricula around interactive and attractive multimedia materials that could provide sufficient guidance to counselors with minimal training. Toolkit materials are described in the following sections.

2.3.1. Introductory videos

We scripted and produced video segments to be viewed in each session. The segments run 10 to 18 minutes each and depict a group of clients in various stages of change (Prochaska & DiClemente, 1992) who are working with the toolkit concepts to achieve abstinence. The RoadMAP video series also includes an ideal counselor who is knowledgeable, warm, empathic, and supportive of client autonomy. Thus, video characters are meant to become role models for how to interact around the material for both clients and counselors. The video series are narrative in style, with an ongoing story line of a couple central characters struggling with recovery, and incorporate humor and mild suspense (cliffhanger endings) intended to draw clients into the presentation. Each video clip presents all of the core concepts necessary for clients to acquire the specific content presented in that module (e.g., coping with craving [CwC]), leaving the counselor free to actively process the material.

2.3.2. Posters, worksheets, and recovery cards

Each session incorporates (a) a colorful interactive poster designed to reinforce the video content and (b) one to two client worksheets that encourage repeatedly returning to and interacting with the content. Worksheets replicate the posters but are blank in key areas for clients to individualize their response to the content. For example, in Session 1, we introduced an important poster meant to recur throughout the series (see Fig. 1). This “Use MAP” poster (used in conjunction with the Session 1 video showing the model counselor using the same poster) is meant to assist the counselor in leading clients through a simplified, flexible functional analysis.

Fig. 1.

Functional analysis activity redesigned into a dry-erase poster for use in group.

The Use MAP poster and worksheet guide the counselor through analyzing substance use or near-use behavior by identifying the triggers, the behavior itself, and its consequences. Counselors are encouraged to complete the Use MAP poster for situations depicted in the instructional videos and in other client-generated situations. Accompanying worksheets serve to further illustrate a topic being covered and offer counselors an additional way to involve clients in take-home activities. Additionally, recovery cards were designed to serve as reminders of key points and useful strategies; by putting these central points on 4 × 6-in. cards, they are accessible and small enough to carry in pockets, wallets, purses, or backpacks (Fig. 2).

Fig. 2.

Client recovery card (4 × 6) that can be personalized and contains basic concepts from each session.

We consider the posters and worksheets to be the core of the toolkits; counselor focus group participants reported that often, when using curricula, they may not read the accompanying manual but instead go straight to the worksheets and start leading groups. Therefore, we designed the toolkits to promote accurate, evidence-based information transfer even if only the posters and worksheets are used.

The RoadMAP Toolkit stresses the importance of client behavioral activation between sessions and encourages repeating a scheduling activity between sessions in an effort to build client mastery. The toolkit contains numerous prompts reminding counselors that client activation is the most important aspect of the series and builds in time for following up on activities outside of group.

2.3.3. Supporting counselor tools and training

The toolkits were designed to be used “out of the box,” with very little training. When implementing toolkits, we provide a 3-hour training that is primarily geared toward orienting counselors to the toolkit components and introducing them to two to three primary clinical behaviors (i.e., assigning and following up on behavioral activation) that they will use repetitively throughout the series. The training does not take counselors through all the content; rather, we emphasize that the material can be self-taught on a session-by-session basis. We developed quick-prep walkthroughs of each module including a “Quick Background” and a step-by-step “Clinician’s Guide.” The Quick Background provides a brief overview of the key principles presented in each module, in-depth instructions for using posters and other materials, and further topics for discussion. The Clinician’s Guide is a two-page outline of each session. The illustrated guide uses colorful prompts and pictures to guide counselors through the activities, with a suggested time frame to deliver the intended content. Throughout the session, suggested wordings are offered to facilitate client engagement of materials and activities and promote discussion. Finally, to supplement these materials, we also provide counselors with a brief handbook that offers a general overview of the toolkit curriculum, rationale for using evidence-based practices, basic and essential counselor skills, an in-depth background of each module and the information provided in each of the laminated Quick Background sheets.

2.4. Research evaluation of the RoadMAP toolkit

2.4.1. Participants

Outpatient counselors consented to allow research staff (with concurring client consent) to sit-in on their group sessions and live-code sets of four groups across two periods: before receiving the toolkit (baseline) and 1 month after the toolkit training. Counselors from seven treatment sites at three agencies in the Philadelphia and southern New Jersey area provided informed consent to participate. The 17 participating counselors who completed pre and post-training observations were 47% female; 29% African American, and 59% Caucasian. None were of Hispanic ethnicity. More than half (71%) had bachelor’s degrees, and 18% had master’s degrees. Only one counselor (6%) was a Certified Addictions Counselor. In terms of professional training, 44% reported attending a continuing education RP workshop in the past 2 years, and 77% reported reading a manual on RP in the past 2 years; 31% of the sample self-identified as being in recovery.

2.4.2. Measures

2.4.2.1. Weekly group calendar

We developed a Web-based data entry system that counselors logged in weekly to provide self-report data on the topics and duration of groups they conducted (overall duration and duration of focus on each topic). Counselors also reported how often they used any educational/or clinical videos or worksheets (not limited to our study materials) and specified what they used. This weekly calendar was administered for 12 weeks prior to the toolkit and training and for 24 weeks posttraining. A research assistant checked counselor-entered content for consistency, regularly reminded counselors to use the system, and followed up with counselors to ask about questionable or confusing entries. This occasionally resulted in the need to recode a counselor’s entry if they misclassified a type of group (i.e., they conducted an RP group but labeled it a process group).

2.4.2.2. Group observation rating system

The most reliable and well-used instrument for examining the integrity, adherence, and skillfulness of CBT/RP delivery is the Yale Adherence and Competence Scale (YACS; Carroll, Nich, Ball, et al., 2000; Carroll, Nich, Sifry, et al., 2000), which has been used in numerous clinical trials. The YACS includes well-anchored scales to rate clinical adherence (frequency and extensiveness of clinical behavior) and skillfulness of clinician behavior on well-specified cognitive–behavioral constructs. However, the YACS has not been validated for evaluating cognitive–behavioral treatment in a group setting. In addition, the YACS assesses the degree to which a counselor engages in specific behaviors consistent with CBT (assigning and reviewing homework, teaching coping skills, and conducting functional analyses) but is less content specific (i.e., did the counselor discuss positives and negatives of avoiding high-risk situations). We adopted the approach used in Sholomskas et al. (2005) and added 10–12 specific content items for six session types: (a) teaching triggers and functional analysis, (b) CwC, (c) drug refusal (DR) skills, (d) managing difficult emotions, (e) seemingly harmless decisions, and (f) finding meaning in recovery. Each specific content item begins with a broad stem, allowing counselors significant latitude for how they choose to cover a content area in group: “To what extent did the counselor teach/process/model/discuss (insert specific topic)?” Of interest for this report, we added 11 specific content items appropriate for a CwC group and 10 specific content items for a DR skills group. Coders assessed adherence using a modified frequency/extensiveness rating system adapted from the YACS. The anchors on the adherence scale run from 1 (not at all) to 4 (somewhat), to 7 (extensively). To code counselor adherent behaviors as they typically manifest in group, we modified the anchors to reflect both the amount of time and counselor effort on a topic, as well as the extent to which group participation and feedback on a content area were incorporated. When coders rated a content item as present, they also assessed skillfulness using a modified YACS skillfulness scale, which also runs on a 1–7 scale (1 = very poor, 4 = adequate, 7 = excellent). Verbal anchors from the YACS skillfulness scale were expanded to incorporate appropriateness of counselor pacing, level of “commitment” to the topic, efforts at translating topics in patient language, and skill at using discussion to advance understanding.

Eight research staff were trained and certified to code live group sessions. One quarter of group observations at baseline and posttraining were coded by two observers to assess interrater reliability. We obtained “good” interrater reliability for adherence (κ = 0.642) and “fair” reliability for skillfulness ratings (κ = 0.475) based on broadly accepted standards (Cicchetti & Sparrow, 1981).

2.4.2.3. Toolkit satisfaction form

Counselors completed a brief satisfaction instrument on ease and utility of the toolkit after completing their posttraining observations. Counselors responded to statements about the curriculum and indicated whether they agreed or disagreed and also the strength of their opinion (strongly or only somewhat).

2.4.2.4. Procedures

For each of our participating counselors, we recruited all clients in one of their specified groups; clients gave consent for an observer to attend groups to code counselor performance. During the baseline period, research staff asked to attend two groups where the counselors would be using their typical approach with any tools they normally used. Staff members coded these “TAU groups.” One week before the third and fourth observations, staff members informed counselors that they would like to observe a group in which the counselor did their best, using whatever tools or resources they wished, to teach clients “how to deal with cravings and urges” (Topic 1) and “how to handle unwanted offers of drugs or alcohol” (Topic 2). Research staff members were thus able to code four groups, two TAU groups and two groups, in which clinicians demonstrated their best efforts on two popular, common RP topics (CwC and DR) for which multiple resources exist to prepare group sessions. Following baseline observations, counselors completed 12 weekly calendar checklists to establish a baseline of types of groups they typically led.

At Month 4, we provided a 3-hour training and orientation in the toolkit components. To minimize the possibility of contamination between periods, we waited to conduct the toolkit training until all counselors at a participating agency had completed the baseline observations. Half of the trainings were conducted by two of the authors (A. C. B. and D. K.), and half of the trainings were conducted by D. K. alone. During the training, approximately 30 minutes was spent reviewing the research on the benefits of RP as an EST, and an additional 30 minutes was spent orienting counselors to the different elements of the toolkit (e.g., posters, videos) During the final 2 hours, counselors were walked through Session 1 (My Addiction Patterns–My Alternative Plan), with emphasis on how each toolkit component worked together to teach this one session. Counselors were given some brief practice during this walk-through on how to conduct a functional analysis and how to use the recurring homework assignment. It is important to note that counselors received no direct training or instruction on how to conduct the rest of the sessions, including CwC and DR skills. Rather, the counselors were told that once they understood how one session worked, they would be able to teach themselves the rest of the curriculum as they prepared ahead of each session. In addition, counselors were given no additional training, supervision, or tips throughout the rest of the study.

Counselors were given 2 weeks to familiarize themselves with the toolkit components and told to call research staff with any questions. At Month 1, we conducted four weekly group observations of the first four sessions of the toolkit (for content, see Table 1) to compare adherence and skill ratings from the counselors’ baseline efforts at CwC and DR skills to their performance when equipped with our toolkit components and brief training.

3. Results

During the 3-month baseline observation period, counselors ran an average of four groups per week (M = 4.2, SD = 2.06) and spent approximately half an hour preparing to run a group (M = 25.35 minutes, SD = 19.13). Of all reported groups, counselors most often ran coping skills training groups (23.7%), interpersonal process groups (14.6%), anger management groups (9.5%), psychoeducational groups (12.9%), and 12SF groups (6.1%). Counselors reporting running coping skills groups used educational/teaching aids in 71% of groups (M = 1.09 materials), and during 12-step groups, they used an educational aid in 71% of groups (if reporting that they used an aid, they uniformly reported using only one aid in 12-step groups). During the 3 months postbaseline observation period, counselors ran an average of four groups per week (M = 4.05, SD = 2.16), and the amount of time they took preparing for groups did not increase (M = 26.47 minutes, SD = 17.03). Types of groups were fairly similar in frequency to the baseline period. Notably, when counselors reported running coping skills groups, they increased their use of educational/teaching aids to 91% of groups (M = 2.39 materials), likely reflecting their use of the toolkit materials.

3.1. Changes in RP topic adherence and skill

Prior to the 3-hour toolkit training, counselors covered few CwC and DR topics (M = 5.76 of 11 possible topics and M = 4.00 of 10 possible, respectively) and obtained very low-frequency/adherence ratings (M = 2.12 and M = 1.89, respectively). On topics counselors did cover, skillfulness ratings were typically in the “adequate” to “average” range (CwC M = 3.74 and DR M = 3.53, respectively). After the toolkit training, counselors significantly improved on frequency/adherence scores on their first attempt to conduct both types of groups (CwC t = 4.69, p b .000; DR t = 3.85, p = .001), representing very large effect sizes (CwC d = 1.49 and DR d = 1.34). The number of topics covered for both group sessions significantly increased (CwC t = 3.98, p = .001; DR t = 3.16, p = .006); this represents a very large effect (CwC d = 1.50 and DR d = 1.21). Average skill rating for CwC and DR groups did not change (see Table 2).

Table 2.

Paired t-test comparisons of counselor adherence and skill pre- and posttraining in the use of the RoadMAP RP toolkit (N = 17)

	Pre toolkit		Post toolkit
Variable	M	SD	M	SD	t	p	Cohen’s d
CwC
Mean frequency count of covered content topics (11 possible)	5.76	2.84	9.18	1.51	3.89	.001	1.51
Mean adherence ratings	2.12	0.71	3.13	0.64	4.69	.000	1.49
Mean skill ratings	3.74	0.63	3.83	0.64	0.67	.511	0.14
DR skills
Mean frequency count of covered content topics (10 possible)	4.00	2.47	6.59	1.73	3.16	.006	1.21
Mean adherence ratings	1.89	0.65	2.73	0.60	3.85	.001	1.34
Mean skill ratings*	3.53	0.93	3.75	0.53	0.93	.367	0.29

^*Note. N = 16. One counselor did not engage in any DR skills behaviors during the pre-toolkit phase, and therefore was not given a skill rating during that phase; this counselor was dropped from the DRS skills analysis.

3.2. Counselor perception of the RP toolkit

Counselors completed a brief satisfaction measure after using the RoadMAP Toolkit. Overall, they reported high levels of satisfaction with the quality of the Toolkit groups they ran (Table 3); they were also satisfied with the curriculum’s ease of use.

Table 3.

Counselor ratings of satisfaction with ease of use and quality of groups resulting from RoadMAP RP toolkit

Variable	Strongly agree	Agree	Somewhat agree	Somewhat disagree	Disagree	Strongly disagree
Ease of toolkit use
Most counselors would learn to use the RP toolkit quickly.	23.5%	58.8%	11.8%	5.9%	0.0%	0.0%
I needed to spend a lot of time learning the RP toolkit materials before  I could effectively use them in group.	5.9%	11.8%	11.8%	23.5%	41.2%	5.9%
Overall, compared to the group materials I have previously used,  I am satisfied with how easy it was to use the RP toolkit materials.	35.3%	47.1%	5.9%	11.8%	0.0%	0.0%
Quality of toolkit groups
The RP toolkit materials met my needs when leading groups.	41.2%	35.3%	23.5%	0.0%	0.0%	0.0%
The RP toolkit materials helped me to lead successful RP groups.	35.3%	35.3%	29.4%	0.0%	0.0%	0.0%
Overall, the RP toolkit materials helped me conduct better RP groups.	41.2%	35.3%	17.6%	5.9%	0.0%	0.0%
I felt more confident leading groups after using the RP toolkit materials.	5.9%	52.9%	23.5%	11.8%	5.9%	0.0%

4. Discussion

This report describes our efforts to develop and test multimedia curricula (toolkits) that break down ESTs such as RP and 12SF into discrete content modules requiring only brief training. Given the paucity of research on curriculum approaches, our initial feasibility outcomes are encouraging. Using the strategy of carefully selecting a limited number of module topics and using professional quality visuals and formatting that assume little to no training appears promising. After only 3 hours of training with no follow-up supervision, counselors greatly increased their content adherence in delivering two common RP topics. The effects for this change were very large and are comparable to size of effect of adherence gains to other training studies using more intensive strategies. For example, in a pre–post training study among community-based counselors to practice CBT, Sholomskas et al. (2005) assessed pre–post training gains on clinicians using standardized patients to assess skill level conducting three CBT modules; they found that clinicians who (a) read the manual for 10 hours and (b) clinicians who read the manual and participated in an additional 15 hours of Web-based training demonstrated large effect improvements in adherence (average d = 0.76). It should be noted that clinicians who read the manual, attended a 3-day workshop, and received ongoing supervision demonstrated very large gains in adherence (average d = 1.37), likely attributable to the very intensive personal instruction and feedback.

Our study results did not demonstrate commensurate gains in counselor skillfulness, which averaged in the adequate-to-average range at baseline. Modifications we made to the YACS skillfulness rating scale to capture group therapy behavior may have obscured any changes here. Coders rated counselors in our sample as already fairly close to an average level of skillfulness at baseline, indicating that counselors were generally engaging their group members on topic content and explaining and processing it adequately. It is possible that our modifications to the YACS may not have been sensitive enough to detect the skillfulness changes resulting from using the toolkit.

Although it is possible that measurement error contributed to our not detecting any skillfulness effects, we actually believe that the scale was adequate and that skill changes did not result from the use of the toolkit. A more likely interpretation of the lack of skill effects is that although a curriculum toolkit and brief training is adequate to quickly improve the level of EST content adherence in community group sessions, more intensive training and supervision may also be needed to improve counselor skillfulness. This is a reasonable interpretation, given that counselors entered the study at close to an adequate level of skill on the few CwC and DR topic areas they did cover. The introduction of a significant amount of new, evidence-based content gave these counselors guidance on how to become more adherent with their session time but did not change their basic skill level; in this case, a minimal-training intervention such as the toolkit is a valuable addition, as it improves the content quality of an average group and exposes clients to more evidence-based content.

In addition, it should be noted that these counselors received no more than three direct hours of training and did not receive any direct training on the topics on which they were assessed. The provision of feedback, supervision, and ongoing coaching was deliberately withheld. The fact that skillfulness did not change does indicate that some additional supervision and feedback would probably be necessary for this variable to shift in a positive direction. We would not want to give the impression that introducing curriculum toolkits into treatment replaces the need for any further training or supervision. However, these results do indicate that this training approach is a promising strategy to equip counselors already functioning at an adequate skill level and can also serve as an excellent starting point for introducing new counselors to using RP/CBT content in group, exposing them to core concepts.

We are encouraged by the results of this study, as the findings indicate that careful attention to the instructional design of dissemination materials may contribute to more efficient workforce training gains. In particular, we capitalized on the learning strategies inherent in the learning-by-teaching literature (Bargh & Schul, 1980; Biswas et al., 2005; Chi et al., 2001), as well as principles central to theories of multimedia learning (Mayer, 2001). The learning-by-teaching literature has consistently demonstrated that preparing content to present in teaching and tutoring results in larger learning gains on the part of the teacher (Biswas et al., 2005); a curriculum-based approach places teaching demands on the counselor and may result in increased counselor learning of the topic material, reducing the need for as much training time. In addition, our multimedia format provided several different avenues for the counselor to acquire the necessary comfort level with the content to then work with them in session, consistent with the cognitive theory of multimedia learning (Mayer, 2001). Finally, we believe that our strategy capitalizes effectively on modeling (Bandura, 1977). The video clip played at the beginning of each session effectively communicates information about the teaching content to the group. However, the video also shows an effective, knowledgeable counselor and an interested group interacting around the session content, providing accessible models in a comparable learning environment.

4.1. Strengths and limitations

This study has several strengths, including being conducted in real-world community settings with practicing counselors and their clients and using reliable coding of groups, which provides some certainty that use of the toolkit can markedly improve counselor delivery of an EST (in this case, RP). In addition, this study provides data from which to estimate the effectiveness of a curriculum-based approach in the absence of any follow-up supervision. The fact that these adherence gains were made with very minimal training and no follow-up feedback or supervision is noteworthy.

This design also has several limitations. First, in the absence of a control group, counselors may have improved anyway simply from practicing the relevant RP topics. Further research requires a randomized controlled trial, which we are currently conducting with NIAAA funding for our 12-Step Open Doors Toolkit curriculum. Second, we used live coders rather than audio recording the sessions; counselors may have felt more pressure to perform in front of a live observer, resulting in an inflated effect size. However, this inflationary pressure would have been felt by counselors at both pre- and posttraining assessments.

More importantly, it is valuable to note that this study indicates that multimedia presentation of an adaptation of an EST is helpful to improve counselor adherence. In this case, the EST was adapted to simplify its presentation. In addition, the fidelity measure used was adapted to capture these changes. This is a significant distinction because without specific client substance outcomes, this feasibility study can only be considered a first step in establishing its utility. Although it is reasonable to assume that this presentation of evidence-based CBT content will result in similar outcomes to other evidence-based CBT approaches, this preliminary report does not include any client clinical outcomes, which are needed to demonstrate that the toolkit approach brings actual differential benefit to clients over and above TAU. In addition, we adapted well-established ESTs, which are 5–15 years old; newer research on EST components may call various chosen elements into question.

4.2. Future research directions

Based on these findings, there are several directions that our team are pursuing related to these multimedia curricula. There is a need for randomized controlled trials exploring the impact of these curriculum toolkits on counselor adherence and skill relative to a TAU control group; furthermore, it is important to establish that exposure to the toolkit content results in meaningful gains in client outcomes. We currently have one fully powered randomized controlled trial underway testing the impact of our Open Doors 12SF Toolkit on counselor and client outcomes. Other controlled trials are needed comparing the differential efficacy and cost-effectiveness of these briefer multimedia presentations relative to the full, standard ESTs from which they were drawn. The added benefits of supervision and corrective feedback to the minimal training we provided in this study should also be examined. Cost-effective, local supervision using a “train-the-trainer” approach (Martino et al., 2011) may result in counselor adherence and skillfulness gains. We have adapted our research observational tool into a brief supervision checklist that local supervisors can use for just this purpose. Finally, we have begun developing a Web-based, multimedia presentation of the RoadMAP Toolkit that brings the content directly to the client in an effort to capitalize on some of the benefits of computerized instruction (Bickel, Christensen, & Marsch, 2011). We are interested in seeing whether a counselor-delivered approach in a group setting provided in tandem with an ancillary computerized presentation conducted at the client’s own pace might result in improved client outcomes.

4.3. Conclusion

Curriculum-based approaches for group counseling are not new. The MATRIX Model (Rawson et al., 1995) and Living in Balance (Hoffman, Landry, & Caudill, 2003) are well known examples of these and in fact are often purchased for use by treatment programs. These excellent curricula are comprehensive and client friendly. In particular, the MATRIX Model has some evidence for its efficacy in the treatment of stimulant disorders (Rawson et al., 2004; Shoptaw, Rawson, McCann, & Obert, 1994), and the core curricula for group sessions is fairly intuitive. Living in Balance offers excellent exercises and information, although it presents with less compelling evidence that it is more effective than TAU (Hoffman et al., 1994, 2003). Exemplary group curricula such as these approaches lack the history of rigorous evaluation that individual-focused ESTs have been afforded, and these evaluations typically do not provide data on whether the curricula leads the counselor to change their group practice behavior or to present the curricula with adherence. Given recent arguments that it may be important to calibrate training efforts based on counselor ability and potential (Carroll, Martino, & Rounsaville, 2010) and to focus on particular frontline ESTs as a “go-to” treatment in which most counselors should be trained (Carroll & Rounsaville, 2007), it may be worthwhile to revisit a curriculum-informed approach. Training needs are likely to expand with the integration of addiction treatment into primary care that will result from health care reform, even as funding for training remains limited. Because much of community-based treatment is conducted in group, investing in curriculum-based approaches that can demonstrate fidelity with minimal training and are for local supervisors to maintain are critically needed.