A Large-scale Dissemination and Implementation Model for Evidence-based Treatment and Continuing Care

Abstract

Multiple evidence-based treatments for adolescents with substance use disorders are available; however, the diffusion of these treatments in practice remains minimal. A dissemination and implementation model incorporating research-based training components for simultaneous implementation across 33 dispersed sites and over 200 clinical staff is described. Key elements for the diffusion of the Adolescent Community Reinforcement Approach and Assertive Continuing Care were: (a) three years of funding to support local implementation; (b) comprehensive training, including a 3.5 day workshop, bi-weekly coaching calls, and ongoing performance feedback facilitated by a web tool; (c) a clinician certification process; (d) a supervisor certification process to promote long-term sustainability; and (e) random fidelity reviews after certification. Process data are summarized for 167 clinicians and 64 supervisors.

A number of experiments have identified effective treatment models for adolescent substance abuse (Brown, D’Amico, McCarthy, & Tapert, 2001; Dennis et al., 2004; Godley, Godley, Dennis, Funk, & Passetti, 2007; Henggeler et al., 2002; Liddle et al., 2001, 2004; Muck et al., 2001; Waldron & Turner, 2008; Williams & Chang, 2000). Despite the development of several evidence-based treatments (EBTs), the diffusion of these EBTs into practice remains minimal (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Garner, 2009; Gotham, 2004; Lamb, Greenlick, & McCarty, 1998; Miller, Sorensen, Selzer, & Brigham, 2006). Gotham (2004) has proposed a diffusion model that includes three overlapping components, including the development of effective treatments, dissemination (spreading information about EBTs so that organizations and clinicians can adopt them), and implementation (after adoption, EBTs are delivered in ways that maintain their quality and effectiveness). Thus, to improve diffusion, effective and feasible dissemination and implementation strategies must be identified.

Treatment for substance use disorders is primarily funded through public funding (http://www.oas.samhsa.gov/2k6/pay/pay.htm), and there are limited private sources to support the dissemination and implementation of EBTs. Brown and Flynn (2002) argued that the federal government is the best resource for diffusing EBTs because only it has the capacity to marshal the resources necessary for successful technology transfer. To date, there has not been a description or quantitative evaluation of a large federally sponsored technology transfer of a psychosocial EBT for substance use disorders across multiple treatment provider organizations.

There is, however, a growing body of research literature specific to diffusion of EBTs in substance abuse treatment. A number of studies have evaluated attitudes toward EBTs, adoption, and implementation (Garner, 2009), and many of these studies focus on the adoption of pharmacological EBTs (e.g., Roman & Johnson, 2002). Most relevant to this article are studies related to the implementation of psychosocial EBTs, which require the training of bachelor and master level professionals, who make up the majority of professionals within the substance abuse treatment system (Mulvey, Hubbard, & Hayashi, 2003). Experiments have provided insights into strategies for helping clinicians achieve competence in EBTs; however, less well understood are strategies for promoting EBT implementation within organizations and especially across multiple organizations simultaneously.

Research evaluating clinician only training strategies

Miller, Yahne, Moyers, Martinez, and Pirritano (2004) randomly assigned 140 substance abuse professionals willing to travel to a Motivational Interviewing workshop to one of five training conditions (workshop only, workshop and feedback on practice tapes, workshop plus coaching, workshop plus coaching and feedback, and a self-training control). Coaching and feedback increased proficiency after training. Sholomskas et al. (2005) assigned 78 community-based volunteer clinicians to three conditions (workshop only, web-based training, or workshop plus supervision) and found the most effective training method included a workshop seminar and supervision. These studies suggest that workshops alone are not enough to change a clinician’s behavior in session, but that supervision and/or feedback on his/her implementation is needed to increase fidelity to interventions. Clinicians in these studies volunteered to participate and thus were probably motivated to learn the EBTs in which they were receiving training.

Research evaluating clinician training within employer organizations

Most clinicians who provide substance abuse treatment are employed by provider organizations and not in private practice. Their organizations and specifically their clinical supervisors have an impact on their satisfaction and ability to implement EBTs (Center for Substance Abuse Treatment, 2007; Knudsen et al., 2008). Studies that take into account that clinicians are part of an organization may provide training in the organizational setting, provide training to management staff, or recruit from one organization. Morgenstern et al. (2001) randomly assigned 29 substance-abuse counselors from two outpatient treatment programs (chosen on the basis of several selection criteria) to receive training in Cognitive Behavioral Therapy (CBT) consisting of 35 hours of didactic instruction plus case training and intensive supervision or to a control group. Based on ratings of videotaped sessions, 90% of the counselors were judged to have adequate CBT skills. Liddle et al. (2006) examined the effectiveness of the implementation of multidimensional family therapy with 10 program staff in one adolescent day treatment program. The technology transfer intervention included methods of training management and clinical staff. The clinical staff training included in-depth didactic training based on a manual, a review of exemplary therapy videotapes, 10 hours of individual supervision, and 15 hours of supervision based on co-therapy. After a 14-month implementation phase, results suggested that there were significant improvements with regard to the number of sessions, the content focus of sessions, and adherence to multidimensional family therapy principles. In an attempt to see if the Miller et al. (2004) study findings generalized to clinicians recruited from one organization, Moyers and colleagues conducted a randomized controlled trial with 129 clinicians recruited from Air Force behavioral health providers (Moyers, Manuel, Wilson, Hendrickson, Talcott, & Durand, 2008). Participants were assigned to receive workshop training or workshop training plus personalized feedback and consultation phone calls. The enrichment did not have the expected additive effect on observed clinician skill levels, and the authors concluded that greater resources may be needed to implement an EBT with clinicians who might have comparatively lower skills than those who self-select independently from an organization context into treatment training studies.

This article describes the dissemination and implementation strategy and reports process data from a federally funded effort to diffuse two EBTs across 33 substance abuse treatment provider organizations, 167 clinicians, and 64 supervisors. The strategies employed drew heavily from the literature on training clinicians, since these studies provided clear support for an interactional approach using a treatment manual, preparatory workshop, and individual performance and ongoing feedback. The diffusion approach also recognized that the federal funding was directed at the organizational level with a goal of addressing gaps in the adolescent treatment system by building infrastructure to sustain EBT use within organizations after the federal funding period ended. Thus, the goals of the implementation effort were to: (a) assist clinicians in achieving EBT competency; (b) maintain clinician competency over time; and (c) assist clinical supervisors to achieve competency in becoming trainers in the EBTs at their organization.

Overview of the Dissemination and Implementation Project

Funding mechanism

The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Substance Abuse Treatment (CSAT) funded Assertive Adolescent and Family Treatment (AAFT; TI-06-007) sites to implement the Adolescent Community Reinforcement Approach (A-CRA) followed by Assertive Continuing Care (ACC) to address gaps in adolescent substance abuse treatment. Each site received approximately $300,000 per year for up to three years. CSAT also funded the training and ongoing support and supervision for the EBTs through a contractor during the grant period and required it to develop a training model that would maximize the sustainability of the EBTs once the grants ended.

Description of evidence-based treatments

A-CRA is an adaptation of the Community Reinforcement Approach (Hunt & Azrin, 1973), which was initially developed and tested with adult samples (e.g., Azrin, Sisson, Meyers, & Godley, 1982; Higgins et al., 1991; Hunt & Azrin, 1973; Smith, Meyers, & Delaney, 1998). CRA is a behavioral therapy that seeks to use social, recreational, familial, school, or vocational reinforcers to help individuals in their recovery process (Meyers & Smith, 1995). The intervention consists of seventeen procedures that clinicians draw upon, dependent on adolescent-specific information and goals, plus a General Clinical Skills and Overall A-CRA category. Sample procedures include (a) Functional Analysis of Substance Use, (b) a Happiness Scale, which is paired with Goals of Counseling to develop treatment goals, (c) Problem Solving, (d) Communication skills, and (e) Increasing Pro-Social Recreation. A-CRA was adapted for adolescents, manualized, and evaluated in the Cannabis Youth Treatment (CYT) study (Dennis et al., 2004), a large randomized clinical trial that compared different outpatient interventions for adolescents. The adaptations for adolescents included: (a) adding procedures for parent/caregiver sessions, (b) revising the Happiness Scale, a CRA tool that measures satisfaction with various life areas, so that its rating categories were more appropriate for adolescents, and (c) including adolescent dialogue in a new treatment manual specific to adolescent issues (Godley et al., 2001). Thus, A-CRA is a procedure-based intervention, which requires therapists to have good working knowledge not only of how to deliver each procedure, but also how to decide which one(s) are appropriate based on what the adolescent says during a treatment session. The EBT is designed for weekly sessions over a 90-day period.

Clinicians delivering Assertive Continuing Care (ACC) structure sessions using A-CRA procedures; however, more emphasis is placed on case management and service delivery in the home and community to increase the likelihood that adolescents link to and participate in this continuing care approach within 14 days of discharge from the prior treatment episode. ACC clinicians continue to meet weekly with adolescents and/or their caregivers for another 90 day period. Responding to NIDA recommendations that longer retention (> 90 days) results in better treatment outcomes, CSAT’s goal for this project was to retain adolescents in treatment for 6 months using evidence-based approaches. Most adolescents received 90 days of outpatient A-CRA followed by 90 days of home-based ACC. This sequencing makes sense because ACC incorporates the A-CRA procedures, but also has the advantage of facilitating retention for adolescents and families by switching to home visits and adding case management services to assist with accessing other services (e.g., GED, medical). Treatment retention has proven to be a significant challenge (OAS, 2005) for clinicians, thus ACC provided a means to increase retention by reducing barriers to ongoing treatment participation and reinforced the skills learned during the initial phase of treatment in the adolescent’s natural environment.

There are complementary research streams for A-CRA and ACC. The CYT study evaluated A-CRA in comparison to a five session Motivational Enhancement and Cognitive Behavioral Therapy (Sampl & Kadden, 2001) and Multidimensional Family Therapy (Liddle, 2002) with 300 adolescents appropriate for outpatient treatment. Outcomes revealed that, on average, adolescents in these three outpatient interventions significantly improved during the treatment phase and maintained gains. A-CRA proved significantly more cost-effective per person in recovery at 12 months after intake than the two other interventions in its study arm (Dennis et al., 2004). Slesnick et al. (2007) compared A-CRA and treatment as usual in a drop-in center setting with 180 street living youth aged 14 to 22. The A-CRA group reported significantly reduced substance use, depression, and increased social stability at the six-month follow-up (Slesnick, Prestopnik, Meyers, & Glassman, 2007). Additionally, process research has shown that exposure to A-CRA procedures was a significant mediator of the relationship between treatment retention and outcome (Garner, Godley et al., 2009).

ACC has been evaluated following residential treatment to help adolescents maintain abstinence upon returning to their home community. In a study of 183 adolescents randomly assigned to ACC and usual continuing care following residential treatment, findings suggested that ACC was more effective than usual continuing care at linking adolescents to continuing care, retaining them in continuing care, and helping them attain long-term abstinence from marijuana (Godley, Godley, Dennis, Funk, & Passetti, 2007). Findings from this study also indicated that ACC’s impact on substance use and substance-related problems was mediated primarily through reductions in environmental (e.g., peer and family) risks (Garner, Godley, Funk, Dennis, & Godley, 2007).

Replication sites

After a competitive application process, CSAT funded 15 grants in October 2006 for three years. A second cohort of 17 grantees was funded in October 2007 (one of these sites was later defunded). Two other sites were added through another grant mechanism for a total of 33 sites. Grantees were located in 14 different states and served diverse communities, including ‘colonias’ located on the Texas border, urban areas, Native American communities, and rural areas in Arkansas and Tennessee. Six of the grantees were led by university-associated project directors, while the others were led by community-based agencies.

Adolescent participants

Collectively, the sites projected that they would serve close to 4,000 adolescents and young adults during their 3-year grant funding. With 7 and 19 months of funding remaining for cohorts 1 and 2 respectively, a sample of 1,923 participants has been recruited to date. Most of the adolescent treatment participants were male (72%) and 15 to 17 years old (78%). The largest ethnic groups represented were Caucasians (32%) and Hispanics (31%). The remainder reported their ethnicity as mixed/other (17%), African American (16%), American Indian\Alaskan Native (3%), or Asian (1%). Eighty percent of the total reported symptoms consistent with diagnoses of past year substance abuse or dependence (5% reported abuse/dependence with multiple substances), while the remainder were eligible for services based on other information known to service providers (e.g., from the juvenile justice system, school authorities, or parents). Seventy percent reported symptoms consistent with a diagnosis of one or more co-occurring psychiatric disorders; the most prevalent included Conduct Disorder (51%), Attention Deficit/Hyperactivity Disorder (49%), Major Depressive Disorder (39%), Posttraumatic Stress Disorder (27%), and Generalized Anxiety Disorder (12%). Most adolescents had been involved in the juvenile justice system (84%).

Implementation Strategies

Figure 1 provides a diagram of the implementation model and shows the dual clinical and supervisor tracks for learning the skills necessary to deliver the EBTs and to train and supervise others in them. Supervisors are encouraged, but not required, to be certified as clinicians since the funder did not make this a grant requirement. The supervision certification process is designed to enhance sustainability of the EBTs in two ways. It helps increase organizational support for clinicians going through their certification process, and supervisors learn to emphasize key points during supervision sessions, rate clinical session recordings, provide the clinician with feedback to maintain fidelity to the model, and provide training to new clinicians in the model. If supervisors can reliably train and certify clinicians at their site before the grant period ends, the expectation is that it will be easier for them to sustain the intervention. The implementation effort is supported by staff with different roles and takes advantage of existing technology (e.g., digital audio recording, the World Wide Web).

Figure 1.

Coordinating Center Team

Trainers/coaches

Treatment developers designed the dissemination and implementation strategies and served as the lead workshop trainers and supervisors on coaching calls. Given each grantee had different timeframes for hiring clinical staff and there was staff turnover, multiple training workshops were offered annually with standardized agendas.

Expert raters

To assure rapid feedback to over 100 clinicians and supervisors learning the interventions simultaneously, a cadre of over 30 “expert raters” were gradually recruited and trained to provide individual performance feedback. The first expert raters were advanced clinical psychology graduate students trained and supervised by one of the model developers; over time,, select clinicians or supervisors were recruited who worked in community-based agencies like the clinicians whose sessions they rated. These individuals had performed well in the certification processes, lived in different areas of the country, and represented and worked with different cultural groups. Some of these individuals were able to rate sessions conducted in Spanish since there was a high percentage of Spanish-speaking treatment participants across sites. Expert raters were required to complete their reviews and upload their feedback within seven business days to the website used for posting session recordings, feedback, and session data for the coordinating center team and the clinicians and supervisors in training. Requirements for becoming an “expert rater” were that the individual had prior training and/or clinical experience in A-CRA and could achieve agreement ratings of at least 80% in comparison with ratings by one of the rating manual authors across six or more procedures. Raters were provided the CRA/A-CRA therapist coding manual (Smith et al., 2007). This over 150 page manual contains operational definitions of each rating possibility (1–5) for each component of the 17 A-CRA procedures (Smith et al., 2007) as well as general clinical skills (i.e., warm/understanding, non-judgmental, maintaining session focus, and appropriately active). Each session is also rated on an overall CRA dimension that includes two components: (a) stayed within CRA protocol (as far as philosophy and objectives) and (b) introduced CRA at appropriate times. Each clinician has a color-coded certification workbook that was developed with an electronic spreadsheet (see Figure 2). Thus, every time a session is reviewed, the clinician gets specific feedback on whether or not he or she deviated from the protocol and in what way. The workbook provides a record of the ratings for each session, and shows which procedures still need to be passed in order to complete certification. Each time a new Digital Session Recording (DSR) is rated, the expert rater completes a new column in the workbook, and both the clinician and his or her supervisor are able to see these ratings, as well as the written narrative feedback, by downloading them from the web tool. When a clinician demonstrates competency in a procedure (3 or better rating in all procedure components), then the far left column of that procedure row turns green to provide a visual cue that this procedure has been passed. Raters also were trained to provide narrative comments (see Figure 3), which were shaped to emphasize what a clinician did well, and to provide specific feedback regarding how the clinician could improve. To increase inter-rater reliability, raters participated in monthly conference calls with the lead author of the coding manual to discuss nuances in the rating process. Based on ratings by one of the authors of the coding manual and five randomly selected raters, who each independently rated 31 session samples (representing 18 different procedures), the average intraclass correlation coefficient (ICC) was .75. According to Cicchetti’s (1994) criteria for classifying the utility of ICC magnitudes, ICCs above .75 are considered excellent. Additionally, in terms of whether or not the procedure was rated as having met the criteria for competency, the five randomly selected raters matched the coding manual author 75% of the time. See Garner, Barnes, & Godley (2009) for a more detailed description of the training process for expert raters.

Figure 2.

Certification Workbook Example¹

¹Column 1 shows if the therapist has met certification requirements for that procedure (shading indicates they have passed), column 2 shows the name of the procedure, columns 3 and 4 show the procedure component number from the rating checklist and the component name, each remaining column lists the ratings for different digital session recordings.

Figure 3.

Sample narrative feedback after session reviews

EBT coordinator

An EBT coordinator position was created to serve as the primary contact for clinicians and supervisors going through the certification process. The coordinator was available during regular business hours by e-mail or telephone to answer questions (e.g.,: regarding feedback on their clinical sessions, the certification process, or how to use the project’s web tools). The coordinator assessed the need for additional training or coaching, assigned session recordings to raters, distributed schedules for coaching calls, ensured that minutes were taken during all coaching and rater calls, and sent e-mail notifications to the funder, site personnel, and trainers when a clinician or supervisor achieved certification. They also compiled data related to clinician, supervisor, and site progress, which was used in monthly management reports for the federal project officer and provided to each site director.

Web Tool

A web tool was developed to facilitate the expert reviewing/certification processes since trainees and expert raters are located throughout the United States. The web tool enables clinicians to upload their session recordings and expert raters can access the digital session recordings (DSRs) for review and upload their ratings and narrative feedback for clinicians. Clinicians also can listen to exemplary sample sessions of A-CRA on the website. Additionally, clinicians enter the following participant data in the web site: treatment admission date, dates of service, type of session (face-to-face vs. telephone), the A-CRA procedures delivered during each session, number of minutes spent on each A-CRA procedure, results of urinalysis if conducted, and the treatment discharge date. Reports based upon information from the participant session data can be used to help monitor implementation and fidelity of the EBTs.

The security of the web tool is ensured through a secure physical location with controlled access and several redundancy capabilities (e.g., different back-up locations, power redundancy, network redundancy). Network security for the system consists of secured web access (SSL) with x.501 certificate authentication controlled by individual user logins and encrypted passwords. The system is encrypted using 128-bit encryption technology and resides behind a monitored firewall to limit access. Weekly, activity logs are monitored, and antivirus and operating system patches are updated. Access to various types of content is determined by an individual’s assigned role per his or her login name and password. For example, clinical supervisors have access to supervision pages that allow them to upload their supervision session recordings and access and review recordings of sessions their clinicians have uploaded to the website. Clinicians are limited to accessing information related to their recordings and their clients. Expert raters are permitted to review only those clinical sessions assigned to them for review. Participant confidentiality was addressed with consents to record that were obtained from the participants and/or guardians at the local site. Only numeric IDs were used as participant identifiers, and no Protected Health Information was entered in the website.

Certification Requirements and Process

As shown in Figure 1, both clinicians and supervisors are required to go through several of the same initial steps. These include: (a) reading manuals and taking knowledge tests; (b) attending a 3.5 day centralized in-person initial training workshop; and (c) attending bi-weekly telephone coaching calls. The initial training workshop includes a mixture of lectures, sample case presentations, demonstrations of procedures, and participant practice with trainer feedback.

A-CRA clinician-specific certification requirements

Clinicians have to demonstrate competency in the intervention based on ratings of actual therapy sessions. All sites are provided digital recorders at the initial training and asked to begin recording and uploading all DSRs to the web tool within the first three weeks after completing the initial training. Clinicians are asked to identify a DSR to be reviewed for certification purposes each week by emailing the EBT coordinator. For basic certification, clinicians have to “pass” nine different procedures and receive a certificate when they do. After achieving basic certification, they are expected to work towards proficiency in the remaining procedures. This two-step process was implemented to help clinicians feel like they were making progress in learning the intervention. To reduce procrastination, clinicians are asked to complete basic certification within six months of attending the training workshop or submit a certification extension request to the CSAT project officer for approval.

A-CRA supervisor-specific certification requirements

Supervisors have to demonstrate competency in conducting supervision that relates to the treatment models. They are asked to record their supervision sessions with their clinicians during the weeks they are not participating in coaching/supervision calls with the trainers. During supervision sessions, the supervisors are expected to review a clinician’s Case Review Report (see Figure 4). The Case Review Report is derived from the session data clinicians enter into the web tool. These reports provide information for managing each clinician’s caseload. Each clinician’s report has rows listing the adolescents on their caseload, and shows, for example, the number of completed sessions. Importantly, this number can be compared with the number of weeks the adolescent’s case has been open, thereby giving the frequency of attended versus missed sessions. Also summarized for each adolescent are the number of times each A-CRA procedure has been completed and the results of urine tests. By reviewing the data, supervisors can assess whether the clinician needs coaching in certain areas. Raters also review supervision DSRs to ensure that supervisors are providing appropriate feedback after listening to a clinician’s recorded therapy sessions. Finally, to ensure they understand what constitutes quality implementation, supervisors also have to achieve high consistency (80% or better) with the expert rater for at least six A-CRA sessions.

Figure 4.

Sample Case Review Report

After attaining certification, supervisors are allowed to train and certify clinicians at their site. However, the coordinating center has to approve the training agendas and validate certification recommendations. When a supervisor assesses that a clinician is ready for certification, the supervisor emails the EBT coordinator a completed clinician certification workbook. The coordinating center team chooses three of the clinician’s most recent DSRs for review to confirm certification. If the expert raters do not agree with the supervisor that the clinician has passed a particular procedure, then the clinician is required to continue to submit recordings of that procedure until it is passed.

ACC clinician-specific certification requirements

Clinical certification in ACC requires certification in A-CRA plus evidence that an ACC clinician can achieve a minimal level of linkage to ACC, successfully complete mid-week telephone calls, and provide appropriate case management. Summary measures for these indicators are derived from the participant session data that clinicians enter into the website.

ACC supervisor-specific certification requirements

ACC clinical supervision certification is based on a review of recorded supervision sessions during which the supervisor demonstrates the ability to review, praise, and/or provide suggestions to the clinician about: (a) linking the client to ACC, (b) the frequency or content of the mid-week telephone call, and (c) providing case management assistance with probation, school, or other needs. After ACC supervision certification, the supervisor can train and certify clinicians at their local site.

Coordinating Center Coaching Structure

After the training workshop, coaching telephone calls involving approximately four sites are provided to assist clinicians and supervisors in learning to implement the EBTs. Telephone coaching calls have been used effectively to train clinicians involved in clinical trial research and in practice-based settings (Carroll et al., 1994; Miller, Yahne et al., 2004). The biweekly calls are led by model experts. Supervisors are expected to attend the calls with their clinicians so that everyone at the site receives the same guidance. These calls include discussions of therapy sessions previously reviewed by the expert raters and conducted by clinicians on the call. Attendees have the opportunity to ask questions about specific procedures or other items related to implementation. After each cohort has been funded for a year, coaching calls are offered monthly and are “optional” for clinicians who have achieved certification.

A supervisor-specific coaching call was instituted at the request of supervisors, so that they have the opportunity to discuss supervision issues and the supervision certification effort. Additionally, individual and site-specific coaching calls were added after it became clear through monthly progress monitoring that certain individuals or sites were experiencing ongoing problems achieving certification.

Fidelity Monitoring After Certification

In order to prevent model drift after certification, every other month, a random DSR for each certified clinician is drawn from the web tool and rated. If the rating results in a 3 or better for every component of a procedure, the therapist “passes” the fidelity check. Knowing that it is possible that a therapist might not complete an A-CRA procedure in every session, if the therapist does not pass a procedure in the first session drawn, another DSR is drawn for a fidelity check. If the second review also does not pass, the therapist is provided with an individual coaching session to refresh his/her skills and is then put back in the fidelity check process. In addition to fidelity checks, clinicians are asked to work towards certification in the remaining A-CRA procedures for full A-CRA certification.

Performance Monitoring Reports

Site progress is measured with a number of performance indicators that are reported monthly to the site and the funder. These reports include each clinician and supervisor’s certification progress (e.g., number of procedures passed, supervision certification processes achieved), number of participants opened, number of DSRs uploaded to the website, fidelity review results, and number of sessions conducted. These indicators are used to determine site needs for additional technical assistance or project officer encouragement.

Process Evaluation Data

Twenty-nine months after the first cohort was funded, and 17 months after funding of the second cohort, over 2,000 adolescents had been opened to the AAFT project across 33 sites. A total of 25,000 sessions had been entered in the database, 10,000 digital sessions had been uploaded to the website, and almost 3,000 session recordings had been rated by experts. A total of 167 clinicians and 64 supervisors have begun certification. Table 1 displays data for several process measures related to clinicians’ progress in A-CRA and ACC certification.

Table 1.

Process Measures for Basic A-CRA & ACC Certification

Processes	A-CRA Only	A-CRA+ACC
Began certification process	167	117
Attained certification	105	26
Pursuing certification	27	48
Left position	35	43
DSRs for certification (median)	19 (8–49)
Months to certification (median)	9 (3–24)	14 (6–26)
Months in process if still pursuing certification (median)	8 (0–30)	12 (0–30)
Training hours for certification (median)	43 (33–56)	50 (37–60)
Didactic training (hours)	28	28
Feedback and coaching calls (hours)	14 (5–28)	21 (9–32)
Fidelity of certified clinicians (n = 105)
Clinicians with fidelity checks	82
Initial fidelity checks (mean)	3.9 (2.6)
Percent passed on initial review (mean)	48% (34%)
Second chance fidelity checks (mean)	1.1 (1)
Percent passed on second chance review (mean)	68% (40%)

Note. All clinicians seeking ACC certification must first attain A-CRA certification. Time to achieve ACC certification includes time spent attaining A-CRA certification.

Knowledge tests, workshop satisfaction, and coaching call attendance

All trainees achieved the required 80% correct on the A-CRA and ACC knowledge tests. Based on a 5-point Likert rating scale, with ‘5’ being “Excellent,” the average score for the item, “Please rate your overall training experience” across 10 multi-site training workshops was 4.62 (N=283).. Overall, clinicians pursuing certification uploaded DSRs at a rate of 44% of the total number of sessions that they had entered into the session data portion of the project website. Average coaching call attendance rates for clinicians during the certification process was 72 percent. For clinical supervisors, the average coaching call attendance rate during the certification process was 71% of available opportunities.

Clinician and supervisor certification

Across the 33 sites, a total of 167 individuals have initiated the A-CRA clinician certification process. Based on a subset of 134 (80.2%) individuals who completed questionnaires on their demographics and training, AAFT staff had the following characteristics: Female (74.6%); an average age of 36.7 years (SD = 11.1); either a master’s degree (61%) or a bachelor’s degree (31.3%); and an average of 3.7 years (SD = 5.4) of prior substance abuse counseling experience. The main ethnic groups represented were Caucasian (53.8%), Hispanic (21.2%), and African American (15.9%).

As shown in Table 1, 105 (62.9%) clinicians have attained the basic A-CRA certification at this time. The average number of DSRs submitted prior to certification was 21.6 (SD = 8.9; range of 8 – 49 DSRs). The average length of time that it took to complete certification after the initial workshop was 9.6 months (SD = 4.1; range of 3 –23.75 months). Of the 62 therapists not certified in A-CRA, 27 (43.5%) were still pursuing certification and have participated in the training process for a median of 8 months, and 35 (56.5%) have turned over. After certification, forty-three percent had passed 1 to 3 additional procedures, 39% had passed 4 to 6 additional procedures, and 18% had passed 7 to 10 additional procedures. A total of 26 clinicians (22.2%) were certified in ACC out of 117 who were attempting ACC certification. The average time to ACC certification was 16.6 months (SD = 5.3; range: 7.4 – 29.3). Recall that certification in A-CRA is a prerequisite to certification in ACC. Of the 91 therapists not certified in ACC, 48 (53%) are still pursuing certification and 43 (47%) have turned over.

Twenty-five (39.1%) of those pursuing supervisor certification also were certified as A-CRA clinicians. Of the 64 A-CRA supervisors-in-training, 35 (54.7%) have been certified as A-CRA supervisors. It took an average of 2.7 recordings of their supervision sessions (SD = 2.0; range: 1 –10) before their ratings met criteria for this aspect of the certification process. Supervisors required an average of 12 ratings of their clinicians’ therapy sessions (SD = 6.0; range: 6 – 41) before meeting criteria on the second requirement for certification (at least an 80% agreement with expert raters across each of 6 A-CRA session ratings). The average time to A-CRA supervisor certification was 9 months (SD = 4.3; range: 1.0 – 18.0). Of the 29 individuals not certified as A-CRA supervisors, 13 (45%) are still pursuing the certification and 16 (55%) have turned over. A total of 48 individuals have begun the ACC supervisor certification process. Of these, 18 (37.5%) have attained certification. Of the 30 not certified as ACC supervisors, 19 (63%) are still pursuing certification and 11 (37%) have turned over. The average length of time to complete ACC certification was 18.6 months (SD = 5.8; range of 9.8 – 28.6 months).

Clinician fidelity reviews

Out of the 105 A-CRA certified clinicians to date, at least one fidelity check has been completed for 81 clinicians (77.1%). Of the 81 clinicians who have had at least one fidelity check completed, the average number of initial fidelity checks was 3.9 (SD = 2.6), and the average percentage of fidelity checks passed on the initial every other month review was 46.8% (SD = 33.8; range: 0 to 100). The average number of second fidelity checks (conducted when a clinician has not passed the initial fidelity review) was 1.1 (SD = 1.0), and the average percentage of second fidelity checks passed was 68.0% (SD = 40.1; range: 0 to 100).

Site progress

Across the 15 sites funded in the first cohort of AAFT, there was an average of 3.47 clinicians certified per site. Every site has had a certified clinician. Across the 17 sites funded in the AAFT2 cohort, there was an average of 2.71 clinicians certified by site. Twelve sites also had certified supervisors. The two sites that were funded from other grant mechanisms had an average of 3.5 clinicians certified, and both had a certified supervisor.

Staff turnover rates

Staff turnover rates can be expected to impact implementation and sustainability and have been reported to be as high as 50% for substance abuse treatment professionals (McLellan, Carise, & Kleber, 2003). When calculated based on those who left after the initial training, the turnover rate for clinicians is 44% to date. A total of 34% of the therapists left within the first year of starting the project. The rates for supervisors are not comparable because there are so many fewer supervisors across the project. Still, using these same criteria, the turnover rate for supervisors based on those who attended an initial training session is 33%, and 27% left within the first year of starting the project. Several individuals who first served in the project as clinicians became supervisors later in the project (which could be considered a positive move that would help sustainability). Of those clinicians who left their positions, 76% had not yet achieved any certification, and 23% had attained a certification.

Discussion

The challenges of disseminating and implementing EBTs in practice settings have been documented (Garner, 2009; Gotham, 2006). Miller and his colleagues (2006) have recommended that since researchers have identified effective training mechanisms (treatment manuals, preparatory workshops, individualized personalized feedback, and coaching) in randomized clinical training trials, these methods should be used in EBT diffusion efforts. This paper describes a large scale dissemination and implementation model incorporating these components with staff working at provider organizations that have received federal grants to implement specified EBTs. The approach takes advantage of technological advances, like the World Wide Web, for uploading digital recordings of therapy sessions for review during a certification process. Process data reveal that it is possible to simultaneously train practice-based clinicians in EBTs across multiple and diverse treatment organizations; however, there needs to be a considerable investment in training and supervision. These findings are congruent with those of other treatment developers who have implemented EBTs in practice settings (Henggeler & Schoenwald, 2002; Liddle et al., 2006).

Unique aspects of the implementation approach are: (a) the clinician and supervisor certification process, with a workbook that provides visual feedback of progress; (b) an EBT coordinator assigned to each trainee to provide support and answer questions; (c) a web tool used for uploading digital recordings for the certification process and collecting treatment implementation data; and (d) random fidelity reviews after certification. There is precedence for certifying therapists in treatment approaches as part of clinical trial research (Carroll et al., 1994), but this may be the first documented example of clearly defined therapist and supervisor certification requirements paired with follow-up fidelity reviews during such a large-scale diffusion effort. Moreover, implementation was monitored with monthly management reports summarizing process indicators by clinician, supervisors, and grantee organization.

One of the recurring challenges in studies of clinician training has been the compliance of trainees with audio/video recording of therapy sessions for quality assurance purposes. Moyers et al. (2008) obtained usable recorded work samples from 38% of their participants at the 8- and 12- month follow-up intervals. Similarly, Miller et al. (2004) found that only 45% of the participants provided all three follow-up tapes (even when financial incentives were provided). Low session recording compliance is most likely due to clinicians’ reluctance to record their sessions for evaluation, but also could be hindered if tapes have to be packaged, postmarked, and shipped, or if there is no employer or funder requirement to record. Clinicians sometimes report that treatment participants are not willing to be recorded; however, anecdotally, we have found that this resistance usually can be overcome by how the clinician presents the request for recording (e.g., it is to help the clinician improve his/her skills). The use of the web-based tool, digital recorders, expectations for regular DSR uploads, and funder oversight in this project are believed to have increased compliance with session recordings. To date, all therapists (n=167) in the certification process have recorded sessions, and an average of 76 therapy session recordings per therapist have been uploaded to the website tool.

Training strategies, length of training, and percentage of clinicians achieving EBT competency can be compared between this project and randomized clinical trials of clinician training. The experiments that have evaluated motivational interviewing (MI) training included 2-day workshops, feedback on four practice tapes, and a fixed number (6) of available individual consultation or coaching calls (Miller et al., 2004; Moyers et al., 2008). The Morgenstern et al. (2001) study of cognitive behavioral therapy (CBT) training included 100 hours of clinical training over a five-month period. Also, the level of supervision in this model was closer to the one provided in the Morgenstern et al. (2001) study in which both individual and group supervision was provided weekly. Comparing the success of different implementation approaches has limitations because the criteria for determining competency differ across EBTs and training approaches. Morgenstern et al. (2001) reported that 90% of the 20 (out of 29 possible) therapists who submitted recordings in the CBT condition were judged adequate on the basis of two video-tape reviews. Sholomskas et al. (2005) reported that 54% of the therapists (out of 78) assigned to the seminar plus supervision condition, 48% of those assigned to the web condition, and 15% of those assigned to the manual only condition were able to attain adequate levels of adherence and skill ratings. Clinicians trained in the project described in this paper compared favorably, with 63% (out of 167) having attained certification thus far, and with 27% still active in the certification process. The median training time to A-CRA certification for clinicians was 43 hours, averaged 9 months, and involved reviews of an average of 20 digital session recordings, a much higher number of therapy session recordings than typically reviewed in the training studies. Differences in attaining competency could be due to variation in the EBTs, the methods used to assess competency, clinician skill levels, or organizational factors. For example, most of the organizations involved in the CSAT project hired new clinical staff and were implementing the EBTs as a new program or service.

Limitations

This dissemination and implementation effort is unusual given the amount of funding that was provided to sites for training and certification. The purpose of developing this approach was to ensure the simultaneous implementation of the two EBTs across more than 30 organizations with over 200 clinicians and supervisors. However, since its development was underwritten for this purpose, we have been able to offer a training and certification package that is more affordable for individual or groups of organizations. Thus, since beginning the AAFT initiative, 30 more organizations that were not part of that initiative have self-funded or acquired small grants to contract for these services. Many state and private payers are now encouraging provider organizations to implement EBTs, and this is contributing to demand and affordability.

It also might be considered a limitation that only 48% of the therapists passed initial fidelity checks after their certification; however, there is a lack of data in the literature for comparison purposes. It is important to note that “failing” a fidelity check was based on a high standard. Thus, a clinician could receive a failing rating on one of five components and demonstrate competency on the other four components, and still fail the fidelity check. This is the reason that additional DSRs were reviewed the next month, and 68% of the clinicians passed during these reviews. These data suggest that learning an EBT is an ongoing process, rather than a discrete event, and it is important to continue to check for drift and provide ongoing feedback and supervision support.

The therapist turnover rate is concerning, given the investment in training resources. Examination of turnover rates during implementation of EBTs (Sheidow, Schoenwald, Wagner, Allred, & Burns, 2007) is a relatively new area of study, so it is difficult to ascertain if the rates reported are comparatively high. Anecdotally, some grantee sites had difficulty finding qualified applicants and did not necessarily know how to choose clinicians who would respond well to the intense training and supervision model. In general, there appears to be unevenness in the availability of qualified clinicians across the United States. These findings suggest that it is important that organizations are educated regarding staff attributes critical for EBT implementation. When clinicians are interviewed for positions, they can be made aware of certification requirements (e.g., need to record sessions and incorporate feedback) and their willingness to participate in the process assessed. Site managers and supervisors can help by supporting their therapists during the training process, promoting a team approach between the site and trainers, and monitoring clinician progress.

Implications for Dissemination and Implementation

A large portion of adolescent substance abuse treatment is financed through public funding, and the framework described here can be replicated by state, county, or other local funders, although it might require re-allocation of financial resources. The funder would identify an EBT to implement and provide resources to fund clinician and supervisor training combined with ongoing implementation monitoring and feedback. Some of the strategies used in this large scale implementation effort could be used by local treatment providers; however, at a minimum, organizations need to allocate enough resources to allow supervisors to become competent in an EBT and then train and supervise clinicians at their organizations. The following is a list of strategies and principles that could be adopted to enhance implementation at local organizations:

• Digital recorders are inexpensive, and most treatment providers now have computers and shared hard drive space. Treatment sessions can be recorded on a regular basis and uploaded to a server for review by clinical supervisors with appropriate training.

• Management can develop measurable clinical competency expectations, monitor implementation, reinforce necessary components (e.g., recording therapy sessions, supervisor review and feedback on performance), and achievement of competency milestones.

• Progress towards achieving competency in an EBT can be integrated into an individual’s performance review process because it is directly related to the performance of his/her clinical duties.

• Consideration can be given to contracting out ongoing EBT clinical supervision given the high rates of turnover in the workforce.

Future Directions

All sites were required to use the GAIN (Dennis, Titus et al., 2003) at intake and at the follow-up intervals of 3, 6, and 12 months after intake. These data will provide the opportunity to examine how participant outcomes from this initiative compare to randomized clinical trials that have evaluated these interventions. Additionally, Garner (2008) has initiated a randomized experiment examining the effectiveness and cost-effectiveness of providing monetary incentives to therapists for demonstrated superior implementation of A-CRA. Finally, given the large investment of federal dollars in this initiative, future research is planned to examine the sustainability of the EBTs after the site’s federal funding periods are completed.