Evaluating Training Methods for Transporting Contingency Management to Therapists

Abstract

The effects of three increasingly intensive training methods on therapist use, knowledge, and implementation adherence of contingency management (CM) with substance abusing adolescents were evaluated. Ten public sector substance abuse or mental health provider organizations were randomized to one of three training conditions: workshop and resources (WS+), WS+ and computer assisted training (WS+/CAT), or WS+/CAT and supervisory support (WS+/CAT/SS). Across conditions, 161 therapists participated in the training experiences, and measures were obtained at baseline and 2-month intervals for 12 months following workshop participation. Across training conditions, therapists reported increased CM use, knowledge, and implementation adherence through the 12-month follow-up. The findings show that community-based practitioners are amenable to the adoption of evidence-based treatments when provided access to useful resources. Moreover, high quality workshops in combination with resource access can increase knowledge of the evidence-based treatment and might enhance intervention adherence to a level needed to improve youth outcomes.

1. Introduction

This study contributes to a broader research program that aims for the efficient, effective, and large-scale transport of evidence-based treatments of adolescent substance abuse to public sector practitioners. Adolescents with substance abuse disorders represent a large and underserved population that is at high risk of presenting significant deleterious outcomes and long-term costs for themselves, their families and communities, and society. Based on national data, Godley et al. (2010) estimated that two million adolescents in the U.S. have substance abuse or dependence disorders. Further, epidemiological data indicate that only about 14% of these youth received formal substance abuse treatment services (Substance Abuse and Mental Health Services Administration [SAMHSA], 2010). Longitudinal research has shown that many substance abusing adolescents continue to abuse substances into emerging adulthood with detrimental results pertaining to educational attainments, mental health, social relationships, employment, and physical health (e.g., Henggeler, Clingempeel et al., 2002; Kaminer & Bukstein, 2008). Such outcomes among substance abusing individuals are estimated to cost society more than $180 billion annually, stemming from expenditures to address health care, drug-related crime, and reduced work productivity (Office of National Drug Control Policy, 2004).

These findings demonstrate a pervasive need for service sectors to provide effective (i.e., evidence-based) interventions to ameliorate the individual, social, and societal difficulties associated with adolescent substance abuse. Indeed, the public health importance of transporting evidence-based substance abuse services to community-based treatment settings has been emphasized in several reviews (e.g., Brown, 2000; Institute of Medicine, 1998; Marinelli-Casey, Domier, & Rawson, 2002) and is a federal research (Compton et al., 2005) and services (SAMHSA, 2007) priority. Yet, as noted in these reviews and elsewhere (e.g., Santa Ana, Martino, Ball, Nich, & Carroll, 2008; McCarty et al., 2007), the vast majority of substance abuse treatment services are not evidence-based.

Leveraged by considerable federal support, however, several treatments of adolescent substance abuse have had promising results in clinical trials (National Institute on Drug Abuse [NIDA], 2009; Waldron & Turner, 2008; Weisz & Kazdin, 2010). For example, NIDA (2009) cited three treatments of adolescent substance abuse as scientifically supported: multisystemic therapy (MST; Henggeler, 2011), multidimensional family therapy (Liddle, 2009), and brief strategic family therapy (Szapocznik, Hervis, & Schwartz, 2003). Although these and other evidence-based family therapy models such as functional family therapy (Alexander & Parsons, 1982) are currently being transported to provider sites across the nation (Henggeler & Sheidow, 2012), we suggest that such complex and multifaceted intervention models are not ideally suited as substitutes for usual public sector outpatient substance abuse treatment services. For example, the adoption of the aforementioned evidence-based family approaches typically requires an extensive site assessment process, dedicated funding for personnel, commitment to an intensive and ongoing quality assurance system, and modification of organizational practices to support the program.

Thus, to address the large-scale public health needs posed by adolescent substance abuse, the present study focuses on the uptake of a multifaceted (i.e., including cognitive behavioral therapy techniques) contingency management (CM) approach among practitioners in the substance abuse service sector as well as the mental health sector. Mental health sector therapists were included to provide expanded service capacity (e.g., they greatly outnumber their substance abuse sector counterparts in many states) as well as for their potential, following CM training, to treat co-occurring substance abuse and psychopathology. CM was selected as a promising intervention for large-scale transport for several reasons. CM and its variations have strong empirical support in the adult substance abuse literature (Higgins, Silverman, & Heil, 2008) and promising outcomes with adolescent substance abusers (Henggeler et al., 2006; Henggeler, McCart, Cunningham, & Chapman, 2012; Stanger & Budney, 2010). In addition, due to its relative simplicity, low cost, and ability to be integrated into a variety of substance abuse treatment approaches (Rogers, 2003); CM is more amenable to adoption by public sector professionals than are the aforementioned evidence-based treatments of adolescent substance abuse. Indeed, Henggeler et al. (2008) demonstrated widespread adoption of CM by such practitioners when provided access to workshop training and the resources (e.g., manuals, urine and alcohol drug screens, and vouchers) needed to implement CM.

In light of the need for large-scale transport of effective substance abuse services to community-based settings, the evaluation of training methods to achieve such transport has become a research priority (e.g., Addis, 2002; Baer et al., 2007; Miller, Sorensen, Selzer, & Brigham, 2006). The intensity of current training methods ranges from access to manuals and attendance at workshops to comprehensive and ongoing training and quality assurance protocols provided for complex interventions such as MST (Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009). In general, reviewers (e.g., Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Institute of Medicine, 1998; Miller et al., 2006) have concluded that manuals and workshops alone have little impact on practitioner behavior. On the other end of the training intensity continuum, considerable research has validated the intensive quality assurance protocols used by the complex evidence-based family therapies to promote therapist adherence and achieve desired youth outcomes (see e.g., Henggeler, 2011). Consistent with such conclusions, experimental studies (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004; Sholomskas et al., 2005) have shown that increased intensity in training efforts has led to greater improvements in therapist implementation of evidence-based substance abuse treatments.

In consideration of these findings and with the aim of identifying an efficient strategy that might be used to facilitate the large-scale transport of CM to community-based practitioners, three increasingly complex training protocols for promoting CM use, knowledge, and adherence were selected for evaluation. Each protocol was more intensive than the provision of a treatment manual and workshop, but considerably less intensive than protocols used to support therapist adherence for the aforementioned evidence-based family therapies. The three training protocols included (1) a workshop (WS+) enhanced with ongoing access to the resources (e.g., worksheets, therapist scripts, urine and alcohol drug screens, vouchers) needed to implement CM; (2) WS+ and computer assisted training for 6 months following the workshop (WS+/CAT); and (3) WS+/CAT and ongoing supervision and coaching support primarily for supervisors by a CM expert for 12 months following the workshop (WS+/CAT/SS).

Although, as noted previously, workshops are generally considered weak methods for transporting evidence-based treatments, a close review of the literature reveals that their value might be underrated. For example, in a study that examined five methods for teaching motivational interviewing (Miller et al., 2004), practitioners who had received only the workshop showed significant gains in their motivational interviewing skills, though not as large as those achieved by participants in the more intensive training conditions. Similarly, in an uncontrolled study, Miller and Mount (2001) found that workshop participants reported large increases in motivational interviewing skills, and that observational measures revealed modest changes in practice behavior that were sustained for at least 4 months post workshop. Hence, the WS+ condition in the present study provides a valuable comparison to prior research and might result in increased CM use, knowledge, and adherence, especially with access to the CM implementation resources noted previously.

The second training condition, WS+/CAT, includes several advantages through the inclusion of online training (Weingardt, 2004). As summarized by Dimeff et al. (2009), CAT can elicit greater participant engagement, provide realistic simulated models of therapist-client interaction, allow therapists to progress at their own pace, and tailor course content specifically toward learner needs. CAT has produced promising results in the training of medical professionals (Curran & Fleet, 2005), community-based mental health therapists (Dimeff et al., 2009), and substance abuse treatment providers (Sholomskas & Carroll, 2006; Sholomskas et al., 2005). Indeed, Martino, Canning-Ball, Carroll, and Rounsaville (2011) found that approximately 50% of substance abuse counselors who received a manual and computer assisted training in motivational interviewing met standards of adequate performance at the end of the course. In light of the advantages of CAT, we hypothesized that therapists in the WS+/CAT condition would evidence greater CM use, knowledge, and adherence than counterparts in the WS+ condition.

The third and most intensive training condition, WS+/CAT/SS, includes ongoing coaching and supervision to support the implementation of CM. In their review of the dissemination of evidence-based substance abuse treatments, Miller et al. (2006) concluded that ongoing coaching and supervision are critical to enhancing and sustaining change in therapist behavior. For example, the most favorable outcomes in the Miller et al. (2004) and Sholomskas et al. (2005) studies were achieved by the training conditions that included supervision; and recent randomized trials have demonstrated the clinical effectiveness of CM when delivered by well-supervised community-based practitioners (Henggeler, McCart et al., 2012; Petry, Alessi, & Ledgerwood, 2012). Moreover, reviewers of training and dissemination in children’s mental health services (e.g., Fixsen et al., 2005) and delinquency (Mihalic & Irwin, 2003) have concluded that ongoing coaching and supervisory support are important in sustaining adherence to evidence-based treatments. Hence, we anticipated that therapists in the WS+/CAT/SS condition would show greater CM knowledge, use, and implementation adherence than counterparts in the other training conditions.

In sum, the primary purpose of the study was to evaluate the effects of three increasingly intensive training methods on therapist use, knowledge, and implementation of CM with substance abusing adolescents. Findings will hopefully inform next steps in researching the large scale transport of CM to address the treatment needs of adolescents with substance use disorders.

2. Materials and Methods

2.1. Design and Procedures

The research team partnered with 10 public sector provider organizations – six in the South Carolina Department of Alcohol and Other Drug Abuse Services (DAODAS) and four in the South Carolina Department of Mental Health (DMH). These organizations were divided into three groups equated on service sector and the number of therapists treating adolescents. Following participant recruitment and baseline data collection, therapists attended one of several workshops provided in April 2008. In May, executive level representatives of the 10 participating agencies attended a meeting during which the three groupings of organizations were randomized to the training conditions. As a result, two DAODAS and two small DMH agencies were randomized to the WS+/CAT/SS condition, and two DAODAS and one DMH agency were randomized into each of the other two conditions.

Baseline data collection assessed practitioner characteristics (e.g., demographics, clinical experience) and CM use, knowledge, and adherence of implementation at the time of consent. CM knowledge was reassessed via paper-and-pencil forms immediately following the workshop. Bi-monthly assessments of the key outcomes (i.e., CM use, knowledge, and implementation adherence) began 2 months post workshop and continued through 12 months post workshop. Thus, eight assessments of CM knowledge and seven assessments of CM use and adherence were obtained. Baseline data collection was completed on 100% of participants, and the post workshop CM knowledge data were also collected for 100% of participants. The collection rate of the bimonthly data was also high, with 713 of the 741 (96%) CM use and implementation assessments completed by phone interview and 644 (87%) of the CM knowledge assessments completed by paper-and-pencil forms returned via fax.

2.2. Participant Recruitment and Attrition

The research team visited each of the participating sites to recruit therapists who treated adolescents for some portion of their caseloads. The researchers described all aspects of the study to the prospective participants, emphasizing the voluntary nature of the research and that participation would have no impact on their job performance evaluation. After answering any questions that the eligible participants might have, the therapists were consented to the study per approval from the university and state institutional review boards. Therapists received a $20 gift certificate for each completed assessment and a bonus of $60 in gift certificates if all assessments were completed during the 12-month follow-up.

Of the 170 therapists initially approached for recruitment, 154 (91%) consented and completed baseline assessments. Between the time of the baseline assessment and workshop offering, 14 therapists had left their agency, 2 retired, and 6 withdrew from the research. The remaining 132 therapists subsequently attended the CM workshop and were contacted for follow-up assessments. In addition, 29 of 29 (100%) of therapists newly hired by the provider organizations within 8 months of the initial workshops were recruited and consented for the study, attended a CM workshop provided to new hires, and completed at least one follow-up assessment. Eight months was selected so that therapists would have enough time remaining in the study to be trained and complete at least one follow-up assessment. Hence, the final sample size was 161 therapists (132 + 29). As noted by investigators (e.g., Eby, Burk, & Maher, 2010), counselor turnover is often high in public sector provider agencies, and the present case was no exception. An additional 43 therapists (27%) either left their position in the agency (n = 33) or withdrew from the study (n = 10) at some point during the follow-up. Data from all 161 participants who attended a workshop, however, are included in the analyses.

2.3. CM Specification

The content of the multifaceted version of CM used in this study has been refined through several iterations (e.g., Henggeler et al., 2008; Henggeler et al., 2006; Henggeler, McCart et al., 2012) and is based on a recently published CM treatment manual (Henggeler, Cunningham et al., 2012). The following components, several of which involve cognitive behavioral approaches, include: (1) using validated assessment instruments and clinical interviews to distinguish experimental substance use from abuse or dependence; (2) introducing CM and engaging the youth and caregivers in treatment; (3) conducting functional analysis of the youth’s substance use; (4) based on the functional analysis, designing and implementing self management planning and drug refusal skills training; (5) designing and implementing a contingency contract with youth and caregivers whereby desired rewards (e.g., vouchers) and privileges (e.g., cell phone access) are provided for negative substance use screens and disincentives for positive screens; and (6) developing plans for sustaining abstinence after treatment ends.

2.4. Training Conditions

Therapists and supervisors from the seven agencies randomized into the CAT conditions (i.e., WS+/CAT and WS+/CAT/SS) were given information on how to access and utilize the CAT program. Computers were purchased for each agency, and technical support was available to ensure that each therapist in these conditions had access to a computer and capacity to log onto the web-based training shortly after the randomization meeting. In addition, supervisors in the agencies randomized to the WS+/CAT/SS condition began receiving coaching and supervisory support at that time.

2.4.1. Workshop and resources (WS+)

A 1-day CM workshop, based on the aforementioned CM manual (Henggeler, Cunningham et al., 2012), was conducted at each agency site by a highly experienced expert in CM. Consistent with the literature on effective training (Daniels & Walter, 2002), each component of the workshop included didactic instruction, trainer role-play, and then dyadic role-plays for each therapist to practice implementing CM. These procedures allowed the trainer to provide hands-on training to each therapist during the workshop. Following completion of the workshop, therapists were provided with or received access to several resources needed to implement CM during the next 12 months. These included a copy of the CM manual with handouts, worksheets, and therapist scripts; 30 drug test cups; and $100 in vouchers per youth for up to six substance-abusing youths per practitioner. Finally, workshop attendees received 6 hours of continuing education units.

2.4.2. WS+ and computer assisted training (WS+/CAT)

The Criterion-Based Development Model (Carter, 2005), which uses an iterative process to develop materials and implement phased evaluation and revision of materials, was followed in the development of the CM-CAT. CM experts, educational technology experts, and multiple subgroups of community-based therapists provided feedback on versions of the CM-CAT, with qualitative and quantitative (i.e., a modified version of the After-Scenario Questionnaire; Lewis, 1995) data collected to evaluate user satisfaction and system usability and to guide revisions. The content of the CM-CAT was based on the CM treatment manual (Henggeler, Cunningham et al., 2012) and includes seven modules: (1) Overview, (2) Determining the Need for Treatment, (3) Introducing CM to Families, (4) ABC Assessment of the Youth’s Drug Use, (5) Point-and-Level System, (6) Self-Management Planning, and (7) Drug Testing Protocol. Within each module, sections described the therapist’s tasks, gave trouble shooting tips and suggestions for engaging families, and provided extensive examples and sample scripts. A few video examples of the most challenging aspects of CM were included as was access to printable PDFs of all forms used to facilitate the implementation of CM. Each module could be completed in 1–2 online sessions, depending on the activities involved (e.g., some sessions include homework assignments). The website’s navigation guided users through the modules, and successful completion of a post-test was required before access to the next module was permitted. Pleasing and relevant photographs, artwork, and design were utilized. Each user had a unique login and access to the CM-CAT for up to 6 months, although participants could complete the CM-CAT within 2–4 weeks. Fifteen hours of continuing education units were provided for successful completion of the CM-CAT.

2.4.3. WS+/CAT and supervisory support and coaching (WS+/CAT/SS)

Supervisor training and coaching were delivered by a highly experienced expert in CM and clinical supervision. The supervisor support training protocol was designed to provide supervisors with the requisite skills and coaching to train their staff in implementing CM and was provided in several phases. (1) Site visits were conducted by the CM expert to assess baseline supervisory practices at each of the four agencies randomized to this condition. (2) Three group booster trainings were provided for the agency supervisors to enhance their knowledge of CM, ability to engage therapists to use CM, and capacity to supervise therapists in CM. The workshops included skill acquisition procedures such as expert modeling, guided practice, behavioral rehearsal, corrective feedback, and positive reinforcement. (3) An on-site booster training was held for each agency to meet the specific CM uptake needs of the supervisors and therapists at that agency. These boosters were individualized to address particular challenges experienced by agency practitioners (e.g., engaging families, managing disruptions in the therapeutic alliance). (4) Bi-weekly telephone consultations were scheduled with supervisors in each of the service sectors to review barriers to CM implementation and critique tapes of supervision or therapist use of CM with adolescents and their families.

2.5. Training Implementation

Available indices suggest that the workshop and CAT were well implemented. Satisfaction with the workshop experience was assessed with an 11-item satisfaction questionnaire (e.g., “I would recommend this training program to others,” “The presentation has enhanced my knowledge of the subject”) in which responses were coded on 5-point scales. Across items, mean scores ranged from 4.50 to 4.75 – reflecting high levels of satisfaction among the vast majority of workshop attendees. Similarly, of the 109 practitioners randomized to one of the two training conditions that included CAT, 83% logged-in to the CAT website and 79% completed the website overview. Of the 89 practitioners who began modules, 86% went on to complete all modules of the CAT. Moreover, consumer satisfaction for the CAT was high, with 99% of completers describing the website as “mostly helpful” or “extremely helpful” on feedback forms.

On the other hand, challenges were experienced in the implementation of the supervisor support protocol. While post-booster evaluations indicated that supervisors were highly satisfied with the trainings (M = 4.88, mode = 5 on a 5-point satisfaction scale) and that the training materials were helpful (M = 4.9, mode = 5 on a 5-point scale), supervisor attendance for the telephone consultations was low. DAODAS supervisors cancelled 25% of the 20 consultation sessions that were scheduled, and fewer than half of the supervisors attended sessions on average. DMH supervisors attended all of the 35 sessions that were scheduled, but fewer than half of the supervisors attended these sessions on average. Moreover, supervisors consistently struggled in assessing and addressing clinicians’ barriers to treatment implementation and with managing clinicians’ follow through regarding supervisor recommendations. Although these challenges were addressed during booster trainings by providing supervisors with assessment tools to identify clinician-level barriers to CM implementation (e.g., knowledge and skill deficits, avoidance due to anxiety) and recommending strategies to effectively address lack of clinician follow through, these interventions were not consistently effective.

2.6. Measures

As noted previously, practitioner characteristics were assessed at baseline only, and the outcome measures of therapist CM use, knowledge, and adherence were collected at baseline and 2, 4, 6, 8, 10, and 12 months post-baseline. Knowledge was also reassessed immediately following the CM workshop.

2.6.1. Practitioner characteristics

The Personnel Data Inventory (Schoenwald, 2003) was used to collect demographics (i.e., gender, ethnic/racial status, age) and information regarding therapist degree, addiction certification, years in the field of counseling, years at the present agency, caseload size, proportion of youth/family clients, and proportion of youth substance abuse clients.

2.6.2. CM use

Consistent with procedures used previously (Henggeler et al., 2008), during each interview the research assistant asked the therapist if he or she treated any adolescents with substance use disorders during the past 30 days. If the therapist answered “yes,” he or she was asked how many youth clients were treated for substance abuse, whether CM was used to treat any of these clients, and if so, how many of these adolescents were treated with CM. Thus, the sample for the CM use outcome was based on reports where therapists had the opportunity to use CM because he or she was treating at least one substance-abusing adolescent. Therapists who did not have at least one substance-abusing adolescent in their caseload were excluded.

2.6.3. CM knowledge

The CM Knowledge Test (Chapman, Sheidow, & Henggeler, 2008; 2010) is a brief, easily administered multiple-choice test developed specifically for this study to assess changes in CM knowledge. The development of the test used methods from Standards for Educational and Psychological Testing of the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education and IRT-based test development resources (Wolfe & Smith, 2007). To reduce exposure to the same test items across repeated test administrations, a pool of approximately 150 multiple-choice items targeting CM knowledge was developed. The items were specifically designed to cover the full range of CM knowledge, from novice to expert. Each test version had 16 or 20 items with sufficient “linking” items for simultaneous scoring of test data on a common scale of measurement across the different test versions. To facilitate interpretation in the present investigation, the Rasch-based logit scores were rescaled to a range of 0–100 during model calibration. Psychometric evaluations (Chapman, Sheidow, & Henggeler, 2008; 2010) showed that: the test measured a single dimension of CM knowledge; items fit the model, targeted a wide range of CM knowledge, and were well-targeted to the test-takers; and the test was able to detect an increase in knowledge from pre-to-post training. However, due to the relatively homogenous sample of therapists and small number of items per test, test reliability was low (Schumacker & Smith, 2007. The Rasch separation statistic was 0.87, which indicates that the test could discriminate therapists with considerable knowledge from those with little knowledge. All therapists were included in this measure.

2.6.4. CM implementation

The initial CM-Therapist Adherence Measure (CM-TAM) was developed and used in a juvenile drug court study (Henggeler et al., 2006). Subsequently, based on data from three CM studies and including therapist, youth, and parent reports on the CM-TAM; Chapman, Sheidow, Henggeler, Halliday-Boykins, and Cunningham (2008) concluded that the instrument performed well, with items targeted to the sample, solid item reliability, and stable performance over repeated measurements. Several areas were identified for improvement, however, including the wording of some items, the addition of items that would more fully define the continuum of therapist adherence to CM, and revision of the original 5-point rating scale. The next iteration of the instrument resulted in a 34-item revised CM-TAM with a 4-point scale that assesses therapists’ use of the two major components of CM: Cognitive-Behavioral interventions (19 items) such as functional analysis of substance use, self-management planning, and drug refusal skills training; and Monitoring interventions (15 items) such as regular drug screening and contingency contracting.

The revised CM-TAM has been further validated in two recent and separate studies. Within a multi-site study of juvenile drug courts, McCart, Henggeler, Chapman, and Cunningham (2012) found that based on caregiver and youth reports, the CM-TAM Cognitive-Behavioral and Monitoring scales differentiated therapists using CM in comparison with those delivering usual services. Highly pertinent to the present study, Chapman, McCart, Letourneau, and Sheidow (in press) examined the associations among CM-TAM adherence scores based on therapist, caregiver, and youth ratings as well as on observational ratings by CM treatment experts and trained raters. Although therapist ratings of adherence were somewhat higher than those of the observational raters, the therapist ratings were highly associated with those of both the experts and trained raters, thereby supporting the validity of therapist reports. On the other hand, caregiver and youth reports showed little correspondence with the observational ratings.

The present study used therapist reports on the CM-TAM for up to three substance-abusing adolescents per assessment. Based on the first CM-TAM each therapist reported in the present study, Cronbach’s coefficient alphas were .91 and .84 for the Cognitive-Behavioral and Monitoring subscales, respectively. The sample for the CM adherence outcome was based on assessments in which therapists reported using CM, and therapists who did not report CM use were excluded.

2.7. Data Analyses

Mixed-effects regression models (MRMs) were used to accommodate several features of the data (e.g., repeated measurements; Raudenbush & Bryk, 2002). Two level MRMs with repeated measurements (level-1) nested within therapists (level-2) were performed using HLM software (v7, Raudenbush, Bryk, & Congdon, 2011). Continuous outcomes (i.e., CM knowledge, CM therapist adherence) were modeled using restricted maximum likelihood estimation. The discrete outcome (i.e., CM utilization) was modeled using a binomial trial distribution, where the outcome was CM use status at each assessment (i.e., 0/1) and the trial term was the number of substance abusing youth clients treated by the therapist (i.e., opportunities for CM use). This outcome was estimated using penalized quasi-likelihood estimation. For each outcome, the level-1 model included an uncentered linear polynomial term computed as the number of months between each assessment and the CM training workshop. The CM knowledge outcome also included a dichotomous, level-1 indicator to differentiate the pre- and post-workshop test administrations. The level-2 model included two dichotomous indicators for training intervention condition (i.e., WS+/CAT, WS+/CAT/SS), and cross-level interactions were specified between the condition indicators and the level 1 terms. Robust standard errors were used to compute the test statistics, and unit-specific results were interpreted for the non-linear outcome. Statistical comparisons not directly produced by the model specifications described below (e.g., comparisons for WS+/CAT versus WS+/CAT/SS) were obtained using multivariate hypothesis tests in HLM.

3. Results

3.1. Participant Characteristics

Table 1 presents participant characteristics for the total sample of research participants (N = 161) as well as the corresponding data for therapists in the three training intervention conditions. The vast majority of the overall sample was female (82%) with an average age of 37 years; and 51% were white/non-Hispanic, 48% were African-American, and 1% were other. Educationally, 77% held master’s degrees, 1% doctorates, 1% had not yet earned a bachelor’s degree, and 18% were certified addictions counselors. The practitioners averaged 8 years of professional experience, with an average of 3.3 years at their present agency; and 53% were in the mental health service sector. Regarding clinical services, the median caseload was approximately 20 clients, and the majority (65%) of therapists treated primarily youth/families. In general, therapists reported that fewer than 20% of their youth clients had problems with substance abuse. Univariate analyses were conducted to examine between-groups differences in therapist characteristics for participants in the three training intervention conditions, and one analysis was statistically significant. The WS+ condition included significantly fewer males than either of the other two conditions, X²(2) = 10.18, p = .002.

Table 1.

Participant Demographic and Professional Characteristics

	WS+ (n = 52)	WS+/ CAT (n =53)	WS+/ CAT/SS (n = 56)	Total (N = 161)
Measure	M, %	M, %	M, %	M, %
Age (years)	37	35	39	37
Gender (female)*	96%	75%	75%	82%
Race
White, non-Hispanic	53%	60%	39%	51%
African-American	45%	38%	61%	48%
Other	2%	2%	0%	1%
Degree (master’s or higher)	69%	83%	80%	77%
Addiction certification	18%	21%	14%	18%
Years in the field of counseling	7.4	7.4	9.3	8.0
Years at present agency	2.6	2.8	4.4	3.3
Size of caseload
≤ 10	39%	30%	32%	34%
11 – 20	14%	17%	21%	17%
21 – 30	17%	11%	16%	15%
31 – 40	15%	23%	11%	16%
≥ 41	15%	19%	20%	18%
Proportion of youth/family clients
≤ 20%	24%	17%	21%	21%
21 – 40%	10%	6%	5%	7%
41 – 60%	10%	4%	7%	7%
61 – 80%	4%	4%	4%	4%
≥ 81%	53%	70%	63%	62%
Proportion of youth substance abuse clients
≤ 20%	78%	74%	71%	74%
21 – 40%	4%	6%	4%	4%
41 – 60%	8%	4%	5%	6%
61 – 80%	0%	2%	2%	1%
≥ 81%	10%	15%	18%	14%
Service sector (mental health)	42%	64%	52%	53%

Note. WS+ = workshop and resources; WS+/CAT = WS+ and computer assisted training; WS+/CAT/SS = WS+/CAT and supervisory support.

^*WS+ included significantly fewer males than did the other training conditions.

3.2. CM Use

Of the 161 participating therapists, 79 reported treating at least one substance-abusing adolescent at baseline, and 92 reported treating at least one substance-abusing youth during the 12 months following the workshop. The results for changes in CM use are based on these numbers. As indicated in Tables 2 and 3, therapists reported an increase in CM use (i.e., significant positive linear slopes) from baseline through the 12-month assessment for each of the training conditions. No significant between-groups differences in trajectories of change emerged. Thus, participation in training led to increased CM use, but the trainings were not differentially effective in promoting such use.

Table 2.

Group Means and Standard Deviations for CM Use, Knowledge, and Implementation Per Assessment

Month	Condition	CM Use*	CM Knowledge	CM Implementation
				MON	CB
0	WS+	0.11 (0.32)	0.52 (0.09)	2.04 (0.64)	2.70 (0.61)
	CAT	0.10 (0.30)	0.51 (0.11)	1.80 (0.72)	2.43 (0.64)
	SS	0.05 (0.22)	0.50 (0.10)	1.92 (0.55)	2.63 (0.56)
Post-WS	WS+		0.58 (0.11)
	CAT		0.63 (0.09)
	SS		0.58 (0.11)
2	WS+	0.50 (0.51)	0.55 (0.13)	3.01 (0.60)	3.32 (0.62)
	CAT	0.60 (0.50)	0.59 (0.12)	2.99 (0.77)	3.23 (0.57)
	SS	0.45 (0.51)	0.52 (0.10)	2.95 (0.76)	3.14 (0.54)
4	WS+	0.42 (0.51)	0.57 (0.12)	3.48 (0.31)	3.64 (0.32)
	CAT	0.44 (0.51)	0.62 (0.15)	3.86 (0.11)	3.82 (0.21)
	SS	0.62 (0.50)	0.57 (0.16)	3.18 (0.40)	3.19 (0.70)
6	WS+	0.41 (0.51)	0.61 (0.15)	3.28 (0.65)	3.38 (0.71)
	CAT	0.71 (0.47)	0.68 (0.13)	3.22 (0.64)	3.26 (0.68)
	SS	0.55 (0.51)	0.67 (0.14)	3.18 (0.84)	3.18 (0.56)
8	WS+	0.30 (0.48)	0.58 (0.10)	2.87 (1.48)	2.94 (0.90)
	CAT	0.67 (0.49)	0.67 (0.12)	3.72 (0.31)	3.64 (0.43)
	SS	0.65 (0.49)	0.64 (0.15)	3.08 (1.06)	3.15 (0.80)
10	WS+	0.56 (0.53)	0.55 (0.16)	2.90 (0.75)	3.18 (0.45)
	CAT	0.67 (0.49)	0.66 (0.17)	3.32 (0.69)	3.26 (0.86)
	SS	0.50 (0.52)	0.60 (0.17)	3.15 (0.65)	3.06 (0.55)
12	WS+	0.57 (0.54)	0.65 (0.14)	3.57 (0.35)	3.54 (0.38)
	CAT	0.58 (0.52)	0.71 (0.14)	3.05 (0.80)	3.31 (0.62)
	SS	0.56 (0.53)	0.71 (0.15)	2.92 (1.04)	3.10 (0.34)

Note. WS+ = workshop and resources; CAT = WS+ and computer assisted training; SS = WS+/CAT and supervisory support; CM = Contingency Management; CB = Cognitive Behavioral; MON = Monitoring.

^*Proportion of therapists that used CM.

Table 3.

Analyses of Training Intervention Effects on CM Use, Knowledge, and Implementation

CM Use	β	SE	df	p	OR	95% CI
Intercept
WS+	−3.31	0.39	106	<.001	0.04	0.02, 0.08
CAT vs. WS+	0.72	0.61	106	.238	2.05	0.62, 6.82
SS vs. WS+	−0.26	0.55	106	.639	0.77	0.26, 2.28
Linear Slope (Months)a
WS+	0.17	0.05	241	.001	1.18	1.07, 1.31
CAT vs. WS+	−0.04	0.08	241	.624	0.96	0.83, 1.12
SS vs. WS+	0.00	0.08	241	.950	1.00	0.87, 1.17
	Est.b	SE	χ2	p
Intercept
CAT v. SS	−0.98	0.61	2.59	.104
Linear Slope (Months)
CAT v. SS	0.04	0.08	0.29	>.500
Linear Slope ≠ 0
CAT	0.13	0.06	4.89	.025
SS	0.17	0.05	10.01	.002
CM Knowledge	β	SE	df	p		95% CI
Intercept
WS+	54.88	0.51	158	<.001		53.88, 55.89
CAT vs. WS+	2.04	0.68	158	.003		0.70, 3.38
SS vs. WS+	0.03	0.74	158	.967		−1.43, 1.49
Linear Slope (Months)c
WS+	0.07	0.08	158	.402		−0.09, 0.23
CAT vs. WS+	0.13	0.12	158	.282		−0.11, 0.37
SS vs. WS+	0.18	0.11	158	.118		−0.04, 0.40
	Est.b	SE	χ2	p
Intercept
CAT v. SS	2.01	0.70	8.13	.005
Linear Slope (Months)
CAT vs. SS	−0.05	0.12	0.17	>.500
Linear Slope ≠ 0
CAT	0.20	0.09	4.95	.024
SS	0.25	0.08	9.57	.002
CM Implementation - MON	β	SE	df	p		95% CI
Intercept
WS+	2.17	0.11	96	<.001		1.95, 2.39
CAT vs. WS+	−0.17	0.18	96	.340		−0.52, 0.18
SS vs. WS+	−0.14	0.17	96	.425		−0.47, 0.20
Linear Slope (Months)a
WS+	0.10	0.02	126	<.001		0.07, 0.14
CAT vs. WS+	0.01	0.03	126	.793		−0.05, 0.06
SS vs. WS+	0.02	0.03	126	.426		−0.03, 0.07
	Est.b	SE	χ2	p
Intercept
CAT v. SS	−0.03	0.19	0.03	>.500
Linear Slope (Months)
CAT v. SS	−0.01	0.03	0.28	>.500
Linear Slope ≠ 0
CAT	0.11	0.02	33.76	<.001
SS	0.13	0.02	43.74	<.001
CM Implementation - CB	β	SE	df	p		95% CI
Intercept
WS+	2.81	0.10	96	<.001		2.61, 3.01
CAT vs. WS+	−0.25	0.15	96	.095		−0.54, 0.04
SS vs. WS+	−0.15	0.16	96	.361		−0.46, 0.17
Linear Slope (Months)a
WS+	0.05	0.02	126	.004		0.02, 0.09
CAT vs. WS+	0.02	0.02	126	.480		−0.03, 0.06
SS vs. WS+	0.00	0.02	126	.903		−0.04, 0.05
	Est.b	SE	χ2	p
Intercept
CAT v. SS	0.11	0.16	0.41	>.500
Linear Slope (Months)
CAT v. SS	−0.01	0.02	0.42	>.500
Linear Slope ≠ 0
CAT	0.07	0.01	24.83	<.001
SS	0.06	0.02	13.75	<.001

Note. WS+ = workshop and resources; CAT = WS+ and computer assisted training; SS = WS+/CAT and supervisory support; CM = Contingency Management; CB = Cognitive Behavioral; MON = Monitoring.

^aFixed effect only for linear slope.

^bEstimates and significance tests obtained using multivariate hypothesis tests as part of the model specification in HLM.

^cFixed and random effects for linear slope.

Findings of increased CM use across the training conditions have clinical significance. At baseline, only seven (9%) of the 79 therapists with substance abusing adolescents in their caseload reported the use of CM. By the 12-month follow-up, however, 65 (71%) of the 92 therapists that had at least one substance-abusing youth reported using CM at least once. Approximately half of these therapists reported using CM with only 1 or 2 youth, 14% with 3 youth, 8% with 4 youth, 6% with 5 youth, and the remaining 24% used CM with 6 or more youth. Thus, during the 12 months following the workshop, the number of therapists using CM increased more than seven-fold, with more than 250 substance abusing adolescents receiving CM.

An examination of therapist uptake after giving CM an initial try is also clinically informative. Across assessments and intervention conditions, therapists treated 45% of their substance-abusing youth clients with CM on average, assuming that a maximum of three of these clients could be treated at any single assessment (please recall that therapists had resources [i.e., vouchers, test cups] to treat six youth with CM during the 12-month study period). For those therapists who used CM at least once, 60% of their substance-abusing youth were treated across assessment periods. Interestingly, however, once a therapist used CM, he or she proceeded to implement the model with 69% of his or her substance-abusing youth.

3.3. CM Knowledge

Based on the full sample of 161 therapists, participants showed an increase on the knowledge test from baseline to post workshop, β = 4.10, SE = 0.42, t (160) = 9.84, p < .001, 95% CI = 3.28, 4.92 (please recall that organizations were not randomized to training conditions until after the workshop). In addition, as shown in Tables 2 and 3, the increase in knowledge continued for participants in the WS+/CAT and WS+/CAT/SS conditions through 12 months, but remained flat for the WS+ participants. The three slopes, however, did not differ significantly from each other.

In terms of practical significance, the percentage of correct responses to the CM knowledge test increased from approximately 51% at baseline to 60% post workshop to 69% at the 12-month follow-up. Moreover, as might be anticipated, secondary analyses revealed that therapists who treated a larger number of youth with CM between months 2 and 12 showed a significantly more positive slope in knowledge, β = 0.05, SE = 0.02, t (159) = 2.39, p = .018, 95% CI = 0.01, 0.09 – suggesting that practice in CM techniques resulted in improved knowledge or vice versa.

3.4. CM Implementation

CM implementation adherence was assessed for those 65 therapists who treated at least one substance-abusing adolescent with CM during the 12-month follow-up. Across training conditions, therapists reported an increase in CM Cognitive Behavioral and Monitoring adherence from baseline through the 12-month follow-up. Again, however, the slopes of the three conditions did not differ. Moreover, in contrast with the aforementioned finding for CM knowledge, secondary analyses did not show that therapists who treated a larger number of youth with CM between months 2 and 12 had improved adherence scores.

The clinical significance of the adherence scores can be evaluated, in part, through comparison with results observed in a recent clinical trial with juvenile drug courts using the same CM intervention model (Henggeler, McCart et al., 2012; McCart et al., 2012). A caveat of this comparison, however, is that adherence scores in the present study were based on therapist report, whereas scores in the drug court study were based on caregiver and youth report. Cognitive Behavioral scores were similar between the two studies: M = 2.85 (sd = .61) for the present study, and Ms = 2.75 (sd = .68) and 2.82 (sd = .58) for youth and caregiver reports, respectively, in the juvenile drug court study. Monitoring scores, however, were about a half standard deviation lower in the present study than in the drug court study: M = 2.40 (sd = .82) for present study, and Ms = 2.84 (sd = .69) and 2.70 (sd = .62) for youth and caregiver reports, respectively, in the juvenile drug court study. It should be noted, however, that the latter means are likely enhanced because monitoring of substance use is a standard practice of drug courts.

3.5. Moderator Analyses

Analyses were conducted to determine whether the positive change trajectories observed for CM use, knowledge, and implementation adherence were moderated by any of the 11 demographic and professional characteristics presented in Table 1. As only three moderators were significant at the p<.05 level, and two were expected by chance given the 44 separate analyses; it seems reasonable to conclude that meaningful moderation effects were not observed. Thus, for example, therapists from the mental health service sector reported similar increases in CM use, knowledge, and adherence of implementation as counterparts from the substance abuse service sector, and the increases of white therapists were similar to those of African-American therapists.

4. Discussion

Although unanticipated, the results were consistent and have encouraging implications for the large-scale transport of CM to practitioners in public sector substance abuse and mental health provider agencies. In general, based on group slopes, therapists in each training condition showed increased CM use, knowledge, and adherence of implementation from baseline through the 12-month follow-up. The only exception was that therapists who received CAT showed significant gains in knowledge from post workshop to the 12-month follow-up, whereas practitioners in the WS+ condition did not. Between-groups differences in slopes were not observed, however.

A combination of several training features might explain why WS+ was as effective as its more complex counterparts at increasing CM use, knowledge, and implementation adherence. First, the content and style (i.e., didactic instruction, trainer role-play, and dyadic role plays for each therapist) of the workshop have been sharpened and polished through numerous research-based iterations during the past decade. Thus, consistent with consumer ratings, the workshop was likely of high quality and included established characteristics of effective training (Daniels & Walter, 2002). Second, the WS+ condition included ready access to the resources needed to adopt and implement CM. These resources included a treatment manual that had received extensive revision over the course of numerous research studies with session-by-session checklists, therapist scripts, worksheets, and handouts for families; test kits for detecting substance use; and vouchers for rewarding abstinence. Thus, the WS+ condition included much more than simply workshop attendance – the practical resources to facilitate CM adoption were readily available. Third, agency colleagues and leadership supported the project. Virtually all adolescent-treating therapists in the agencies attended a workshop and participated in the research, and agency leadership cooperated fully with training (e.g., providing time off for workshop attendance) and research arrangements. Moreover, organizational data (i.e., the Program Needs subscale of the Organizational Readiness to Change; Lehman, Greener, & Simpson, 2002) collected for a concurrent study showed significant (ps <.002) decreases in therapist reported training needs from baseline to the 12-month follow-up for each training condition. Thus, we hypothesize that the combination of a strong workshop experience, ready access to implementation resources, and organizational support for CM adoption led many therapists (i.e., 71%) to try CM. Further, as noted in the results section and consistent with diffusion of innovation’s “trialability” concept (Rogers, 2003), attempts to use CM with their substance-abusing adolescents were associated with increased therapist knowledge about the approach and an increased likelihood of using CM again.

Why, in contrast with expectations, didn’t the more intensive training conditions add value to WS+? If we assume that the aforementioned strengths of WS+ training are valid, then the inclusion of CAT might not be sufficiently additive to impact CM use and implementation adherence (though some evidence supported the ability of CAT to increase CM knowledge). Regarding the addition of the supervisory component (i.e., WS+/CAT/SS), there is little doubt that the inclusion of intensive supervision and quality assurance has been critical to the successful transport of complex evidence-based treatments to community settings (Fixsen et al., 2005). Such transport, however, requires reasonable stability of organizational and supervisory structures. Unfortunately, organizational stability and effective supervisory structures are often challenged in public sector substance abuse and mental health provider organizations (Aarons & Sawitzky, 2006; Knudsen, Johnson, & Roman, 2003; McLennan, Carise, & Kieber, 2003). Such was also the case here, as evidenced by the high practitioner turnover in the participating organizations (i.e., 32% turnover between baseline and 12-month follow-up) and the aforementioned struggles experienced by agency supervisors. Thus, as suggested for CAT, the addition the supervisory component might not have been sufficient to significantly enhance the effects of WS+.

With certain caveats, which generally pertain to the limitations of the study noted subsequently, these findings might have important implications for the large-scale transport of CM to community-based providers. The provision of high quality workshops, ready access to implementation resources, and CAT (because of its low cost and possible effect on knowledge) or a stepwise approach based on therapist performance following each step (Martino et al., 2011) might be efficient and inexpensive strategies for public sector service systems to increase the use of evidence-based interventions that are not highly complex. This contention is supported by several sets of findings in the literature. First, our previous research (Henggeler et al., 2008) demonstrated widespread public sector interest in the adoption of CM, with 58% of the practitioners in that study and 71% in the present study, including therapists in the mental health sector, using CM to treat a substance-abusing adolescent in their caseload. Second, adherence for the cognitive behavioral component of CM increased to a level observed in a recent clinical trial (Henggeler, McCart et al., 2012), and adherence for the monitoring component was about a half sd lower than in that trial. Third, investigators (e.g., Henggeler, McCart et al., 2012, Petry et al., 2012) have demonstrated that community-based practitioners are fully capable of implementing CM effectively (i.e., achieving favorable client outcomes). Fourth, Martino et al. (2011) found that provision of a treatment manual and CAT was sufficient for 50% of the substance abuse treatment counselors in their sample to reach an adequate standard of performance for a similarly complex evidence-based treatment (i.e., motivational interviewing). Likewise, Sholomskas et al., (2005) found that therapists who received a cognitive-behavioral treatment manual and CAT showed larger (e.g., mean effect size = .88) gains in treatment adherence and skill compared with counterparts who received only the treatment manual. Thus, the emerging literature supports a view that public sector practitioners are willing to adopt evidence-based treatments of substance abuse, and that relatively efficient methods can be used to achieve modest gains in the quality of treatment implementation.

Several limitations of the study and our conclusions should be noted. First, because client-level outcomes were not assessed, it is not known whether increases in the quality of therapist CM implementation were sufficient to achieve improved clinical outcomes. Second, the measurement of CM use and implementation adherence were based solely on therapist self-report, and investigators have found that therapists tend to over estimate their use of elements of evidence-based treatments (e.g., Beidas & Kendall, 2010; Carroll, Martino, & Rounsaville, 2010). Thus, the rates of CM use and degree of adherence reported in the present study might be overly optimistic. Third, the participating therapists and provider organizations were volunteers, and it is not known whether a random or population-wide sample would have been as responsive to the training opportunity. Fourth, the findings should not necessarily be generalized to other evidence-based treatments of substance abuse, especially those that are more intensive and complex. Finally, the therapists in the study were nested in provider organizations; however, there were too few organizations to model this level of nesting. Therefore, the effect of the training conditions was modeled at the level of therapists.

In conclusion, consistent with findings emerging in the field, the present results suggest that community-based practitioners are amenable to the adoption of evidence-based treatments of substance abuse when provided access to useful resources. Moreover, high quality workshops in combination with resource access can increase knowledge of the evidence-based treatment and might enhance intervention adherence to a level needed to improve youth outcomes. This latter aim, however, requires rigorous experimental testing (i.e., evaluation in randomized clinical trial).