Predictive validity of addiction treatment clinicians’ post-training contingency management skills for subsequent clinical outcomes

Abstract

In the context of a contingency management (CM) implementation/effectiveness hybrid trial, the post-training implementation domains of direct-care clinicians (N=19) were examined in relation to a targeted clinical outcome of subsequently CM-exposed clients. Clinicians’ CM skillfulness, a behavioral measure of their capability to skillfully deliver the intended CM intervention, was found to be a robust and specific predictor of their subsequent client outcomes. Analyses also revealed CM skillfulness to: 1) fully mediate an association between a general therapeutic effectiveness and client outcome, 2) partially mediate an association of in-training exposure to CM and client outcome, and 3) be comprised of six component clinical practice behaviors that each contributed meaningfully to this behavior fidelity index. Study findings offer preliminary evidence of the predictive validity of post-training CM skillfulness for subsequent client outcomes, and inform suggestions for the design and delivery of skills-focused CM training curricula for the addiction treatment workforce.

1. Introduction

The transport of empirically-supported behavior therapies for routine use in addiction care settings remains a challenge. This shared undertaking involves the therapy purveyors who promote and train others to use particular therapies, community treatment programs as intended settings wherein those therapies would be implemented, and the treatment programs’ direct-care clinicians who would serve as primary agents of therapy implementation (Beidas, Koerner, Weingardt, & Kendall, 2011). Dissemination efforts necessarily rely on purveyors to make information available about a focal therapy so community treatment programs and their staff are aware of its conceptual basis and empirical support (Hartzler & Rabun, 2014). On this front, the field is progressing. Increased awareness of empirically-supported therapies in community settings has been fueled by large-scale efforts funded by the Substance Abuse and Mental Health Services Administration, like Addiction Technology Transfer Centers (www.nattc.org) and a National Registry of Evidence Based Programs and Practices (www.nrepp.samhsa.gov). Likewise, the findings of multisite effectiveness studies conducted in the National Institute on Drug Abuse Clinical Trials Network (Hanson, Leshner, & Tai, 2002) have promoted greater community awareness of empirically-supported therapies. However, greater awareness alone appears insufficient to prompt effective transport of promising behavior therapies to community settings, as adoption rates for a range of empirically-supported practices remain modest (Knudsen, Abraham, & Roman, 2011; Roman, Abraham, Rothrauff, & Knudsen, 2010).

Beyond treatment community awareness, challenges to behavior therapy dissemination include any number of issues at the level of individual direct-care clinicians. A prominent issue involves fidelity, or clinician capability to skillfully deliver a therapy as intended (McHugh & Barlow, 2010) and thereby offer therapy-exposed clientele an opportunity to approximate the therapeutic outcomes reported in corresponding efficacy trials. Efforts to prepare direct-care clinicians to effectively deliver such behavior therapies encompass a broader struggle across the healthcare field with questions about sufficient therapy promotion, training, and supervision methods (Beidas & Kendell, 2010; Herschell, Kolko, Baumann, & Davis, 2010). Answers to such questions may implicate clinicians’ professional background (i.e., educational attainment, program role, setting tenure), level of exposure to the therapy via structured training or other means, and impact of such therapy exposure on clinician-level implementation domains (i.e., conceptual knowledge of therapy principles, self-efficacy to skillfully deliver the therapy, attitudes toward therapy-specific precepts, adoption readiness). Unfortunately, the relative weight to be given these clinician-level implementation domains in structured dissemination efforts is ambiguous. This places therapy purveyors in a challenging position about where to focus time-limited efforts when engaging direct-care clinicians in therapy training processes.

One widely-studied behavior therapy for substance abusers is contingency management (CM), encompassing a family of approaches informed by principles of behavioral reinforcement (Higgins, Silverman, & Heil, 2008). Petry (2012) notes as core CM tenets that: 1) a focal, desired patient behavior be monitored, 2) timely provision of tangible, positive reinforcers when the behavior occurs, and 3) withholding of reinforcers when the behavior does not occur. Meta-analyses show reliable efficacy across procedurally-diverse CM methods (Benishek et al., 2014; Lussier, Heil, Mongeon, Badger, & Higgins, 2006; Prendergast, Podus, Finney, Greenwell, & Roll, 2006). Yet, community treatment interest in CM is limited relative to alternative behavior therapies (Haug, Shopshire, Tajima, Gruber, & Guydish, 2008; McGovern, Fox, Xie, & Drake, 2004). Interest in CM appears moderated by clinician background variables, with greater educational attainment, a supervisory setting role, and lengthier employment tenure all predictive of greater interest (Aletraris, Shelton, & Roman, in press; Hartzler et al., 2012; Kirby, Benishek, Dugosh, & Kerwin, 2006; McCarty et al., 2007). Further, exposure to CM seems to encourage clinician interest in its adoption (Aletraris et al., in press; Ducharme, Knudsen, Abraham, & Roman, 2010). Nevertheless, there is much still to learn about how such interest translates into consistently skillful delivery of CM interventions that improve targeted clinical outcomes.

Extant literature from the addiction treatment community on impacts of CM training for direct-care clinicians is limited with respect to implementation domains (i.e., therapy knowledge, skillful delivery, attitudes, self-efficacy, adoption readiness). A nationwide training effort with VA program leaders suggests multi-day workshop exposure promotes knowledge and conducive attitudes (Rash, DePhillipis, McKay, Drapkin, & Petry, 2013), though unfortunately excluded direct-care clinicians from those VA programs. Success of later VA implementation attempts, aided by federal funding support and presumably involving untrained direct-care clinicians, also is unknown (Petry, DePhillipis, Rash, Drapkin, & McKay, 2014). More is known from a single-site trial in which Hartzler and colleagues’ (2014) demonstrated a skills-focused training with an intact group of direct-care clinicians led to robust, durable improvements in CM delivery skill, knowledge, and adoption readiness. That all of the CM-trained clinicians in this single-site trial met and maintained a suggested competency benchmark for delivery skill is encouraging, and offers a unique opportunity to examine questions about effective CM dissemination via a set of exploratory analyses. One such question concerns the extent to which CM delivery skill and other post-training clinician-level implementation domains may predict clinical improvement in targeted outcomes of CM-exposed clientele. Another question is how clinician background attributes, broader clinical acumen, and level of therapy exposure contribute to development of post-training therapy expertise. Answers to such questions are paramount to understanding of effective CM dissemination processes, given published report of well-intentioned yet ultimately unsuccessful implementation attempts undermined or discontinued after community settings encountered clinician-involved logistical, procedural, or philosophical challenges (Tuten, Svikis, Keyser-Marcus, O’Grady, & Jones, 2012; Walker et al., 2010).

Design features of the aforementioned trial by Hartzler and colleagues (2014) allow for further examination of these links between clinicians’ post-training implementation domains and subsequent therapeutic effects among their CM-exposed clientele. In this trial, direct-care clinicians were afforded voluntary opportunity to implement a contextualized intervention with their caseload clients over a 90-day period immediately following the CM training. Thus, trial data enable scrutiny of the predictive validity of post-training implementation domains for subsequent clinical outcomes among the CM-exposed clientele. Further, the trial also gathered baseline data about the direct-care clinicians, allowing additional examination of the extent to which their background attributes (i.e., educational attainment, program role, setting tenure, common psychotherapy skills) influence such associations. Likewise, available clinician data also specified their level of CM exposure via a range of potential professional activities prior to training and their attendance rate during the identified training. Herein, we examine the interplay of this collection of clinician variables in predicting clinical effectiveness of a CM intervention.

2. Materials and method

2.1 Parent Trial Design

All trial procedures were approved by a university-based institutional review board, and these as well as trial outcomes are comprehensively described elsewhere (Hartzler et al., 2014). As for design, an implementation/effectiveness hybrid ‘type 3’ trial (Curran, Bauer, Mittman, Pyne, & Stetler, 2012) was modeled insofar as the primary focus was formal testing of a set of implementation strategies for a contextualized CM intervention with secondary focus on the corresponding clinical effectiveness later observed during setting implementation. Specific trial design features included voluntary recruitment of the setting’s direct-care clinicians to participate in a CM training process; serial training outcome assessments completed by these clinicians prior to, just after, and three months following training; and clinician delivery of a contextualized CM intervention with targeted clients on their caseload over a 90-day provisional implementation period. Among the previously-reported trial outcomes are robust immediate impacts of training on clinicians’ CM delivery skill, knowledge, adoption readiness, and attitudes as well as significant clinical effects of the CM intervention on targeted outcomes (d = .45–.53, relative to historical controls) among the setting’s CM-exposed clientele (Hartzler et al., 2014).

In the current report, the analytic work was exploratory in nature yet guided by a set of working hypotheses. First, post-training clinician implementation domains (outlined later in Materials and Method) were expected to be positively associated with the aggregated rate of counseling visit attendance (the target behavior for reinforcement) among CM-exposed clients on clinicians’ caseload. Second, it was anticipated that clinicians’ background attributes (i.e., education, program role, employment tenure, common psychotherapy skills) and level of CM exposure would be positively associated with their post-training implementation domains and the caseload-aggregated attendance rate of CM-exposed clients. Third, it was hoped these collective analyses may help explain interplay among clinician background attributes, CM exposure, and post-training implementation domains in ultimately influencing clinicians’ caseload-aggregated attendance rate. Results of these collective analyses prompted subsequent examination of the contribution of CM-specific clinical practice behaviors to clinicians’ overall CM skillfulness.

2.2 Collaborating Treatment Setting

The collaborating setting is a private, non-profit opiate treatment program (OTP) located in an urban area of a large northwestern United States city. The OTP has provided medication-assisted treatment to a diverse population of clientele for more than four decades, and maintains a census of approximately 1000 clients. In addition to opiate agonist medication, other core clinical services include individual and group counseling, case management and support services, access to psychiatric/medical care, and monthly drug screen urinalysis. The setting is affiliated with the NIDA Clinical Trials Network, and prior to its involvement in the parent trial had participated in multisite trials of pharmacotherapies, alternative behavior therapy approaches, and a range of other federally-funded research.

2.3 Contextualized CM Intervention

The CM intervention and its collaborative design process are fully described elsewhere (Hartzler, 2015; Hartzler et al., 2014). Briefly, the CM purveyor oriented the OTP director to aforementioned CM tenets, and invited designation of intervention features according to setting needs, interests, and operating budget. Director-designated features included: 1) a target client population in their initial 90 days of services, 2) a target behavior of attendance at weekly counseling visits, 3) $5 gift cards to local vendors as behavioral reinforcers, and 4) a ‘point-based’ reinforcement system that empowered clients to select specific earned reinforcers and timing of their receipt. With these features in place, the CM purveyor outlined a reinforcement schedule specifying the rate at which clients would have opportunity to earn points. To enhance clinical impacts, this reinforcement schedule integrated priming and escalation/reset elements such that clients earned bonus points for initial and consecutively-attended counseling visits. During an eventual 90-day period of provisional setting implementation, trained clinicians had opportunity to deliver the intervention with eligible clients on their caseload. Thus, direct-care clinicians monitored the target behavior, tracked points, and delivered reinforcers amidst usual care in these weekly counseling visits. In evaluating clinical impacts, a caseload-aggregated attendance rate was computed for each clinician encompassing all of their CM-exposed clientele. To track intervention delivery, the electronic medical record was adapted to enable clinicians to document point totals (and reinforcers provided) in all visit notes for CM-exposed OTP clientele.

2.4 CM Training Process

The parent trial’s training process was informed by preferences reported by a national sampling of OTP programs (Hartzler & Rabun, 2014). These include: 1) scheduling of on-site training sessions to limit fiscal and logistical barriers, 2) a temporally-distributed format, with shorter sessions spaced a week apart for incremental learning, and 3) a skills-focused curriculum guided by active learning strategies to develop targeted clinical practice behaviors via trainer demonstration, experiential trainee role-plays, and performance-based trainer feedback. The 16-hour CM training was structured as four weekly half-day sessions, all occurring at the OTP in a large group room allowing fishbowl observational method for trainer demonstrations and dyadic trainee role-plays. Training was facilitated by two psychologists, each with CM expertise and OTP experience. Prior to each training session, the trainers met with the OTP’s managerial staff for 30 minutes as consultative support for preparatory activities (i.e., identification of local implementation leaders, revision of patient orientation materials, adaptation of electronic medical record system, purchase and accounting of reinforcers). At training conclusion, copies of all training materials (i.e., handouts, role-play instructions, training session audio-recording) were compiled as a ‘CM library’ and provided to the OTP as a continuing setting resource.

2.5 Measures

2.5.1 Clinician Background Attributes

A pre-training survey enabled direct-care clinicians to self-report their demography and aspects of professional background like educational attainment, program role, and employment tenure. Indices of clinician demography included age, gender, ethnicity, and race. Educational attainment reflected five potential levels (i.e., high-school diploma/GED, Associate Degree (i.e., AA), Bachelor Degree (i.e., B.A., B.S.), Master Degree (i.e., M.A., M.S., M.S.W, M.P.H), and Doctoral Degree (i.e., Ph.D., M.D.). Program role was simplified to a binary scale based on primary functions typically performed (i.e., supervisory, direct-service). Employment tenure was the number of years (to the month) of continuous employment in the OTP setting.

2.5.2 Clinician Exposure to CM

This was subcategorized into two variables, pre-training CM exposure and in-training CM exposure. Pre-training CM exposure was the sum of affirmative clinician responses to six binary items (yes, no) on a pre-training survey of prior CM exposure via reading of seminal textbooks or journal articles, review of the Promoting Awareness of Motivational Incentives blending product, or attendance at a presentation, brief training, or multi-day workshop. In-training CM exposure was the number of hours (up to 16) each clinician attended the identified CM training process, as documented by trainers via attendance lists kept at weekly training sessions. As direct-care clinicians were recruited for voluntary participation in the parent trial, attendance of training was not mandated and did vary somewhat in the sample.

2.5.3 Post-Training Clinician Implementation Domains

All clinician implementation domains incorporated in the current report’s analyses were components of the parent trial’s training outcome assessment occurring just after conclusion of training. To assess CM delivery skill, standardized patient (SP) interactions were conducted wherein an actor presented at clinicians’ offices to enact a 20-minute, audio-recorded encounter to portray a newly-enrolled OTP client. Two independent raters reviewed all SP encounters, scoring each with the Contingency Management Competence Scale (Petry, Alessi, Ledgerwood, & Sierra, 2010; Petry & Ledgerwood, 2010). The CMCS contains nine Likert-rated items (1 = Very Poor, 7 = Excellent), six of which reflect CM-specific clinical practice behaviors: 1) notifying of earned reinforcement, 2) planning for prospective reinforcement, 3) delivering earned reinforcers, 4) assessing client interest in reinforcers, 5) providing social reinforcement, and 6) linking the reinforced behavior to broader client treatment goals. The three remaining CMCS items each represented a common psychotherapy skill. One was general therapeutic effectiveness, or skill with which the clinician engaged the client and enacted well-conceived, well-timed therapeutic interventions. A 2^nd common psychotherapy skill was session structure, or skill in sequencing the session to include CM intervention procedures and comfortably transition to other therapeutic foci. A 3^rd common psychotherapy skill was empathy, or skill in communicating understanding of the client perspective. Inter-rater reliability for all nine CMCS items was excellent (intra-class correlations = .77–.89). Accordingly, each of the three common psychotherapy skill items (i.e., general therapeutic effectiveness, session structure, empathy) was retained for analyses, and a CM skillfulness summary score was computed as a sum of the six CM-specific items (range: 6–42).

Other implementation domains were assessed by paper/pencil methods. For conceptual knowledge of CM principles and practices, clinicians completed an adapted 18-item multiple-choice knowledge test (Petry & Stitzer, 2002), for which adequate internal consistency (α = .76) led to computation of a CM knowledge summary score. To assess CM attitudes, they completed the Provider Survey of Incentives [PSI; (Kirby et al., 2006)] with its 42 ratings (1 = Strongly Disagree, 5 = Strongly Agree) that form positive and negative subscales (α = .83–.90). For adoption readiness, a Readiness-to-Adopt Scale (McGovern et al., 2004) consisting of a single, six-point item (0 = Not Familiar, 5 = Using With Efforts To Maintain) was completed. To assess self-efficacy to implement, clinicians completed a scale consisting of 15 Likert-rated items of their confidence (1 = Not At All, 7 = Extremely) to perform CM implementation tasks. Strong internal consistency (α = .95) led to computation of a self-efficacy summary score.

2.6 Participants

At the time of their recruitment, all 19 direct-care clinicians were active in delivery of clinical services at the OTP. The mean age was 59.32 years (S.D. = 12.73). The large majority (89%) were female. Hispanic ethnicity was identified by 5%, and distribution of race was 79% Caucasian, 16% Multi-racial, and 5% Native American. In terms of their educational attainment, 58% of these direct-care clinicians had masters, 26% bachelors, and 16% associate degrees. As for program role, 26% performed supervisory functions in the OTP setting. Mean employment tenure in the sample was 12.24 years (S.D. = 9.72). With respect to prior CM exposure, 11% had attended a presentation, 31% had reviewed published or online materials, 27% noted both types of prior CM exposure, and 31% reported no prior CM exposure. In-training CM exposure during the described training process varied from 6–16 hours (M = 12.00, S.D. = 2.91).

In terms of post-training implementation domains, summary scores of CM skillfulness (M = 33.06, S.D. = 3.29) were strong in this sample with each direct-care clinician exceeding a competency benchmark proposed by the CMCS instrument originators (Petry et al., 2010). As for CM knowledge test scores (M = 15.76, S.D. = 1.82), strong understanding of CM principles and practices was suggested across this clinician sample. Respective mean positive and negative attitudinal scores (M = 3.91, S.D. = .50; M = 2.16, S.D. = .48) suggested setting presence of strongly pro-adoption perspectives, as did the modal response on the adoption readiness scale indicating the most clinicians were “leaning in the direction of adopting.” Finally, the mean summary score for the self-efficacy to implement CM scale (M =6.29, S.D. = .59) suggested that this sample of direct-care clinicians exited their training process with confidence that they could capably implement the CM intervention with new clients assigned to their caseloads during the subsequent 90-day period of provisional implementation.

2.7 Analytic Strategy

Nesting considerations (e.g., therapy attendance of clients nested in clinician caseloads) were inherent in a subset of planned analyses, prompting computation of a design effect based on variance estimates previously reported in the parent trial (Hartzler et al., 2014). This design effect (1.15) was a product of: 1) an intraclass correlation (.03) computed from disaggregated variance estimates at the level of clinicians (.003) and clients (.088), and 2) the average number of clients per clinician caseload (5.58). According to Muthen and Satorra (1995), the value of this design effect is quite low—indicating use of a sophisticated multilevel analytic approach, such as hierarchial linear modeling, is unnecessary. Instead, simpler correlation and linear regression analyses were conducted using the Statistical Package for the Social Sciences (SPSS), Version 21. Analyses were exploratory, and de-emphasized attention to statistical significance due to limited statistical power afforded by the small size of this clinician sample. Accordingly, a conservative a priori statistical criterion of .30 was set to initially identify correlations and standardized regression coefficients of salient constructs, a value corresponding to Cohen’s (1988) designation of a medium-strength association. In a data reduction approach intended to limit the number of predictors in any one regression, bivariate correlations were first examined with eventual multivariate regression analyses including only predictors meeting this a priori statistical criterion. To elucidate how clinicians’ background attributes, CM exposure, and post-training implementation domains operate to influence their caseload-aggregated client attendance rates, Baron and Kenny’s (1986) multi-step mediational testing process was undertaken. The CM skillfulness summary score figured prominently in these analyses, prompting later computation of CMCS item-scale and inter-item correlations to determine the relative contributions of component clinical practice behaviors.

3. Results

3.1 Post-training Implementation Domains Predicting Clinical Outcome

Only CM skillfulness demonstrated an association to the clinical outcome, caseload-aggregated attendance rates, that exceeded the a priori statistical criterion. When included in a regression analysis as the sole predictor of this clinical outcome, a standardized beta of .50 was found (see Table 1). This precipitated analyses to identify correlates of CM skillfulness from among clinician background attributes, CM exposure, and common psychotherapy skills.

Table 1.

Post-Training Clinician Implementation Domains as Predictors of Client Attendance

Predictor	Zero Order Correlation	Regression
	r	Standardized beta
Conceptual CM Knowledge	.19	--
Positive Attitudes Toward CM	−.12	--
Negative Attitudes Toward CM	.29	--
Self-efficacy To Implement CM	−.03	--
Adoption Readiness	.07	--
CM Skillfulness	.50	.50 (.05, .94)

Data based on 19 CM-trained direct-care clinicians; CM skillfulness reflects a summary score of six CM-specific intervention delivery skills, as scored by independent raters via Contingency Management Competence Scale; Client attendance based on caseload-aggregated rate of weekly individual counseling visits among CM-exposed clients during a post-training 90-day provisional implementation period; Italicized correlation and beta coefficients are those that met an a priori statistical criterion (x = .30). The upper and lower limits of the 95% confidence interval follow the standardized beta value in parentheses.

3.2 Correlates of Clinicians’ Post-training CM Skillfulness

Zero-order correlations of five different clinician variables with post-training CM skillfulness met the a priori statistical criterion. In order of magnitude, these were: 1) general therapeutic effectiveness (r = .64), 2) empathy (r = .51), 3) in-training CM exposure (r = .44), 4) supervisory role (r = .40), and 5) employment tenure (r = .30). All of these associations were in the expected direction, with higher CM skillfulness associated with greater general therapeutic effectiveness and empathy, more hours of in-training CM exposure, a supervisory role in the setting, and a longer tenure of employment. In a set of three step 2 regression analyses (in which predictors were grouped into respective domains of clinician background, CM exposure, and common psychotherapy skills), the attributes for which standardized beta coefficients met the a priori statistical criterion were general therapeutic effectiveness (.63), in-training CM exposure (.44), and supervisory role (.35). With these three attributes subsequently included in a single step 3 regression analysis, the standardized beta coefficients for only general therapeutic effectiveness (.45) and in-training CM exposure (.34) met the a priori statistical criterion. Test statistics for this full sequence of regression analyses are outlined in Table 2.

Table 2.

Clinician Variables as Predictors of CM Skillfulness

Predictor	Step 1: Zero Order Correlation	Step 2: Within-domain Regressiona	Step 3: Between-domain Regression
	r	Standardized beta	Standardized beta
Clinician Background Attributes
Supervisory Role	.40	.35 (−.24, .94)	.20 (−.26, .66)
Educational Attainment	.26	--	--
Employment Tenure	.30	.10 (−.49, .69)	--
CM Exposure
Pre-training Exposure to CM	.27	--	--
In-training Exposure to CM	.44	.44 (−.02, .90)	.34 (−.07, .75)
Common Psychotherapy Skills
General Therapeutic Effectiveness	.64	.63 (−.20, 1.45)	.45 (−.02, .93)
Session Structure	.36	−.10 (−.76, .56)	--
Empathy	.51	.14 (−.46, .74)	--

Data based on 19 CM-trained direct-care clinicians; CM skillfulness reflects a summary score of six CM-specific intervention delivery skills, as scored by independent raters with Contingency Management Competence Scale; Italicized correlation and beta coefficients are those meeting an a priori statistical criterion (x = .30);

^aThree step 2 regressions were conducted, one for clinician background attributes, one for CM exposure, and one for common psychotherapy skills; One step 3 regression was conducted, involving the three indicated predictors from Step 2. The upper and lower limits of the 95% confidence interval follow the standardized beta values in parentheses.

3.3 Mediational Model Testing

Study analyses to this point had suggested the likely interplay between a post-training clinician implementation domain (i.e., CM skillfulness) and two other clinician variables (i.e., general therapeutic effectiveness, in-training CM exposure) in predicting caseload-aggregated attendance rates. This prompted investigator derivation of a corresponding hypothetical model of mediation. This hypothetical model specifically posited that both stronger general therapeutic effectiveness and more in-training CM exposure would result in greater CM skillfulness, which in turn would then result in a higher caseload-aggregated attendance rate among clinicians’ CM-exposed clientele. This hypothetical model was tested using the well-established multi-step analytic procedures established several decades ago by Baron and Kenny (1986).

Two of Baron and Kenny’s (1986) mediation testing steps are to examine associations of the mediator and outcome, and of the causal variable(s) and meditator. Analyses in the current study had to this point already demonstrated that: 1) CM skillfulness (the mediator) specifically predicted caseload-aggregated attendance rate (the outcome), and 2) both general therapeutic effectiveness and in-training CM exposure (two causal variables) predicted CM skillfulness (the mediator). Thus, analyses in the current report subsequently focused on the remaining two steps in Baron and Kenny’s (1986) mediation model testing procedure.

Table 3 shows analytic results for the 3^rd and 4^th steps in Baron and Kenny’s (1986) mediation model testing procedure. The 3^rd step is to examine the associations between causal variables and outcome, for which a regression specified standardized beta coefficient for in-training CM exposure (.36) and general therapeutic effectiveness (.24). The last of Baron and Kenny’s (1986) steps is to add the mediator to that regression, for purpose of examining to what extent strength of the associations of the causal variables and outcome change. As listed in Table 3, further inclusion of CM skillfulness in this regression: 1) substantively reduced the association of general therapeutic effectiveness and caseload-aggregated attendance rate (reduction of standardized beta coefficient from .24 to .04), and 2) somewhat reduced the association of in-training CM exposure and caseload-aggregated attendance rate (reduction of standardized beta coefficient from .36 to .25). Thus, CM skillfulness appears to fully mediate the influence of general therapeutic effectiveness on caseload-aggregated attendance rate in this sample. Further, CM skillfulness appears to partially mediate the influence of in-training CM exposure on caseload-aggregated attendance rate in this sample, as in-training CM exposure maintained both a direct effect on this clinical outcome as well as an effect mediated by CM skillfulness. The mediational model derived from these analyses is illustrated in Figure 1.

Table 3.

Direct and Mediated Effects of CM Skillfulness of Clinician Variables Predicting Client Attendance

Predictor	Regression 1	Regression 2
	Standardized beta (95% CI)	Standardized beta (95% CI)
CM Exposure
In-training CM Exposure	.36 (−.13, .84)	.25 (−.26, .76)
Common Psychotherapy Skill
General Therapeutic Effectiveness	.24 (−.24, .73)	.04 (−.57, .64)
Post-training Implementation Domain
CM Skillfulness	--	.36 (−.28, 1.01)

Data based on 19 CM-trained direct-care clinicians; Client attendance based on caseload-aggregated rate of weekly individual counseling visits among CM-exposed clients during a post-training 90-day provisional implementation period; In-training CM exposure was the number of hours (0–16) that the clinicians attended a focal CM training process; General Therapeutic Effectiveness is a single CMCS item reflecting clinician skill in engaging clients and enacting well-timed, well-conceived interventions; CM skillfulness reflects CMCS summary score of six CM-specific intervention delivery skills, as scored by independent raters. The upper and lower limits of the 95% confidence interval follow the standardized beta values in parentheses.

Figure 1.

Mediational Model.

3.4 Contribution of Component Clinical Practice Behaviors to CM Skillfulness

As CM skillfulness emerged as a robust predictor of the caseload-aggregated attendance rate among CM-exposed clientele, respective contributions of the six component clinical practice behaviors from the CMCS were examined. Accordingly, item-scale correlations and inter-item correlations were computed, both presented in Table 4. With respect to item-scale correlations, all six were fairly strong in magnitude—suggesting each of the six component clinical practice behaviors contributed meaningfully to clinicians’ demonstration of overall CM skillfulness. The item-scale correlations for five of six component clinical practice behaviors were quite strong (.72-.81). The remaining clinical practice behavior, clinician capability to accurately inform the client of earned reinforcement (.49), is conceptually simple and a restricted range at the top of the Likert rating scale was evident of clinician performances. Notably, the component clinical practice behavior with the most robust item-scale correlation was clinician capability to deliver earned reinforcement to clients in an encouraging and timely manner. In terms of the inter-item correlations, these varied in magnitude (range of .07 – .67) with a solid majority at or above Cohen’s (1988) designation for a medium-strength association (see Table 4).

Table 4.

Inter-item and Item-scale Correlations Among Component Clinical Practice Behaviors in CM Delivery

	1	2	3	4	5	6	7
1. CM skillfulness	--
2. Informing client of earned reinforcement	.49*	--
3. Informing client of prospective reinforcement	.72**	.28	--
4. Delivering earned reinforcement	.81**	.07	.51*	--
5. Assessing client interest in incentives	.71**	.54*	.45	.52*	--
6. Communicating social reinforcement	.77**	.38	.49*	.47*	.48*	--
7. Linking earned reinforcement to other treatment goals	.75**	.22	.34	.67**	.28	.44	--

Data based on 19 CM-trained direct-care clinicians; CM skillfulness is a summary score of the six listed component clinical practice behaviors listed as #2–7);

^*p < .05,

^**p<.01

4. Discussion

Broad dissemination of empirically-supported behavior therapies persists as a shared goal in the addiction treatment community, though one challenged by issues surrounding direct-care clinicians likely to serve as agents of therapy implementation in most settings. Current findings, based on data of a single-site implementation/effectiveness hybrid ‘type 3’ trial of CM (Hartzler et al., 2014), may help begin to clarify the interplay of clinicians’ background attributes, therapy exposure, and post-training implementation domains in predicting the clinical outcomes of their CM-exposed clients. These preliminary findings include: 1) specification of CM skillfulness among post-training implementation domains as a single, robust predictor of clinical outcomes, 2) strong association of a common psychotherapy skill (i.e., general therapeutic effectiveness) and clinician exposure to the therapy (i.e., in-training CM exposure) with CM skillfulness, 3) demonstration that CM skillfulness fully mediated a relation of general therapeutic effectiveness and clinical outcome, and partially mediated a relation of in-training CM exposure and clinical outcome, and 4) documentation of salient contribution by each of six component clinical practice behaviors to CM skillfulness. These preliminary findings carry implications for future efforts to disseminate CM, and perhaps other behavior therapies, to the addiction treatment community.

The emergence of post-training CM skillfulness, independently-rated during direct-care clinicians’ interaction with an SP, as a specific predictor of a targeted clinical outcome among their subsequently CM-exposed caseload clients is notable. Use of SPs is a longstanding practice in medical education, one for which technological advances continue to open new and exciting applications (Shah et al., 2012). Inclusion of SP interactions in behavior therapy training trials as a principal means to assess skill in therapy delivery is reasonably well-established (Baer et al., 2009; Sholomskas et al., 2005). This methodology affords several key benefits in such trials, in that the therapy purveyor retains stimulus control for the client character portrayal, and involved clinicians receive a consequence-free behavioral rehearsal opportunity to deliver the therapy. Further, removal of many sources of between-client variance in SP interactions is demonstrated to produce more valid, reliable estimates of clinician fidelity than in clinical work-samples (Imel et al., 2014). This current finding augments a developing literature for SP methodology (Beidas, Cross, & Dorsey, 2014; Lane & Rollnick, 2007), offering initial evidence of predictive validity of these post-training clinician performances in SP interactions for the clinical outcomes later achieved by their therapy-exposed OTP clients.

The full mediational model of general therapeutic effectiveness and subsequent clinical outcomes by CM skillfulness requires scientific replication, but may ultimately hold implications for behavior therapy training and practice. Therapist effects are long-recognized as influences of behavior therapy outcomes (Project MATCH, 1997, 1998), with common psychotherapy skills (like how well clinicians engage clients and offer timely interventions) identified as a source of between-clinician variance (Barber et al., 2001; Connors, Carroll, DiClemente, Longabaugh, & Donovan, 1997; De Weert-Van Oene, Schippers, De Jong, & Schrijvers, 2001; Meier, Barrowclough, & Donmall, 2005). In the current work, a therapy-specific factor fully mediated an association between a common psychotherapy factor and clinical outcome. That the involved clinical outcome was one specifically targeted for behavioral reinforcement is important to note; nevertheless, this full mediational finding lends credence to the development of therapy-specific clinician competencies as a valuable component of enhancing clinical practice. Debates about the relative weight to be given to common vs. specific psychotherapy factors are lengthy and ongoing (Carroll & Rounsaville, 2010; Luborsky et al., 2002; Luborsky, Singer, & Luborsky, 1975; Miller & Moyers, 2015; Sellman, 2010). The current finding supplements considerable existing literature that informs such debates, in this case reinforcing the importance of specific psychotherapy factors in the context of a local CM dissemination and implementation endeavor.

The finding of partial mediation of in-training therapy exposure and clinical outcome by CM skillfulness may offer insight for how best to prepare an addiction treatment workforce for therapy implementation. The CM training in the parent trial was guided by active learning strategies, with clinician development of therapy-specific skills via observation of live trainer demonstrations, participation in dyadic role-plays, and receipt of performance-based trainer feedback. As Hartzler and colleagues (2014) report, this resulted in robust and durable benefits across clinician-level implementation domains—the most prominent of which was that all CM-trained clinicians exceeded a competency benchmark for CM skillfulness (Petry & Ledgerwood, 2010). While partial mediation suggests potential additional influence of other factors, the development of therapy-specific skills did figure prominently in how in-training exposure to CM predicted subsequent outcome of CM-exposed clients. The finding is broadly consistent with that of Henggeler and colleagues (2013), who in a multisite CM training trial similarly recruited clinical supervisors and direct-care staff from addiction settings, offered skills-focused training curricula, and found durable benefits for therapy adoption and implementation. An implication of this cumulative research is that in planning trainings therapy purveyors: 1) recruit members of the addiction treatment workforce poised to serve as agents of therapy implementation, and 2) prioritize time and access to these persons by employing active learning strategies to maximize exposure to and behavioral rehearsal of key therapy-specific skills. How this may be done with greater reach within the broader addiction treatment community at-large awaits future research.

A supplemental finding of interest from the current study, given the emergence of CM skillfulness as an influential post-training implementation domain, was strong correlation of each of six component clinical practice behaviors with this behavioral fidelity index. These six component clinical practice behaviors are communication-focused, and prescribed for technical delivery of CM interventions that target therapy attendance (Petry & Ledgerwood, 2010). Yet, they are closely drawn from a larger set of clinical practice behaviors intended in staff delivery of prize-based CM interventions targeting substance abstinence, for which large-scale instrument validation was previously conducted (Petry et al., 2010). Relative to that reported in the prior CMCS instrument development work, current analyses revealed greater consistency among item-scale correlations (.49–.81) and suggest that each of the component clinical practice behaviors contributed to the computed CM skillfulness summary score. In the parent trial, the CM training curriculum was structured to expose direct-care clinicians (via brief conceptual rationale, live trainer demonstrations, dyadic role-play exercises, & performance-based trainer feedback) to each of these component clinical practice behavior in a serial manner that progressively built toward full intervention delivery. This approach is consistent with current recommendations for behavior therapy training (Beidas & Kendell, 2010; Herschell et al., 2010; Miller, Sorensen, Selzer, & Brigham, 2006), and with instructional design elements for therapy training programs for health professionals in a range of care settings (Cucciare, Weingardt, & Villafranca, 2008).

The current study has several limitations. Foremost among these are its single-site design and relatively small clinician sample size, both of which necessitated an exploratory analytic approach. Clearly, the conduct of a multisite trial that selected a larger and more diverse sample of direct-care clinicians would strengthen confidence in certainty and generalizability of study findings, and future research may offer such opportunity. While study participation was a voluntary opportunity rather than organizational mandate for these clinicians, an inherent self-selection bias must also be acknowledged. A further study caveat may be that some clinician implementation domains relied on a single-item measure (i.e., adoption readiness, common psychotherapy skills) or instrument not been previously-validated (i.e., self-efficacy to implement). Preliminary analytic results of these and other included measurement methods and instruments in the study may mitigate some such concerns. For example, the focal construct of CM skillfulness is supported by strong, previously-reported evidence of inter-rater reliability, internal consistency, and an absence of assessment reactivity (Hartzler et al., 2014). An additional caveat may concern the manner in which CM-eligible OTP clients were assigned to caseloads of direct-care clinicians. Occurring within the context of a real-world implementation effort, new clients were assigned to a given clinician based on space availability and other clinician-client matching considerations. While this models a common new client assignment process in community OTP settings and enrollment-as-usual at this particular OTP, the lack of randomization in this process leaves open the possibility of temporal or other 3^rd-variable influences. Finally, the caseload-aggregated attendance rates were based on all client data available during a relatively brief, 90-day provisional implementation period. A lengthier interval of provisional implementation would have encompassed more clientele, and presumably may have more conclusively ruled out possible historical influences on these therapy attendance rates by clinician caseload.

Caveats notwithstanding, the current work offers preliminary evidence that if replicated may inform future behavior therapy dissemination efforts for the addiction treatment community. In the context of this single-site implementation/effectiveness trial, these exploratory analyses identified CM skillfulness from among a range of post-training clinician implementation domains to be a specific predictor of subsequently targeted clinical outcomes among the clinicians’ CM-exposed clientele. Further, this behavioral fidelity index also fully or partially mediated associations between other salient clinician variables (i.e., general therapeutic effectiveness, in-training CM exposure) and targeted clinical outcomes. The collective findings suggest an apparent value in providing skills-focused training in CM, and perhaps other empirically-supported behavior therapies, to the addiction treatment workforce. Skills-focused training does not necessarily preclude trainer use of didactic and discussion elements in CM training curricula, presumably for purposes of enhancing clinician knowledge of core operant conditioning principles and practices as well as to dispel myths and misconceptions that deter adoption readiness. But current findings provide preliminary evidence to suggest such passive learning strategies are insufficient if the goal of behavior therapy training is to prepare a workforce to effectively implement a new approach.

Highlights.

• CM skillfulness is a behavioral measure of how well clinicians deliver CM interventions

• It was found to be a robust, specific post-training predictor of subsequent client outcomes

• It fully mediated an association between a common psychotherapy skill and client outcomes

• It partially mediated an association between in-training CM exposure and client outcomes

• Six component clinical practice behaviors each contribute meaningfully to CM skillfulness