Skip to main content
NCBI home page
Springer Nature - PMC COVID-19 Collection logoLink to Springer Nature - PMC COVID-19 Collection
. 2022 Oct 12:1–10. Online ahead of print. doi: 10.1007/s41347-022-00283-1

Virtual Training Is More Cost-Effective Than In-Person Training for Preparing Staff to Implement Contingency Management

Bryan Hartzler 1,✉, Jesse Hinde 2, Sharon Lang 3, Nicholas Correia 3, Julia Yermash 3, Kim Yap 3, Cara M Murphy 3, Richa Ruwala 2, Carla J Rash 4, Sara J Becker 3, Bryan R Garner 2
PMCID: PMC9553630  PMID: 36246531

Abstract

Behavior therapy implementation relies in part on training to foster counselor skills in preparation for delivery with fidelity. Amidst Covid-19, the professional education arena witnessed a rapid shift from in-person to virtual training, yet these modalities’ relative utility and expense is unknown. In the context of a cluster-randomized hybrid type 3 trial of contingency management (CM) implementation in opioid treatment programs (OTPs), a multi-cohort design presented rare opportunity to compare cost-effectiveness of virtual vs. in-person training. An initial counselor cohort (n = 26) from eight OTPs attended in-person training, and a subsequent cohort (n = 31) from ten OTPs attended virtual training. Common training elements were the facilitator, learning objectives, and educational strategies/activities. All clinicians submitted a post-training role-play, independently scored with a validated fidelity instrument for which performances were compared against benchmarks representing initial readiness and advanced proficiency. To examine the utility and expense of in-person and virtual trainings, cohort-specific rates for benchmark attainment were computed, and per-clinician expenses were estimated. Adjusted between-cohort differences were estimated via ordinary least squares, and an incremental cost effectiveness ratio (ICER) was calculated. Readiness and proficiency benchmarks were attained at rates 12–14% higher among clinicians attending virtual training, for which aggregated costs indicated a $399 per-clinician savings relative to in-person training. Accordingly, the ICER identified virtual training as the dominant strategy, reflecting greater cost-effectiveness across willingness-to-pay values. Study findings document greater utility, lesser expense, and cost-effectiveness of virtual training, which may inform post-pandemic dissemination of CM and other therapies.

Keywords: Contingency management, Therapy training, Virtual instruction, Implementation research

Introduction

Calls to increase adoption of evidence-based practices (EBPs) by the addiction workforce date back over two decades to a seminal Institute of Medicine (1998) report. At the core of many subsequent undertakings has been design and delivery of professional education activities whereby workforce members encounter, adopt, and prepare to implement EBPs. Despite such efforts to accelerate these workforce development processes, the economic toll of substance use disorders on US systems is staggering. With respect to the opioid epidemic alone, one annual estimate–encompassing fatalities and demands placed on systems of healthcare and criminal justice–exceeds $500 billion (CEA, 2017). Scaling up of EBPs may reduce this burden, though scrutiny is warranted for both the utility and expense of professional education activities intended to prepare workforce members to skillfully implement these practices.

An EBP available to the addiction workforce is contingency management (CM), which encompasses a family of behavioral reinforcement paradigms wherein client behavior is shaped toward a treatment adherence goal. Meta-analyses note a reliable range of mean effect sizes (d = 0.46–0.68) across CM paradigms (Benishek et al., 2014; Griffith et al., 2000; Lussier et al., 2006). One paradigm, referred to as prize-based CM (Petry et al., 2000), rewards treatment adherence through earning of draws from an urn (or “fishbowl”) for prizes of varying magnitude. With workforce adoption across CM paradigms estimated at 10–30% (Ducharme et al., 2010; Herbeck et al., 2008; McGovern et al., 2004), educational efforts merit prioritization. Such efforts for EBPs like CM have historically hinged on training, typically in a workshop format (Walters et al., 2005). Based on convergent evidence from large-scale RCTs of EBP training methods (Miller et al., 2004; Sholomskas et al., 2005), observational and experiential learning (i.e., trainer demonstration, behavioral rehearsal) are long-recognized as emphases of effective training workshops. Corresponding recommendations (Beidas & Kendall, 2010; Miller et al., 2006) still inform notions of workshop training as a preparatory learning context wherein workforce members observe and rehearse delivery of an EBP to build requisite clinical skills. Later emergence of phasic models of EBP implementation, like Aarons and colleagues’ (2011) Exploration-Preparation-Implementation-Sustainment (EPIS) framework, have increased awareness of other implementation strategies. These may include exploratory strategies that predate and inform workshop training (i.e., initial needs assessment or leadership consultation), or implementation and sustainment strategies subsequently applied to augment and sustain workshop gains. Within this phased framework, workshop training is a critical preparatory event whereby workforce members become capable of demonstrating fidelity, or one’s ability to skillfully deliver an EBP like CM as its developers intended (McHugh & Barlow, 2010), to signal individual readiness for its subsequent implementation.

Until recently, workshop training for EBPs like CM has typically occurred in-person. However, onset of COVID-19 diversified the nature of workforce development activities such that many now use virtual platforms (Cross-Technology Transfer Center Workgroup on Virtual Learning, 2021). While virtual trainings may eliminate some traditional costs (i.e., travel, in-person facility), their utility for preparing attendees to demonstrate EBP delivery skills–and thereby implementation readiness–remains unknown. Likewise, the expense of virtual trainings is not well-documented, nor does extant literature clearly address comparative cost-effectiveness of in-person and virtual workshop training for EBPs like CM. Consequently, the current work seeks to address these gaps in the EBP training literature, using data from an ongoing, multi-cohort trial (Becker et al., 2021) that examines implementation of prize-based CM in opioid treatment programs (OTPs). In this trial, OTP staff participate in initial workshop training and complete a role-play as a post-training assessment of fidelity. Mid-trial advent of COVID-19 necessitated a shift in procedures between its 1st and 2nd cohorts, recruited 15 months apart, with workshop training offered in-person with the initial cohort and virtually with the latter cohort. The resulting natural experiment provided a timely opportunity to compare the utility, expense, and cost-effectiveness of these two modalities for workshop training delivery.

Methods

Study Design

A comprehensive review of the parent trial, project MIMIC (maximizing implementation of motivational incentives in clinics), is available elsewhere (Becker et al., 2021). Briefly, it is a type 3 hybrid trial (Curran et al., 2012) with cluster-randomization of New England OTPs receiving one of two sets of implementation strategies: (1) an addiction technology transfer center (ATTC) approach, consisting of workshop training followed by performance feedback and monthly facilitation calls (Squires et al., 2008); or (2) an enhanced ATTC approach, with the ATTC components augmented by implementation and sustainment facilitation (Garner et al., 2017) and pay-for-performance incentives (Garner et al., 2018). Over its 5-year span, the trial will enroll 28–30 OTPs via staggering of three cohorts. OTP administrators each nominate up to five counselors, who must meet as inclusion criteria: (1) maintenance of an active caseload and (2) willingness to engage in the trial’s training and implementation support activities.

The trial adheres to the EPIS framework (Aarons et al., 2011), with an exploration phase wherein trial participation is discussed with OTP leadership and staff. Participating staff then complete 4 months of preparation phase activities, including workshop training, to establish readiness to deliver a CM protocol. In a 9-month implementation phase, they deliver the CM protocol whilst receiving active implementation support. In a 6-month sustainment phase, staff can maintain the CM protocol with aid of didactic resources on a centralized website, but absent active implementation support. The current work analyzes the utility and expense of the trial’s preparation phase activities, namely workshop training and a subsequent role-play serving as an outcome assessment. Other aforementioned activities, like performance feedback and monthly facilitation calls, are initiated in the trial’s implementation phase and not further discussed here. Notably, this trial was approved by the Brown University institutional review board (Protocol #1811002260) in December 2018, with subsequent amendments approved in May 2020 to enable virtual delivery of workshop training.

CM Training Approaches

The in-person and virtual workshop trainings shared as common elements: (1) curriculum development and delivery by a national subject matter expert (CR), who was assisted by project support staff and offered resources and recommendations principally by the lead author as well as by the study PIs; (2) a primary learning objective to train attending OTP staff to a skill-based fidelity criterion; (3) blended educational strategies with didactic instruction, small group discussions to develop local implementation plans, observation of live and recorded trainer demonstrations of prize-based CM protocols, and behavioral rehearsal by attending OTP staff; and (4) conceptual focus of education activities on core CM principles, local design, and planning for a prize-based CM protocol, application of clinical session fidelity ratings, dyadic role-plays of the CM protocol amongst groups of local OTP staff, and orientation to trial procedures. Table 1 lists the timing, duration, location, and training staff involved with workshop training for each cohort, in addition to these common elements of the in-person and virtual workshop trainings.

Table 1.

Dimensions of in-person vs. virtual workshop training

In-person workshop training Virtual workshop training
Timing June, 2019 August, 2020
Location University event center Zoom-facilitated webinars
Training staff

Trainer: Carla Rash

Support: Project Staff

Trainer: Carla Rash

Support: Project Staff
Duration 8.5 h, single-day event 5 h, segmented over multiple days
Learning objective Training to skills-based criterion* Training to skills-based criterion*
Learning structure Synchronous Mix of synchronous/asynchronous
Educational strategies

Didactic instruction

Small group discussion

Trainer demonstration/modeling

Behavioral rehearsal

Didactic instruction

Small group discussion

Trainer demonstration/modeling

Behavioral rehearsal
Educational activities

Introduction to CM principles

Implementation planning

Trainer modeling of CM delivery

Application of fidelity ratings

Dyadic role-play with peer

Orientation to parent trial

Introduction to CM principles

Implementation planning

Trainer modeling of CM delivery

Application of fidelity ratings

Dyadic role-play with peer

Orientation to parent trial
Remuneration 6.0 continuing education units 5.0 continuing education units
Open in a new tab

*Skill-based criterion was assessed via recorded role-play scored by study staff via the contingency management competence scale, with criterion for implementation readiness informed by recommendations of Petry and colleagues (2010)

In-Person CM Training

Occurring in June 2019 with an aggregate cohort of 26 OTP staff, in-person workshop training took place in a single day. Its duration was 8.5 h, with 75 min of planned breaks. In-person workshop training utilized a hotel or university-affiliated event center as a central location for an entirely synchronous learning experience. Due to variable scheduling and coverage needs of the recruited settings, OTP staff selected from three dates on which an identical set and sequence of training activities occurred. Attendees were reimbursed for travel costs, given lunch, and provided six continuing education credits.

Virtual CM Training

Occurring in August 2020 with a cohort of 31 OTP staff, virtual training was occurred over a multi-day period. This was delineated into three segments, mixing asynchronous and synchronous learning structure over five total hours of Zoom-enabled virtual instruction. Specifically, training participants individually reviewed an initial 1-h pre-recorded presentation on CM at a time and location of their choosing, and then as a group subsequently attended a pair of two-hour live sessions addressing: (a) local design of a prize-based CM protocol and (b) review of session fidelity ratings and trial procedures. For the pair of sessions, Zoom breakout rooms extended synchronous learning to include small group activities wherein OTP staff were asked to have a makeshift ‘fishbowl’ (e.g., vase, soup bowl, coffee mug) available to incorporate in role-plays. All virtual training participants received a Visa gift card to cover local per diem costs for meals, in addition to five continuing education units.

Post-training Assessment

To assess post-training readiness to deliver CM, each counselor submitted a 3–5 min audio-recorded role-play (for which they selected from among standardized case examples) within 30 days of workshop conclusion. Submitted recordings were scored via the contingency management competence scale [CMCS; (Petry et al., 2010)], which assesses six CM specific skills and three general therapy skills on 7-point Likert scales (1 = very poor, 7 = excellent). In this trial, a subset of 10% of these recordings was double-coded by two independent raters with inter-rater agreement exceeding 90%. Counselors received a written feedback report detailing their performance relative to two CMCS-based benchmarks: (1) beginning proficiency as a mean scale score of 4.0, corresponding with an “adequate” CMCS scale anchor; and (2) advanced proficiency as a mean scale score of 5.8, as the mean level of counselor proficiency attained in a CM trial wherein community-based clinicians received rigorous supervision (Petry et al., 2012).

Analytic Plan

Analyses encompassed comparative examination of workshop training effectiveness, cost-based evaluation, and cost-effectiveness of in-person vs. virtual workshop training.

Workshop Training Effectiveness

To compare effectiveness of in-person vs. virtual workshop training, CMCS-rated skillfulness from submitted role-plays served as the outcome. Initial compilation of the rates at which beginning and advanced proficiency benchmarks were met or exceeded in each OTP staff cohort was followed by bivariate t tests assessing statistical significance of between-cohort differences.

Cost-Based Evaluation

With primary intent to inform OTP decision-makers and potential sponsoring entities of preparation-related costs to competently deliver prize-based CM, activity-based costing estimated per-clinician costs for each cohort over the 4-month preparation phase. Costs that did not vary at a clinician-level (e.g., trainer time, facilities) were allocated across a given OTP’s staff, whereas those for research activities (e.g., obtaining informed consent for trial participation) that would not have been incurred were an OTP to independently implement CM were excluded. Development costs (e.g., creating training materials) were tracked separately as they would not otherwise be incurred in future replication of workshop training. Costs were estimated for two major activities–workshop training, and post-training role-plays. With respect to workshop training, this included labor costs corresponding to time spent by OTP staff, trainers, and study staff to attend and travel to the training; actual travel expenses; and actual costs paid for materials, training space, and meals. For post-training role-plays, this included labor costs for time spent by OTP staff in recording and submission as well as scoring by study staff; time for study staff to edit pre-recorded training videos; and actual material costs of audio-recorders. Costs for the initial cohort were inflated to 2020 dollars based on the 2019–2020 Consumer Price Indexes, as reported by the Bureau of Labor Statistics (2021).

In terms of labor costs, annual wage estimates for OTP staff were based on self-report of annual income elicited via a counselor survey completed at trial enrollment, whereas annual income for study staff was extracted from administrative records. The trainer was an external consultant paid $100/h. All wages were adjusted to hourly rates with data from the US Office of Personnel Management (2021), to which a 30% fringe and overhead rate was applied. Detailed logs tracked the amount of time spent by OTP and study staff on specific activities, whereas the trainer submitted invoices with detailed time breakdowns.

As for costing of travel time and expenses, the distance and time spent by the trainer, study staff, and OTP staff traveling to and from workshop training was calculated using the physical address of each individual’s respective place of employment as a starting point and the physical address of the training venue as endpoint. Mileage for all involved was costed at the 2019 federal mileage reimbursement rate ($0.56). Hotel costs for an overnight stay of three OTP staff in the initial cohort who worked a great distance from the in-person workshop training venue were obtained from administrative records.

Costing for materials (e.g., recorders, technology support), meals, and training space was obtained from administrative records. In the initial cohort, one of the three in-person trainings was held at a local venue, whereas the two others were held in a heavily subsidized university conference center. Thus, technology support and meals for all three in-person trainings were costed at standard rates for each local venue. In the latter cohort, Zoom access was subsidized by the university, but was costed using a standard 2020 license rate for an organization.

Cost-Effectiveness

Comparative cost-effectiveness of in-person vs. virtual workshop training was determined via two key steps: (1) adjusted between-cohort differences were estimated for costs relative to achievement of beginning/advanced proficiency benchmarks, using ordinary least squares, and (2) an incremental cost effectiveness ratio (ICER) was calculated to specify the ratio of between-cohort cost differences to the difference in training outcomes. The ICER indicates what resources must be invested to obtain an additional unit of outcome at a given willingness-to-pay. While confidence intervals and tests of statistical significance can be independently calculated from linear regression for cost-outcome differences, standard ICER methods are problematic as a ratio of two variables. Thus, to estimate statistical uncertainty for the current ICER, cost-effectiveness acceptability curves were calculated via 500 bootstrapped samples from original trial data to create an empirical distribution of expected costs and outcomes. Across replications, the strategy with highest probability of being optimal for a range of willingness-to-pay values was then identified.

Results

Sample Characteristics

Table 2 presents demographic and professional characteristics of the two OTP staff cohorts. No between-cohort differences were found in biological sex, race, ethnicity, or work experience, though a prominent difference was found in educational attainment–with a greater proportion of counselors in the latter cohort having earned a graduate degree (81% vs. 46%), χ2 (2) = 9.80, p < 0.001. Educational attainment was consequently included as a covariate in subsequent analyses. Overall, the demography and professional background of these collective OTP staff were representative of the broader OTP workforce in New England reported in a prior study (Becker et al., 2019).

Table 2.

Demographic and background characteristics of opioid treatment program staff

In-person workshop training cohort Virtual workshop training cohort
N 26 31
Age in years 37.48 36.65
  (Standard deviation) (10.95) (10.90)
Clinical experience in years 9.01 6.28
  (Standard deviation) (7.53) (6.65)
Workplace tenure in years 3.70 2.40
  (standard deviation) (4.86) (2.54)
Educational attainment*
  Less than a bachelor’s degree 8% 10%
  Bachelor’s degree 46% 10%
  Master’s degree 46% 80%
Demography
  Female 81% 87%
  Hispanic 8% 6%
  Non-Hispanic White 81% 87%
Open in a new tab

The initial cohort, drawn from 8 recruited opioid treatment programs, attended an in-person training workshop in June, 2019; the latter cohort, drawn from 10 recruited opioid treatment programs, attended a virtual training workshop in August, 2020

*indicates between-cohort difference at p < 0.05, based on χ2 test

Workshop Training Effectiveness

Among OTP staff completing in-person workshop training, 85% demonstrated CM skills that met or exceeded a beginning proficiency benchmark and 31% further achieved the advanced proficiency benchmark. Of those completing virtual workshop training, 97% evidenced CM skills that met or exceeded a beginning proficiency benchmark, and 45% further achieved the advanced proficiency benchmark. In purely descriptive terms, this associates virtual training with more widespread success in achieving skill-based benchmarks–by a margin of 12–14%. As for t tests assessing statistical significance of between-cohort differences in these attainment rates, there was a trend toward significance for the beginning proficiency benchmark, t = −1.54, p = 0.07; and a non-significant effect for the advanced proficiency benchmark, t = −1.11, p = 0.14.

Cost-Based Evaluation

As for aggregated costs of the two workshop training approaches, total expense for in-person and virtual delivery was $24,547 and $13,725, respectively. Relative to in-person workshop training, this equates to a 44% reduction in overall expense for virtual workshop training. Table 3 further details per-clinician expenses, and indicates an incremental cost difference of $399 between in-person and virtual workshop trainings.

Table 3.

Actual expenses for in-person and virtual workshop training

In-person workshop training Virtual workshop training
Workshop training $719 $322
  Labor costs
    Counselors $260 $151
    Training Staff $130 $65
  Travel costs
    Counselors $112 –
    Training Staff $44 –
    Space and Meal Costs $163 $44
    Materials Costs $10 $62
Submitted role plays $73 $71
  Labor costs
    Counselors $4 $2
    Training staff $28 $29
  Materials costs $41 $40
Total costs $792 $393
Open in a new tab

All costs are actual per-counselor costs, based on the recruited cohorts of opioid treatment programs’ staff for in-person workshop training (n = 26) and virtual workshop training (n = 31)

Of the $792 per-counselor expense of in-person workshop training, 49% was labor costs, 20% for travel, 22% for space/meals/materials, and 9% for recording and scoring of role-plays. For the $393 per-counselor expense of virtual workshop training, more than half (55%) was labor costs, and 11% was for meals. Materials costs represented a larger proportion (16%) of expense, and included arrangement of virtual conferencing capability, video-editing, and printing/shipping of training materials to attendees. Expense of role-play recording and scoring was stable across cohorts, though a greater proportional cost (18%) of virtual training workshop.

Cost-Effectiveness

Table 4 presents adjusted per-counselor cost of in-person and virtual workshop training along with residual determination based on ICER calculations examining the ratio of between-cohort differences in those costs and each of the two CMCS-based proficiency benchmarks. Controlling for educational attainment of OTP staff, the $423 estimated incremental difference between in-person vs. virtual workshop training was statistically significant (p < 0.05). Notably, ICER values are not presented due to clear and consistent identification in these analyses of the virtual workshop training as the dominant strategy (i.e., both more effective and less costly).

Table 4.

Cost-effectiveness metrics for achievement of proficiency benchmarks

Adjusted cost estimates Adjusted rate for staff achievement of the beginning proficiency benchmark Beginning proficiency ICER Adjusted rate for staff achievement of the advanced proficiency benchmark Advanced proficiency ICER
In-person workshop training $811 0.86 0.36
Virtual workshop training $388 0.96 0.41
Incremental difference $423* −0.10 Dominant −0.05 Dominant
Open in a new tab

The adjusted cost and proficiency estimates in this table reflect estimates controlling for the effects of educational attainment, and therefore may not match the actual costs or attainment rates reported elsewhere; the incremental cost effectiveness ratios (ICERs) were calculated to specify the ratio of between-cohort cost differences to the difference in training outcomes, to indicate what resources must be invested to obtain an additional unit of outcome at a given willingness-to-pay; proficiency benchmarks used to represent training outcomes were derived from the contingency management competence scale; scale average scores of 4.0 as threshold for achievement of beginning proficiency and 5.8 as threshold for achievement of advanced proficiency; the dominant training strategy is that which is both more effective and less costly

*p < 0.05

Figure 1 presents cost effectiveness acceptability curves for the probability that virtual workshop training is cost-effective for OTP staff to achieve beginning and advanced proficiency benchmarks across a range of willingness-to-pay values. The solid line indicates a near certainty that virtual workshop training is cost-effective for OTP staff to achieve beginning proficiency up to a willingness-to-pay threshold of $5500 per counselor, which converges to 98% likelihood of greater cost-effectiveness as the willingness-to-pay threshold increases (i.e., further down the X-axis). The hatched line suggests a near certainty that virtual workshop training is cost-effective for OTP staff to achieve advanced proficiency up to a willingness-to-pay threshold of $1000 per counselor. Beyond that threshold, the cost-effectiveness acceptability curve decreases to a probability of 0.9 at $3750, 0.8 at $8000, and 0.78 above $8000–which means that, even at willingness-to-pay thresholds above $8000, there remains a 78% chance that virtual workshop training is the more cost-effective of the two modalities.

Fig. 1.

Fig. 1

Open in a new tab

Cost effectiveness acceptability curve

Discussion

Efforts to disseminate EBPs like CM continue, in hopes of improving the quality of care received by persons with substance use disorders, and for which evaluation of cost-effectiveness is sorely needed. Through opportunistic use of data from a multi-cohort trial seeking to implement CM in New England–based OTPs, the current work examined in-person vs. virtual workshop trainings via quasi-experimental comparison. Advantages of virtual workshop training included (1) more widespread success, albeit by a statistically non-significant margin, in preparing OTP staff to develop CM skills signifying beginning (+ 12%) and advanced proficiency (+14%), (2) a 44% reduction in overall expense, largely due to absence of travel and facilities costs, equating to a $399 per-counselor incremental savings, and (3) greater cost-effectiveness across a range of willingness-to-pay values that decision-makers are apt to consider. Findings offer converging evidence for virtual delivery of educational activities for health professionals, as observed in its feasible application in medical education (Almarzooq et al., 2020; O’Doherty et al., 2018) as well as its comparable effectiveness to more resource-intensive in-person methods of psychotherapy training (Moore et al., 2021; Soll et al., 2021). Thus, current findings are among those that may inform future decisions by EBP purveyors and sponsoring entities alike regarding fiscal and educational benefits of opting for virtual delivery of workshop training and other implementation support activities for behavior therapies like CM.

With regard to effectiveness, in-person and virtual workshop trainings each provided adequate preparation for most OTP staff to begin delivering prize-based CM–given the broader context of a hybrid type 3 trial wherein these staff were to subsequently receive longitudinal implementation support. This extended opportunity to subsequently improve CM skillfulness is salient, given that a minority of OTP staff in both cohorts (31–45%) achieved a more aspirational fidelity benchmark that signifies advanced proficiency in this behavior therapy. Attainment of more advanced CM skillfulness is strongly recommended for implementation success, given its identification in prior training research as a predictor of both sustained skill in delivery of CM protocols and enhanced therapeutic response among clients with whom those protocols are implemented (Hartzler et al., 2014, 2017; Petry et al., 2012). In the current work, as in broader extant therapy training literature, the focus was on comparing learning modalities and methods rather than identifying characteristics of individual learners that predict successful training outcomes. Notably, prior CM training research reported no moderating influences of traditional indices of demography and professional background (Henggeler et al., 2013). Nevertheless, the burgeoning emergence of virtual therapy training–and its apparent cost-effectiveness–suggests there may be utility in examining other characteristics of individual learners. Measurement of one’s experience and/or comfort with virtual technology may have been unavailable in the current work, but inclusion of such constructs in future research on virtual therapy training will enable examination of their potential influence.

Current findings regarding comparative costs and cost-effectiveness of these trainings reflect an important, often overlooked focus on fiscal considerations surrounding EBP training and implementation support activities (Reeves et al., 2019; Roberts et al., 2019). As outlined by Hinde and colleagues (2020), several established methods are now available for such economic evaluation. In the current work, the intent was to utilize available counselor-level outcome data to inform organization-level perspectives of decision-makers about the respective value of two approaches to EBP training–itself a necessary, but rarely sufficient, preparatory activity for which community-based expectations for skills-based outcomes can vary tremendously. By examining differences in costs of the two training approaches in relation to differences in their respective skill-based outcomes (via incremental cost-effectiveness ratios), the resulting guidance will aid decision-makers across a wide range of circumstances wherein workforce development efforts may be undertaken. Future economic evaluation is merited concerning virtual delivery of other implementation support activities for CM as well as for other useful treatment and recovery practices for persons with substance use disorders. As regionally based intermediary purveyor organizations linked within a national network, the SAMHSA-funded ATTCs have a primary role to play in such work. Beyond the breadth of useful treatment and recovery practices for which they accelerate adoption, Covid-19 has spurred a host of virtual innovations (e.g., listening sessions, virtual process walk-throughs, ‘live’ supervision) within this network to facilitate implementation support for EBPs (Cross-Technology Transfer Center Workgroup on Virtual Learning, 2021).

Study caveats should be acknowledged. Foremost among these are a quasi-experimental design that did not allow for control of potential 3rd-variable influences–leaving open possibility that observed advantages of the latter training cohort may be attributable to factors other than the virtual delivery of their workshop training. As one example, advent of COVID-19 necessitated efforts by study staff to respond to pervasive change in OTPs’ medication dosing and clinical services (Center for Medicare & Medicaid Services, 2020; SAMHSA, 2020). Notably, this responsivity broadly targeted necessary assistance with integrating the focal CM paradigm into amended OTP service routines (i.e., use of virtual “fishbowls” in telehealth visits), but did not alter the training workshop’s core learning objectives or educational activities. Relatedly, onset of COVID-19 prolonged the time between the two cohorts, during which the lead trainer and study staff engaged in quality improvement discussions that may have augmented subsequent training gains through greater opportunity for reflective practice. Corresponding adaptations to the training curriculum included (1) its distributed delivery in briefer sections over multiple weeks (contrasted with en masse delivery in a single workshop day), which coincides with a previously identified training preference among OTP personnel (Hartzler & Rabun, 2014), and (2) curriculum exposure via synchronous and asynchronous learning. A further caveat was the prominent between-cohort difference in educational attainment–given its identification as a predictor of receptive CM adoption attitudes in prior research (Fuller et al., 2007; Hartzler et al., 2012a, b)–though notably, it was a covariate in all inferential comparisons of the two cohorts. Other sampling caveats concerning the size and representativeness of this collective workforce sample may be mitigated by its derivation from 18 OTPs, located across the New England region. Use of the described peer role-plays to derive fidelity-based outcome measures, in lieu of standardized patient procedures traditionally employed in clinical education, is another acknowledged methodological caveat. A final caveat is uncertainty about the reliability with which CM workshop training effects reported herein may be achieved by other community-based trainers, given absence of credentialing processes that exist as trainer preparation for other EBPs (e.g., motivational interviewing network of trainers). The current workshop training effort may have been enhanced by (1) subject matter expertise of the involved trainer (CR), (2) availability of consultative feedback from the lead author (BH) and principal investigators of the parent trial (SB, BG), and (3) assistance from the study’s training support team (SL, NC, JY, KY, CM).

Caveats notwithstanding, the current evaluation offers initial evidence to support virtual training as a useful and cost-effective means of preparing OTP personnel to capably deliver a prize-based CM protocol. These preliminary findings await replication through future study. To that end, similar economic evaluation efforts may be directed toward CM training workshops targeted to clinical personnel of alternative settings, to other implementation support activities for CM, and broadly to training and implementation support activities for other useful treatment and recovery practices. Future efforts may also consider the cost-effectiveness of hybrid approaches to workforce development that mix or combine virtual and in-person modes of delivery of distinct segments of an EBP training curriculum. Such hybridization may retain unique benefits of each, perhaps offering synergy in parallel to the manner in which flipping the classroom (Bergmann & Sams, 2012) is thought to enhance a combination of didactic and applied learning. With virtual technology having now enveloped professional work in many sectors, the addiction field stands to benefit from greater understanding of the fiscal and conceptual implications of its role in workforce development.

Acknowledgements

The authors wish to thank the staff of the participating OTPs for allowing their efforts to be examined.

Author Contribution

All authors contributed to the study conception and design. Jesse Hinde performed all data analyses. All authors contributed to an iterative process whereby the manuscript was drafted, reviewed, and revised, culminating in the eventual version accepted for publication by this journal.

Funding

This research was supported by NIDA R01DA046941, Implementing Contingency Management in Opioid Treatment Centers Across New England: A Hybrid Type 3 Trial (MPI, Becker/Garner). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIDA.

Declarations

Disclaimer

The funder had no direct role in the study design and analysis plan, nor in interpretation of data or its presentation in manuscripts.

Ethics Approval

The Brown University Institutional Review Board approved all procedures for the noted trial (#1811002260) as the board of record.

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Consent to Publication

Not applicable.

Competing Interests

The authors declare no competing interests.

Footnotes

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. Aarons GA, Hurlburt M, McCue Horwitz S. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services. 2011;38(1):4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Almarzooq ZI, Lopes M, Kochar A. Virtual learning during the COVID-19 panedemic: A disruptive technology in graduate medical education. Journal of American College of Cardiology. 2020;75(20):2635–2638. doi: 10.1016/j.jacc.2020.04.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Becker SJ, Kelly LM, Kang AW, Escobar KL, Squires DD. Factors associated with contingency management adoption among opioid treatment providers receiving a comprehensive implementation strategy. Substance Abuse. 2019;40(1):56–60. doi: 10.1080/08897077.2018.1455164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Becker SJ, Murphy CM, Hartzler B, Rash CJ, Janssen T, Roosa M, Madden LM, Garner BR. Project MIMIC (Maximizing Implementation of Motivational Incentives in Clinics): A cluster-randomized type 3 hybrid effectiveness-implementation trial. Addiction Science and Clinical Practice. 2021;16(1):61. doi: 10.1186/s13722-021-00268-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beidas RS, Kendall PC. Training therapists in evidence-based practice: A critical review from a systems-contextual perspective. Clinical Psychology: Science and Practice. 2010;17(1):1–31. doi: 10.1111/j.1468-2850.2009.01187.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Benishek LA, Dugosh KL, Kirby KC, Matejkowski J, Clements NT, Seymour BL, Festinger DS. Prize-based contingency management for the treatment of substance abusers: A meta-analysis. Addiction. 2014;109(9):1426–1436. doi: 10.1111/add.12589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bergmann J, Sams A. Flip your classroom: Reach every student in every class every day. International Society for Technology; 2012. [Google Scholar]
  8. Bureau of Labor Statistics. (2021). CPI for all urban consumers, 2011–2021. Retrieved August 10, 2021, from https://data.bls.gov/timeseries/CUUR0000SA0?amp%253bdata_tool=XGtable&output_view=data&include_graphs=true
  9. CEA. (2017). The understimated cost of the opioid crisis. Retrieved August 10, 2021, from https://www.whitehouse.gov/cea
  10. Center for Medicare & Medicaid Services. (2020). Medicare telemedicine helath care provider fact sheet. Retrieved August 10, 2021, from https://www.cms.gov/newsroom/fact-sheets/medicare-telemedicine-health-care-provider-fact-sheet
  11. Cross-Technology Transfer Center Workgroup on Virtual Learning Virtual reality for behavioral health workforce development in the era of COVID-19. Journal of Substance Abuse Treatment. 2021;121:108157. doi: 10.1016/j.jsat.2020.108157. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care. 2012;50(3):217–226. doi: 10.1097/MLR.0b013e3182408812. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ducharme LJ, Knudsen HK, Abraham AJ, Roman PM. Counselor attitudes toward the use of motivational incentives in addiction treatment. American Journal of Addiction. 2010;19(6):496–503. doi: 10.1111/j.1521-0391.2010.00081.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Fuller BE, Rieckmann T, Nunes EV, Miller M, Arfken C, Edmundson E, McCarty D. Organizational readiness for change and opinions toward treatment innovations. Journal of Substance Abuse Treatment. 2007;33(2):183–192. doi: 10.1016/j.jsat.2006.12.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Garner BR, Lwin AK, Strickler GK, Hunter BD, Shepard DS. Pay-for-performance as a cost-effective strategy: Results from a cluster-randomized trial. Implementation Science. 2018;13(1):1–11. doi: 10.1186/s13012-018-0774-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Garner BR, Zehner M, Roosa M, Martino S, Gotham HJ, Ball EL, Stilen P, Speck K, Vandersloot D, Rieckmann TR, Chaple M, Martin EG, Kaiser D, Ford JH. Testing the implementation and sustainment facilitation (ISF) strategy as an effective adjunct to the Addiction Technology Transfer Center (ATTC) strategy: Study protocol for a cluster-randomized trial. Addiction Science & Clinical Practice. 2017;12(1):32. doi: 10.1186/s13722-017-0096-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Griffith JD, Rowan-Szal GA, Roark RR, Simpson DD. Contingency management in outpatient methadone treatment: A meta-analysis. Drug and Alcohol Dependence. 2000;58(1–2):55–66. doi: 10.1016/s0376-8716(99)00068-x. [DOI] [PubMed] [Google Scholar]
  18. Hartzler B, Beadnell B, Donovan DM. Predictive validity of addiction treatment clinicians’ post-training contingency management skills for subsequent clinical outcomes. Journal of Substance Abuse Treatment. 2017;72(1):126–133. doi: 10.1016/j.jsat.2015.11.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hartzler B, Donovan DM, Tillotson C, Mongoue-Tchokote S, Doyle S, McCarty D. A multi-level approach to predicting community addiction treatment attitudes about contingency management. Journal of Substance Abuse Treatment. 2012;42(2):213–221. doi: 10.1016/j.jsat.2011.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hartzler B, Jackson TR, Jones BE, Beadnell B, Calsyn DA. Disseminating contingency management: Impacts of staff training and implementation at an opiate treatment program. Journal of Substance Abuse Treatment. 2014;46(4):429–438. doi: 10.1016/j.jsat.2013.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hartzler B, Lash SJ, Roll JM. Contingency management in substance abuse treatment: A structured review of the evidence for its transportability. Drug and Alcohol Dependence. 2012;122(1–2):1–10. doi: 10.1016/j.drugalcdep.2011.11.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hartzler B, Rabun C. Training addiction professionals in empirically-supported treatments: Perspectives from the treatment community. Substance Abuse. 2014;35(1):30–36. doi: 10.1080/08897077.2013.789816. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Henggeler SW, Chapman JE, Rowland MD, Sheidow AJ, Cunningham PB. Evaluating training methods for transporting contingency management to therapists. Journal of Substance Abuse Treatment. 2013;45(5):466–474. doi: 10.1016/j.jsat.2013.06.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Herbeck DM, Hser Y, Teruya C. Empirically supported substance abuse treatment approaches: A survey of treatment providers’ perspectives and practices. Addictive Behaviors. 2008;33:699–712. doi: 10.1016/j.addbeh.2007.12.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Hinde JM, Bray JW, Cowell AJ. Implementation science on the margins: How do we demonstrate the value of implementation strategies? Family, Systems, & Health. 2020;38(3):225–231. doi: 10.1037/fsh0000535. [DOI] [PubMed] [Google Scholar]
  26. Institute of Medicine . Bridging the gap between practice and research: Forging partnerships with community-based drug and alcohol treatment. US: National Academies Press; 1998. [PubMed] [Google Scholar]
  27. Lussier JP, Heil SH, Mongeon JA, Badger GJ, Higgins ST. A meta-analysis of voucher-based reinforcement therapy for substance use disorders. Addiction. 2006;101(2):192–203. doi: 10.1111/j.1360-0443.2006.01311.x. [DOI] [PubMed] [Google Scholar]
  28. Management, U. S. O. o. P. (2021). Policy, data, and oversight: Pay and leave. Retrieved August 10, 2021, from https://www.opm.gov/policy-data-oversight/pay-leave/pay-administration/fact-sheets/computing-hourly-rates-of-pay-using-the-2087-hour-divisor/
  29. McGovern MP, Fox TS, Xie H, Drake RE. A survey of clinical practices and readiness to adopt evidence-based practicies: Dissemination research in an addiction treatment system. Journal of Substance Abuse Treatment. 2004;26:305–312. doi: 10.1016/j.jsat.2004.03.003. [DOI] [PubMed] [Google Scholar]
  30. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. American Psychologist, 65(2), 73–84. 10.1037/a0018121. PMID: 20141263. [DOI] [PubMed]
  31. Miller WR, Sorensen JL, Selzer JA, Brigham GS. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment. 2006;31(1):25–39. doi: 10.1016/j.jsat.2006.03.005. [DOI] [PubMed] [Google Scholar]
  32. Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology. 2004;72(6):1050–1062. doi: 10.1037/0022-006X.72.6.1050. [DOI] [PubMed] [Google Scholar]
  33. Moore, A. L., Miller, T. M., & Ledbetter, C. (2021). Remote vs. in-person delivery of LearningRx one-on-one cognitive training during the COVID-19 pandemic: A non-inferiority study. Frontiers in Psychology, 12, 749898. 10.3389/fpsyg.2021.749898 [DOI] [PMC free article] [PubMed]
  34. O'Doherty D, Dromey M, Lougheed J, Hannigan A, Last J, MacGrath D. Bariers and solutions to online learning in medical education: An integrative review. BMC Medical Education. 2018;18:130. doi: 10.1186/s12909-018-1240-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Petry NM, Alessi SM, Ledgerwood DM. A randomized trial of contingency management delivered by community therapists. Journal of Consulting and Clinical Psychology. 2012;80(2):286–298. doi: 10.1037/a0026826. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Petry NM, Alessi SM, Ledgerwood DM, Sierra S. Psychometric properties of the contingency management competence scale. Drug and Alcohol Dependence. 2010;109(1–3):167–174. doi: 10.1016/j.drugalcdep.2009.12.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Petry NM, Martin B, Cooney JL, Kranzler HR. Give them prizes, and they will come: Contingency management for treatment of alcohol dependence. Journal of Consulting and Clinical Psychology. 2000;68(2):250–257. doi: 10.1037//0022-006x.68.2.250. [DOI] [PubMed] [Google Scholar]
  38. Reeves P, Edmunds K, Searles A, Wiggers J. Economic evaluations of public health implementation-interventions: A systematic review and guideline for practice. Public Health. 2019;169:101–113. doi: 10.1016/j.puhe.2019.01.012. [DOI] [PubMed] [Google Scholar]
  39. Roberts SLE, Healey A, Saevdalis N. Use of health economic evaluation in the implementation and improvement science fields–a systematic literature review. Implementation Science. 2019;14:72. doi: 10.1186/s13012-019-0901-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. SAMHSA. (2020). Opioid treatment program guidance. Retrieved August 10, 2021, from https://www.samhsa.gov/sites/default/files/otp-guidance-20200316.pdf
  41. Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 2005;73(1):106–115. doi: 10.1037/0022-006X.73.1.106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Soll, D., Fuchs, R., & Mehl, S. (2021). Teaching cognitive behavior therapy to postgraduate health care professionals in times of COVID-19 - An asynchronous blended learning environment proved to be non-inferior to in-person training. Frontiers in Psychology,12, 657234. 10.3389/fpsyg.2021.657234 [DOI] [PMC free article] [PubMed]
  43. Squires DD, Gumbley SJ, Storti SA. Training substance abuse treatment organizations to adopt evidence-based practices: The Addiction Transfer Center of New England Science-to-Service Laboratory. Journal of Substance Abuse Treatment. 2008;34:293–301. doi: 10.1016/j.jsat.2007.04.010. [DOI] [PubMed] [Google Scholar]
  44. Walters ST, Matson SA, Baer JS, Ziedonis DM. Effectiveness of workshop training of psychosocial treatments in addiction: A systematic review. Journal of Substance Abuse Treatment. 2005;29:283–293. doi: 10.1016/j.jsat.2005.08.006. [DOI] [PubMed] [Google Scholar]

Articles from Journal of Technology in Behavioral Science are provided here courtesy of Nature Publishing Group

RESOURCES