Abstract
Objective
To evaluate the cost and cost-effectiveness of three strategies for teaching community program clinicians motivational interviewing (MI): self-study (SS), expert-led (EX), and train-the-trainer (TT).
Methods
This economic analysis was conducted as part of a three-arm clinician training trial comprising 12 community treatment programs randomly assigned to the three conditions (n = 92 clinician participants). EX and TT conditions used skill-building workshops and three monthly supervision sessions. SS provided clinicians MI training materials only. The primary outcome measure was the number of clinicians meeting MI performance standards at 12-week follow-up. Unit costs were obtained via surveys administered at the 12 participating programs. Resource utilizations and clinician outcomes were obtained from the training trial. Costs and outcomes were normalized to account for differing numbers of clinicians across programs and conditions. Incremental cost-effectiveness ratios and cost-effectiveness acceptability curves were used to evaluate the relative cost-effectiveness of the three training strategies.
Results
SS is likely to be the most cost-effective training strategy if the threshold value to decision makers of an additional clinician meeting MI performance standards at 12-week follow-up is less than approximately $2,870, and EX is likely to be the most cost-effective strategy when the threshold value is greater than approximately $2,870.
Conclusions
This study provides accurate estimates of the economic costs and relative cost-effectiveness of three different strategies for training community program clinicians in motivational interviewing and should be of interest to decision makers seeking to implement empirically supported addiction treatments with scarce resources.
Keywords: training costs, cost-effectiveness, implementation science, motivational interviewing, training strategies, dissemination
1. Introduction
The systematic evaluation of different strategies for training clinicians to implement empirically-supported psychosocial addiction treatments has increased in the literature (Beidas and Kendall, 2010; Madson et al., 2009; Martino, in press), but to date evaluation of the costs and cost-effectiveness of clinician training strategies are very rare, even though many studies have noted that the cost of training clinicians must be weighed against the degree of benefits expected from them (Cucciare et al., 2008; Miller et al., 2005). Without knowing the cost-effectiveness of different approaches for training clinicians in empirically supported addiction treatments, program directors and policy-makers have little guidance in deciding if expenditures on training initiatives are justifiable.
One of the few strategies that has been demonstrated to be effective in training clinicians in empirically supported addiction treatments is the use of skill-building workshops followed by supervision that includes treatment integrity rating-based feedback and coaching, all delivered by treatment experts (Miller et al., 2004; Sholomskas et al., 2005). However, one of the main barriers associated with the use of expert-delivered workshop training and supervision is the high cost of obtaining experts to train clinicians. As a result, the train-the-trainer (TT) model has become a popular method for extending the reach of expert trainers by having experts train practitioners in the designated treatment and in how to teach it to others (Hawkins and Sinha, 1998; Hein et al., 2009; Miller and Rollnick, 2002; Martino et al., in press). In the TT model, trained practitioners are then expected to return to their programs and train, supervise, and monitor staff members’ treatment implementation. The initial costs of preparing the practitioners as trainers presumably are offset by lower clinician training and supervision costs for those trained by the practitioner/trainers.
Recently, Martino and colleagues (in press) examined the relative effectiveness of self-study (SS), expert-led (EX), and train-the-trainer (TT) strategies for teaching community program clinicians motivational interviewing (MI) and found that, in comparison to SS, at a 12-week follow-up point both EX and TT resulted in a significantly higher percentage of clinicians reaching standards of MI performance that have been commonly used to certify clinicians in MI effectiveness trials (Carroll et al., 2006, 2009; Ball et al., 2007). However, the costs and relative cost-effectiveness of all of these efforts is unknown. Thus, in the present report we present an evaluation of the costs and cost-effectiveness of SS, EX, and TT approaches for teaching community program clinicians MI. MI, developed by Miller and colleagues (Miller et al., 1998; Miller and Rollnick, 2002) is a well-established effective brief treatment for improving client program retention and alcohol and drug use treatment outcomes in community treatment programs (Carroll et al., 2006, 2009; Ball et al., 2007; Lundahl et al., 2010). International and U.S. policymakers have strongly encouraged clinicians to learn MI as one of several evidence-based addiction treatments to improve client care (Giuseppe and Clerici, 2006; Hintz and Mann, 2006; Miller et al., 2006; Reickmann et al., 2009).
To our knowledge, this is the first economic analysis of multiple strategies for training clinicians in empirically validated therapies, in this case, MI. Training costs are presented in a variety of ways to be useful to decision makers. We also provide incremental cost-effectiveness ratios (ICERs) and cost-effectiveness acceptability curves (CEACs) that define the range of values over which each training strategy would likely be the most cost effective for increasing the number of clinicians achieving adequate levels of MI performance.
2. Methods
Methods and results of the training strategies trial are described in the main report (Martino et al., in press). The study design and main outcomes are summarized briefly below, followed by the methods used for the cost analysis and the cost-effectiveness analysis. Clinician outcome and resource utilization data for these analyses are taken from the main trial and combined with cost data obtained prospectively from the participating programs.
2.1 Description of training strategies trial
Twelve outpatient substance abuse treatment programs in Connecticut, USA were randomly assigned to one of three MI training conditions (4 programs per condition): SS, EX, and TT. Ninety-two clinicians within these programs participated in the program-assigned conditions (SS = 31, EX = 32, TT = 29). Eligibility criteria for clinicians included working a minimum of 20 hours per week, having a caseload that included English-speaking substance-using clients, no formal MI workshop training 12 weeks prior to study enrollment (the period of time in which workshop training effects are likely to dissipate without follow-up structured supervision; Walters et al. 2005), and no prior supervision using rating-based feedback and coaching. Clinicians completed baseline, post-workshop, post-supervision, and 12-week follow-up assessments and received financial compensation for these activities. At each assessment point, clinicians submitted audiotaped 40-minute sessions in which they were instructed to conduct MI with clients who had substance use problems. Clinicians, not research staff, selected clients based on clinical judgment of the suitability of substituting MI for their standard practice and the client’s willingness to be audiotaped (signed consent obtained).
The primary trial outcomes were the clinicians’ 1) MI adherence and competence in actual and role-played client sessions, and 2) the percentage of clinicians performing MI adequately (described in detail below). Baseline clinician characteristics (demographics, education, discipline, licensure status, years of counseling experience, MI training experiences), primary outcomes, and retention rates (98% post-workshop; 90% post-supervision; and 82% 12-week follow-up) did not differ between the training conditions.
2.1.1. Training conditions
2.1.1.1 Self-Study (SS)
Clinicians assigned to this condition were provided with a copy of the MI textbook (Miller and Rollnick, 2002), training videotapes (Miller et al., 1998), and a MI treatment manual (Martino et al., 2004). Clinicians reviewed these materials in a 1-hour meeting with the expert and were asked to spend an additional 20 hours reviewing these materials over the next 12 weeks.
2.1.1.2. Expert-Led (EX)
Clinicians were provided with the same training materials as the SS participants. In addition, clinicians completed a 15-hour workshop in MI and then practiced the approach in audiotaped client sessions. These sessions were rated for MI adherence and competence by the expert trainer (S.M.) and served as the basis for individual face-to-face supervision in which clinicians received personal rating-based feedback about their use of MI strategies and coaching to further improve their MI performance. Clinicians received this supervision once per month for three months.
2.1.1.3. Train-the-Trainer (TT)
Two clinicians for each program in the TT condition were trained as trainers. Trainers learned MI from the expert trainer within 15-hour workshops and supervision as described above. After demonstrating adequate MI performance with actual client sessions, they completed another 15-hour workshop training to learn how to conduct the MI workshop and rating-based supervision. Trainers received monthly consultation calls from the expert for 3 months and delivered the workshop and supervision to their clinicians in the same manner as the expert. Trainers and clinicians received the same training materials as the SS participants.
2.1.2. Measurement of adequate performance
The Independent Tape Rater Scale (ITRS; Ball et al., 2002), which has been demonstrated to be a reliable and valid measure of the integrity of MI delivery in multiple studies (Gibbons et al., in press; Martino et al., 2008; Santa Ana et al., 2009), was used to assess the clinicians adherence (frequency of use) and competence (skill or quality) in implementing MI. In the trial, 10 MI consistent items were rated along 7-point Likert scales. To determine if clinicians’ met adequate MI performance standards, the same threshold was used as in prior clinical trials of MI with community program clinicians, that is, the clinicians conducted sessions in which they were rated as having at least half of the ITRS MI consistent scale items rated average or above in terms of adherence and competence (Carroll et al., 2006; Ball et al., 2007; Carroll et al., 2009).
2.2 Sample used in the economic analyses
A substantial number of clinicians in each training arm met MI performance standards at baseline (29 total; 7 in SS, 13 in EX, 9 in TT). Since this occurrence is both (a) unlikely to happen in practice (i.e., programs would want to reserve scarce training dollars for clinicians not meeting MI performance standards), and (b) likely to bias the cost effectiveness results for the three training strategies, clinicians meeting MI performance standards at baseline were excluded from the present study. Five additional clinicians were excluded due to either missing or malfunctioning baseline assessment audiotapes (2 in SS, 2 in EX, 1 in TT). Thus, the training costs and measure of effectiveness in the present study are based on the subsample of clinicians who did not meet independently-rated adequate MI performance standards at baseline (n = 58; 22 in SS, 17 in EX, 19 in TT). As described in detail further below, to allow direct comparisons across programs and arms, the training costs and effectiveness measure for this subsample were normalized assuming eight clinicians per cohort (eight was both the median and modal number of clinicians per cohort in the trial).
2.3 Cost analysis
Cost data were collected prospectively alongside the training strategies trial. A research assistant administered a survey to the Chief Financial Officer, Chief Executive Officer, or accountants at each program to obtain necessary cost information (e.g., program overhead and fringe rates; annual expenditures on rent/mortgage, utilities and maintenance). In addition, participants provided their annual salary/hourly wage at the baseline assessment. Data on resources used (e.g., number of supervision sessions attended, time spent by expert and program-based trainers in activities, time spent by clinicians reviewing the training materials) came from the training trial.
Costs were calculated from the perspective of the program and adjusted to 2006 US dollars using the Consumer Price Index. Such costs included those related to workshop training sessions, rating audiotaped practice sessions, supervision sessions, training of program-based trainers, and time spent by participants reviewing training materials. All labor costs included fringe benefits and overhead. Research-specific costs (e.g., incentive payments for study participation) were excluded from the analysis.
Although most of the items presented in the detailed cost analysis are straightforward, several require explanation. First, to estimate the training-related space costs at each program, we obtained the annual economic space cost for each program’s entire facility (including rent/mortgage, utilities, and maintenance), and then prorated this annual cost by (1) the fraction of the entire facility’s space that was used for training-related activities, and (2) the fraction of the total number of program operating hours for which the space was used for training-related activities (Olmstead et al., 2010). Second, to estimate travel costs (time and mileage), the round-trip distance between the expert trainer’s workplace and each study program was assumed to be equal to the average round-trip distance across all study programs (81.3 miles). Third, to estimate the cost (from the program’s perspective) of the self-reported time clinicians spent reviewing the training materials, we assumed (based on clinician anecdotes) that 25% of this time was done “on company time” with the balance occurring on personal time (we also explored the sensitivity of the results to this assumption).
2.3.1 Economic costs
Detailed economic costs were estimated at each program within each arm as follows. In the SS condition, program costs included a) time spent by the expert conducting the 1-hour orientation meeting and traveling to the program (including mileage reimbursement), b) time spent by clinicians in the meeting and reviewing training materials, c) space associated with the meeting, and d) equipment (TV/VCR) and materials (books, manuals, training videos).
In the EX condition, program costs included a) time spent by the expert trainer conducting the 15-hour workshop, rating audiotaped practice sessions and completing feedback forms, traveling to the program (including mileage reimbursement), and providing face-to-face supervision, b) time spent by clinicians in the workshop, receiving supervision, and reviewing training materials, c) space associated with the workshop and supervision, d) express mail costs from the supervisory process (i.e., audiotapes mailed to the expert for review), and e) equipment (TV/VCR, tape recorders, microphones and audiocassette tapes) and materials (books, manuals, training videos).
In the TT condition, program costs included a) time spent by the expert trainer conducting two 15-hour workshops (the first to train the trainers in MI, and the second to train the trainers in how to deliver MI workshops to clinicians), traveling to the program (including mileage reimbursement), certifying the trainers (including rating audiotaped trainer practice sessions, completing feedback forms, and providing phone-based feedback), and providing phone-based trainer supervision throughout the study, b) time spent by trainers in three 15-hour workshops (the third to deliver MI training to clinicians at the program), receiving phone-based feedback during the certification process, preparing materials to deliver the third workshop to clinicians, receiving phone-based supervision from the expert during the study, rating audiotaped clinician practice sessions and completing feedback forms, and providing face-to-face supervision of clinicians, c) time spent by clinicians in the third workshop, receiving face-to-face supervision from the trainers, and reviewing the training materials, d) space associated with the workshops and face-to-face supervision, e) express mail costs from the trainer certification process (i.e., audiotapes mailed to the expert for review), f) phonecalls between the expert and trainers during certification and supervision, and g) equipment (TV/VCR, tape recorders, microphones and audiocassette tapes) and materials for trainers and clinicians (books, manuals, training videos).
2.3.2 Normalized costs – based on eight clinicians per training cohort
As noted earlier, to allow direct comparisons of training costs across programs and arms, the actual economic costs incurred by the present study sample were normalized assuming eight clinicians per cohort. To this end, the actual economic costs were partitioned into fixed costs (i.e., costs that do not vary with the number of clinicians trained, such as the cost of the expert’s time delivering the workshops) vs. variable costs (i.e., costs that vary with the number of clinicians trained, such as the cost of the clinicians’ time in the workshops). Normalized costs were then obtained as “fixed cost + 8*(variable cost/# of clinicians in cohort).”
2.3.3. One-time costs vs. recurrent costs
One potential comparative advantage of the TT strategy is that the initial costs of preparing the trainers may be offset by lower clinician training and supervision costs for subsequent cohorts trained by the trainers. To examine how the relative costs and cost-effectiveness of the three training strategies would change as a function of the number of cohorts trained, the normalized training costs were partitioned into one-time costs vs. recurrent costs. One-time costs in the SS and EX arms were minimal, comprising the TV/VCR, training videos, tape recorders (EX only) and microphones (EX only). In contrast, one-time costs in the TT arm were substantial, comprising the two expert-led workshops, trainer certification activities, and expert supervision of the trainers. The normalized total cost of training any given number of cohorts was then obtained as “one-time cost + (# of cohorts) * recurrent cost.”
2.4 Effectiveness measure
Effectiveness in the present study was assessed in terms of the number of clinicians meeting a priori MI performance standards (as defined in section 2.1.2) in actual client sessions at the 12-week follow-up. This outcome was chosen for three reasons. First, although the training trial assessed effectiveness using a number of training outcomes (e.g., MI adherence scores, MI competence scores), these outcomes are relatively unintuitive for decision-makers to understand in economic terms. Said differently, it is more intuitive to think about “$ per additional clinician meeting MI performance benchmark standards” than “$ per point increase in MI adherence or competence scores.” Second, although the training trial made assessments at several points prior to the 12-week follow-up (baseline, post-workshop, post-supervision), the present study chose the 12-week follow-up to be conservative inasmuch as clinicians’ MI skills may be expected to diminish slightly following the end of supervision. Third, although the training trial assessed MI performance using both actual and role-played client sessions, the present study focused exclusively on MI performance using actual client sessions because (1) role-played client sessions might not reflect how clinicians implement MI with clients they treat within their programs, a critical assessment area when studying treatment dissemination and implementation strategies (Martino, in press), and (2) the only previous cost analysis of a clinician training strategy in the field of addiction assessed performance exclusively in terms of actual client sessions (Tober et al., 2005).
As noted earlier, to allow direct comparisons of effectiveness across programs and arms, the number of clinicians meeting MI performance standards at 12-week follow-up (who did not do so at baseline) was normalized assuming eight “below criterion” clinicians per cohort at baseline. For example, if five clinicians in a given cohort did not meet MI performance standards at baseline and two of them did so at the 12-week follow-up, then the normalized effectiveness of the training at the program would be 8*(2/5) = 3.2 clinicians.
2.5. Cost-effectiveness analysis
Following standard economic theory (Drummond et al., 2005; Gold et al., 1996), the ICERs are defined as the incremental cost of using a given training strategy, compared to the next least costly strategy, to produce an additional clinician meeting MI performance standards, and are presented as a function of the number of training cohorts expected. For a given number of cohorts, training strategies were eliminated if they were either strictly dominated (i.e., another strategy was less expensive and more effective) or extended dominated (i.e., a more costly strategy had a lower ICER). Economic theory then suggests that the most cost effective training strategy is the one with the greatest ICER that is less than the decision maker’s willingness to pay for an additional clinician meeting MI performance standards. Both incremental costs and incremental effects used to calculate the ICERs were based on normalized costs and normalized effects as described above.
Cost-effectiveness acceptability curves (CEACs) are presented to illustrate the statistical uncertainty in our study due to our sample (Briggs, 2001; Fenwick et al., 2001). CEACs show the probability that each training strategy would be the most cost-effective, given the observed data, under different assumptions about the value of an additional clinician meeting MI performance standards. Costs and effects for each training strategy were bootstrapped (with 2,000 replicates) to produce the CEACs (Briggs, 2001; Fenwick et al., 2001).
3. Results
3.1. Cost analysis
Tables 1-3 show the total economic cost of the SS ($6,141), EX ($28,020), and TT ($49,217) arms, respectively. Within each arm, costs are also presented by major component and program. In the SS arm, training materials (39%) and clinician time to review the training materials (35%) accounted for the largest portions of the overall cost. In the EX arm, overall costs comprised the workshop (58%), supervision (33%), and clinician time to review the training materials (9%). And in the TT arm, overall costs comprised the expert-led workshop to train the trainers in MI (27%), the expert-led workshop to train the trainers in how to give the MI workshop to clinicians (23%), certifying the trainers in MI (6%), the trainer-led workshop to train the clinicians in MI (31%), supervision (10%), and clinician time to review the training materials (4%).
Table 1.
Economic cost of SS training (in 2006 US dollars) – by program
| Component | All programs combined (n = 22) ($) | SS-1 (n = 4) ($) | SS-2 (n = 7) ($) | SS-3 (n = 6) ($) | SS-4 (n = 5) ($) |
|---|---|---|---|---|---|
| ORIENTATION MEETING | |||||
| Clinician time in meeting | 678 | 156 | 192 | 137 | 193 |
| Expert | |||||
| Time in meeting | 324 | 81 | 81 | 81 | 81 |
| Travel time | 440 | 110 | 110 | 110 | 110 |
| Mileage cost | 125 | 31 | 31 | 31 | 32 |
| Space costs | 6 | 1 | 1 | 1 | 3 |
| Materials/Equipment | 2399 | 527 | 682 | 630 | 560 |
| Orientation Subtotal | 3972 | 906 | 1097 | 990 | 979 |
| Clinician time reviewing training materials | 2169 | 273 | 927 | 342 | 627 |
| TOTAL COST | 6,141 | 1,179 | 2,024 | 1,332 | 1,606 |
Table 3.
Economic cost of TT training (in 2006 dollars) – by program
| Component | All programs combined (n = 19) ($) | TT-1 (n = 5) ($) | TT-2 (n = 4) ($) | TT-3 (n = 4) ($) | TT-4 (n = 6) ($) |
|---|---|---|---|---|---|
| WORKSHOP #1a | |||||
| Trainer time in workshop | 4851 | 1021 | 1112 | 1162 | 1556 |
| Expert | |||||
| Time in workshop | 4828 | 1206 | 1206 | 1210 | 1206 |
| Travel time | 873 | 218 | 218 | 219 | 218 |
| Mileage cost | 255 | 64 | 64 | 63 | 64 |
| Space costs | 102 | 7 | 36 | 33 | 26 |
| Materials | 2219 | 550 | 550 | 569 | 550 |
| Workshop #1 Subtotal | 13128 | 3066 | 3186 | 3256 | 3620 |
| WORKSHOP #2b | |||||
| Trainer time in workshop | 4851 | 1021 | 1112 | 1162 | 1556 |
| Expert | |||||
| Time in workshop | 4828 | 1206 | 1206 | 1210 | 1206 |
| Travel time | 873 | 218 | 218 | 219 | 218 |
| Mileage cost | 255 | 64 | 64 | 63 | 64 |
| Space costs | 102 | 7 | 36 | 33 | 26 |
| Materials | 166 | 41 | 41 | 43 | 41 |
| Workshop #2 Subtotal | 11075 | 2557 | 2677 | 2730 | 3111 |
| TRAINER CERTIFICATION | |||||
| Trainer time rec’ving feedback | 376 | 93 | 74 | 77 | 132 |
| Expert | |||||
| Time rating trainer tapes | 1449 | 402 | 322 | 323 | 402 |
| Time giving trainers feedback | 724 | 201 | 161 | 161 | 201 |
| Phonecalls | 143 | 36 | 35 | 37 | 35 |
| FedEx fees | 117 | 25 | 20 | 27 | 45 |
| Materials | 44 | 12 | 10 | 10 | 12 |
| Trainer Certification Subtotal | 2853 | 769 | 622 | 635 | 827 |
| WORKSHOP #3c | |||||
| Clinician time in workshop | 8409 | 1513 | 2170 | 1907 | 2819 |
| Trainers | |||||
| Time prepping workshop | 646 | 136 | 148 | 155 | 207 |
| Time in workshop | 4851 | 1021 | 1112 | 1162 | 1556 |
| Space costs | 110 | 15 | 36 | 33 | 26 |
| Materials | 1250 | 327 | 261 | 270 | 392 |
| Workshop #3 Subtotal | 15266 | 3012 | 3727 | 3527 | 5000 |
| SUPERVISION | |||||
| Trainers | |||||
| Time rec’ving feedback from PI | 240 | 51 | 56 | 58 | 75 |
| Time rating clinician tapes | 2052 | 477 | 445 | 426 | 704 |
| Time giving feedback to clinicians | 1025 | 238 | 222 | 213 | 352 |
| Expert time giving feedback to trainers | 483 | 121 | 121 | 121 | 120 |
| Clinician time rec’ving feedback from trainers | 727 | 142 | 217 | 175 | 193 |
| Space | 11 | 1 | 4 | 4 | 2 |
| Phonecalls | 95 | 24 | 23 | 25 | 23 |
| Materials | 123 | 34 | 29 | 27 | 33 |
| Supervision Subtotal | 4756 | 1088 | 1117 | 1049 | 1502 |
| Clinician time reviewing training materials | 2139 | 363 | 513 | 177 | 1086 |
| TOTAL COST | 49,217 | 10,855 | 11,842 | 11,374 | 15,146 |
In workshop #1, the expert trained the trainers in MI.
In workshop #2, the expert trained the trainers in how to deliver the MI workshop to and supervise clinicians.
In workshop #3, the trainers trained the clinicians in MI.
Tables 1-3 indicate that training costs vary considerably across programs within each arm, due primarily to (1) different clinician compensation rates across programs, and (2) different numbers of clinicians trained at each program. For example, the overall cost of TT at program TT-4 ($15,146) is much higher than at program TT-1 ($10,855) because (1) the average fully-loaded hourly wage (including fringe benefits and overhead) at program TT-4 ($30.35) was higher than its counterpart at program TT-1 ($19.54), and (2) program TT-4 (n = 6) trained more clinicians than program TT-1 (n = 5).
To allow direct comparisons of total economic costs across programs and arms, Table 4a shows the results of normalizing the total costs assuming eight clinicians per cohort at each program (thirty-two clinicians per arm). Remaining cost differences across programs within a given arm are due primarily to differences across programs in clinician compensation rates.
Table 4.
| a. Fixed, variable, and normalized total costs (in 2006 US dollars) – by program and training arm |
b. Normalized outcomes – by program and training arm |
||||||||
|---|---|---|---|---|---|---|---|---|---|
| Program | Fixed cost ($) | Variable cost ($) | Total cost ($) | # clinicians in cohort | Normalized total costa ($) | # clinicians in cohort | # clinicians meeting MI perf stds at follow-up | % of clinicians meeting MI perf stds at follow-up | Normalized # of clinicians meeting MI perf stds at follow-upb |
| SS – 1 | 543 | 636 | 1179 | 4 | 1815 | 4 | 0 | 0 | 0 |
| SS – 2 | 543 | 1481 | 2024 | 7 | 2236 | 7 | 2 | 29 | 2.3 |
| SS – 3 | 543 | 789 | 1332 | 6 | 1595 | 6 | 1 | 17 | 1.3 |
| SS – 4 | 535 | 1071 | 1606 | 5 | 2249 | 5 | 1 | 20 | 1.6 |
| SS – total | 2164 | 3977 | 6141 | 22 | 7,894 | 22 | 4 | 18 | 5.2 |
| EX – 1 | 2369 | 5912 | 8281 | 6 | 10252 | 6 | 2 | 33 | 2.7 |
| EX – 2 | 2353 | 5011 | 7364 | 4 | 12375 | 4 | 3 | 75 | 6.0 |
| EX – 3 | 2350 | 4387 | 6737 | 4 | 11124 | 4 | 2 | 50 | 4.0 |
| EX – 4 | 2368 | 3270 | 5638 | 3 | 11088 | 3 | 2 | 67 | 5.3 |
| EX – total | 9440 | 18580 | 28020 | 17 | 44,839 | 17 | 9 | 53 | 18.0 |
| TT – 1 | 7761 | 3094 | 10855 | 5 | 12711 | 5 | 2 | 40 | 3.2 |
| TT – 2 | 7981 | 3861 | 11842 | 4 | 15703 | 4 | 1 | 25 | 2.0 |
| TT – 3 | 8174 | 3200 | 11374 | 4 | 14574 | 4 | 2 | 50 | 4.0 |
| TT – 4 | 9570 | 5576 | 15146 | 6 | 17005 | 6 | 1 | 17 | 1.3 |
| TT - total | 33486 | 15731 | 49217 | 19 | 59,993 | 19 | 6 | 32 | 10.5 |
Normalized for 8 clinicians at each program. Normalized total cost = fixed cost + 8*(variable cost/# clinicians in cohort).
Normalized for 8 clinicians at each program. Normalized # of clinicians meeting MI performance standards at follow-up = 8*(% of clinicians meeting MI performance standards at follow-up).
All of the costs presented in Tables 1-4 relate to training the initial cohort at each program as implemented in the trial. Columns 2-4 of Table 5 show how the total costs of the three training strategies would change as a function of the number of additional cohorts expected to be trained (normalized to assume eight clinicians per cohort and four programs per arm). Results in Table 5 are based on one-time costs across the four programs in each arm of $1,270, $1,725, and $27,874 for SS, EX, and TT, respectively, and recurring costs of $6,624, $43,114, and $32,119 for SS, EX, and TT, respectively. Although one-time costs are minor for SS and EX, they account for 46% of the total training cost of the first cohort for TT. As a result, although TT is more expensive than EX in training one or two cohorts, TT is less expensive than EX for training three or more cohorts.
Table 5.
Normalized total costs, outcomes, and incremental cost-effectiveness ratios (in 2006 US dollars) - by number of training cohorts
| Normalized total training costa | Normalized # of clinicians meeting MI perf stds at follow-up | Normalized ICERs – Initial | Normalized ICERs – Final | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # of cohorts | SS ($) | EX ($) | TT ($) | SS | EX | TT | SS ($) | EX ($) | TT ($) | SS ($) | EX ($) | TT ($) |
| 1 | 7,894 | 44,839 | 59,993 | 5.2 | 18.0 | 10.5 | - | 2886b | SD | - | 2886b | SD |
| 2 | 14,518 | 87,953 | 92,112 | 10.4 | 36.0 | 21.0 | - | 2869b | SD | - | 2869b | SD |
| 3 | 21,142 | 131,067 | 124,231 | 15.6 | 54.0 | 31.5 | - | 304c | 6484d | - | 2863b | ED |
| 4 | 27,766 | 174,181 | 156,350 | 20.8 | 72.0 | 42.0 | - | 594c | 6065d | - | 2860b | ED |
| 5 | 34,390 | 217,295 | 188,469 | 26.0 | 90.0 | 52.5 | - | 769c | 5814d | - | 2858b | ED |
| 10 | 67,510 | 432,865 | 349,064 | 52.0 | 180.0 | 105.0 | - | 1117c | 5312d | - | 2854b | ED |
| 25 | 166,870 | 1,079,575 | 830,849 | 130.0 | 450.0 | 262.5 | - | 1327c | 5011d | - | 2852b | ED |
ICER = incremental cost-effectiveness ratio
SD = strictly dominated by EX
ED = extended dominated by the combination of SS and EX
One-time costs across the four programs in each arm were $1,270, $1,725, and $27,874 for SS, EX, and TT, respectively. Recurrent costs were $6,624, $43,114, and $32,119 for SS, EX, and TT, respectively.
Defined as (column #3 – column #2)/(column #6 – column #5)
Defined as (column #3 – column #4)/(column #6 – column #7)
Defined as (column #4 – column #2)/(column #7 – column #5)
3.2. Effectiveness measure
Column 2 of Table 4b shows the number of clinicians meeting MI performance standards at 12-week follow-up (who did not do so at baseline) at each program. To be consistent with the normalized costs in Table 4a, the effectiveness results are normalized assuming eight clinicians per cohort at each program (thirty-two clinicians per arm). As seen in Column 4 of Table 4b, the normalized effectiveness results were fairly consistent across the programs within each arm. Aggregating the normalized effectiveness results across programs within each arm (i.e., assuming each arm started with thirty-two clinicians not meeting MI performance standards at baseline), the SS, EX, and TT training strategies would be expected to produce 5.2, 18.0 and 10.5 clinicians meeting MI performance standards at 12-week follow-up, respectively.
3.3. Cost-effectiveness analysis
Columns 8-10 of Table 5 present the initial ICERs of the three training strategies as a function of the number of training cohorts expected. As seen in column 10, TT is strictly dominated by EX for one or two cohorts and extended dominated by the combination of SS and EX for three or more cohorts. Accordingly, the ICERs were recalculated excluding TT and are presented in columns 11-13. As seen in columns 11-13, compared to SS, the incremental cost of using EX to produce an additional clinician meeting MI performance standards is approximately $2,870, ranging from $2,886 for the initial cohort to $2,852 for twenty-five cohorts.
Figure 1a and Figure 1b show the CEACs for one and twenty-five training cohorts, respectively. Each CEAC shows the probability that a given training strategy is the most cost-effective given the observed data (Fenwick et al., 2001). Note that CEACs are a function of the threshold willingness-to-pay of the decision maker for an additional unit of outcome. For example, if only one training cohort is expected and the threshold willingness-to-pay of the decision maker for an additional clinician meeting MI performance standards is $2000, then Figure 1a shows that the probability that SS, EX, and TT is the most cost-effective strategy is 94%, 6%, and 0%, respectively. In contrast, if the threshold value is $5000, then the probability that SS, EX, and TT is the most cost-effective strategy is 7%, 91%, and 2%, respectively. Due to economies of scale associated with TT’s comparatively large one-time costs, if twenty-five training cohorts are expected then Figure 1b shows that TT has a non-trivial (though still quite low) probability of being the most cost-effective strategy across a wide range of threshold values. Note that even though TT was dominated (either strictly or extended) for all numbers of training cohorts, it is included in the CEACs in Figures 1a and 1b to avoid underestimating the uncertainty in SS and EX (Olmstead et al., 2007).
Figure 1.
Cost-effectiveness acceptability curves for clinicians meeting MI performance standards based on one cohort (a), and twenty-five cohorts (b). For a given threshold value, the probability of being cost effective is equivalent to the proportion of the 2,000 bootstrapped replicates for which each training strategy (SS, EX, and TT) had the highest net benefit (Fenwick et al., 2001).
Finally, if 100% of the time that clinicians spent reviewing the training materials was done “on company time,” as opposed to 25% as occurred in this study, then (1) the normalized total cost of SS, EX, and TT would have been $17,094, $58,848, and $70,230, respectively; (2) TT would still be dominated (either strictly or extended) for all numbers of training cohorts; and (3) compared to SS, the incremental cost of using EX to produce an additional clinician meeting MI performance standards would have been approximately $3,250, ranging from $3,262 for the initial cohort to $3,228 for twenty-five cohorts.
4. Discussion
This study estimated the cost and cost-effectiveness of three strategies – SS, EX, and TT – for teaching community program clinicians to implement MI. Focusing only on costs, SS was the least expensive training strategy, followed by TT (assuming at least three training cohorts), and EX. Although EX was the costliest training strategy, it was also the most effective. In terms of cost-effectiveness, SS is likely to be the most cost-effective training strategy if the threshold value to decision-makers of an additional clinician meeting MI performance standards is less than approximately $2,870 (i.e., $2,852 - $2,886, depending on the number of training cohorts expected), and EX is likely to be the most cost-effective strategy when the threshold value is greater than approximately $2,870. In economic parlance, TT was dominated by EX for one or two training cohorts (i.e., TT was more costly and less effective than EX), and TT was extended dominated by the combination of EX and SS for three or more cohorts.
To our knowledge, this is only the second study to estimate the costs of any type of clinician training strategy in addiction, and the first study to examine the relative cost-effectiveness of two or more such training strategies. Tober et al. (2005) found that the costs of an expert-led workshop followed by supervision (similar to EX in the present study) averaged £1,260 and £1,390 per clinician “trained to criterion” in motivational enhancement therapy and social behavior and network theory, respectively (2000/2001 prices). Converting to 2006 US dollars (conversion factor = 1.92; http://www.measuringworth.com/calculators/exchange/), these training costs equate to $2,419 and $2,669, both of which are comparable to EX’s average training cost per clinician meeting MI performance standards as found in the present study ($2,491 = $44,839/18.0).
As noted by Tober et al. (2005), training is an investment expected to generate benefits into the future. That is, although training costs are substantial, when spread over time they account for only a small fraction of the total cost of treatment. For example, assuming that (a) five cohorts are trained at a given program, (b) a MI-trained clinician can deliver 736 MI sessions to clients per year (i.e., 46 weeks X 16 sessions per week), and (c) the effects of training would endure for only one year, then the training costs for SS, EX, and TT would add only $1.80 (i.e., ), $3.28 (i.e., ), and $4.88 (i.e., ) per client session, respectively. As a practical matter, the normalized yield of MI-trained clinicians was low for all three training strategies in the trial (16% in SS, 56% in EX, 33% in TT). Two suggestions for potentially improving the yield of trained clinicians would be to increase the time available at the program for clinicians to review the training materials (SS, EX, and TT) and increase the frequency and/or lengthen the duration of supervision (EX and TT only). Of course, both of these suggestions would also increase training costs, so it is an empirical question as to whether either suggestion would improve the overall cost-effectiveness of the training strategies.
A related issue with cost implications is the criterion used to define adequate MI performance. To date, no empirically derived training criteria directly linked to client outcomes exist for any evidence-based behavioral treatment. While it is not certain if the criterion level of MI performance used in this study would translate to improved clinical outcomes, it was based on the MI certification standard used to train clinicians in several multi-site effectiveness trials wherein MI relative to treatment-as-usual achieved improved program retention or substance use outcomes (Ball et al., 2007; Carroll et al., 2006, 2009). If a higher performance criterion is needed to improve client care, presumably the motive for which program directors would agree to incur training expenses, then the cost of training would increase and potentially change the cost-effectiveness of these strategies.
The present study has several strengths. First, it is based on a randomized training trial conducted at 12 community treatment programs. Second, all costs were collected prospectively from the trial participants. Third, costs are presented in a variety of ways to be useful to decision makers (e.g., by major component, fixed vs. variable, one-time vs. recurrent, normalized for eight clinicians per cohort, etc.). Finally, in the absence of consensus threshold values for an additional MI-trained clinician, ICERs and CEACs are presented that define the range of values over which each training strategy would likely be the most cost-effective. Decision makers can use this information in combination with their own evaluation of the value of an additional MI-trained clinician to make policy decisions.
There are also several limitations. First, although the training trial implemented SS, EX, and TT as they are commonly implemented in the field, other implementations are possible which might alter the costs and clinician outcomes. For example, the median and modal number of clinicians per cohort was eight in the trial, but a different number (e.g., seven or nine) would have altered the costs (and perhaps the clinician outcomes) slightly. Second, the results in Table 5 depend on the following two assumptions: (1) trainers remain at their respective programs, and (2) the trainers do not get better or worse over time at delivering the MI workshop and supervising clinicians. The first assumption seems reasonable inasmuch as seven out of the eight trainers in the trial remained at their respective programs for at least one year following the end of the trial. The second assumption has yet to be tested empirically. Finally, the logistical costs included in Tables 1-3 are likely to vary depending on the proximity of experts to programs. Fortunately, the costs in these tables are modular and decision makers can easily insert their own expected logistics costs in place of those incurred in the trial.
In conclusion, this study provides an important analysis of the cost and cost-effectiveness of three commonly used clinician training strategies in addiction. The detailed cost information provided here should be useful for decision makers seeking to implement empirically supported treatments in addiction with scarce resources. In addition, the results demonstrating that training costs are a relatively minor component of total treatment costs may encourage more programs to adopt empirically supported treatments in the future.
Table 2.
Economic cost of EX training (in 2006 dollars) – by program
| Component | All programs combined (n = 17) ($) | EX-1 (n = 6) ($) | EX-2 (n = 4) ($) | EX-3 (n = 4) ($) | EX-4 (n = 3) ($) |
|---|---|---|---|---|---|
| WORKSHOP | |||||
| Clinician time in wrkshp | 7914 | 2605 | 2449 | 1359 | 1501 |
| Expert | |||||
| Time in workshop | 4840 | 1210 | 1210 | 1210 | 1210 |
| Travel time | 876 | 219 | 219 | 219 | 219 |
| Mileage cost | 252 | 63 | 63 | 63 | 63 |
| Space costs | 57 | 22 | 7 | 6 | 22 |
| Materials | 2429 | 726 | 590 | 590 | 523 |
| Workshop Subtotal | 16368 | 4845 | 4538 | 3447 | 3538 |
| SUPERVISION | |||||
| Clinician time receiving feedback | 746 | 241 | 224 | 131 | 150 |
| Expert | |||||
| Time rating tapes | 3951 | 1372 | 888 | 965 | 726 |
| Time giving feedback | 1975 | 686 | 444 | 482 | 363 |
| Travel time | 1314 | 329 | 329 | 327 | 329 |
| Mileage | 378 | 94 | 94 | 96 | 94 |
| Space | 7 | 2 | 1 | 2 | 2 |
| FedEx fees | 311 | 109 | 70 | 74 | 58 |
| Materials | 567 | 155 | 140 | 137 | 135 |
| Supervision Subtotal | 9249 | 2988 | 2190 | 2214 | 1857 |
| Clinician time reviewing training materials | 2403 | 448 | 636 | 1076 | 243 |
| TOTAL COST | 28,020 | 8,281 | 7,364 | 6,737 | 5,638 |
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- Ball SA, Martino S, Corvino J, Morganstern J, Carroll KM. Independent tape rater guide. Unpublished psychotherapy tape rating manual 2002 [Google Scholar]
- Ball SA, Martino S, Nich C, Frankforter TL, Van Horn D, Crits-Christoph P, Woody GE, Obert JL, Farentinos C, Carroll KM. Site matters: Motivational enhancement therapy in community drug abuse clinics. J Consult Clin Psych. 2007;75:556–567. doi: 10.1037/0022-006X.75.4.556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Beidnas RS, Kendall PC. Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010;17:1–30. doi: 10.1111/j.1468-2850.2009.01187.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Briggs A. Handling uncertainty in economic evaluation and presenting the results. In: Drummond M, McGuire A, editors. Economic Evaluation in Health Care: Merging Theory with Practice. Oxford University Press; Oxford, UK: 2001. pp. 172–214. [Google Scholar]
- Carroll KM, Ball SA, Nich C, Martino S, Frankforter TL, Farentinos C, Kunkel L, Mikulich-Gilbertson S, Morgenstern J, Obert JL, Polcin D, Snead N, Woody GE. Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: A multisite effectiveness study. Drug Alcohol Depend. 2006;81:301–312. doi: 10.1016/j.drugalcdep.2005.08.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Carroll KM, Martino S, Suarez-Morales L, Ball SA, Miller WR, Añez L, Paris M, Nich C, Frankforter TL, Matthews J, Farentinos C, Szapocznik J. Multisite randomized controlled effectiveness trial of motivational enhancement for Spanish-speaking substance users. J Consult Clin Psych. 2009;77:993–999. doi: 10.1037/a0016489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cucciare MA, Weingardt KR, Villafranca S. Using blended learning to implement evidence-based psychotherapies. Clin Psychol Sci Pract. 2008;15:299–307. [Google Scholar]
- Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programs. third. Oxford University Press; Oxford, UK: 2005. [Google Scholar]
- Fenwick E, Claxton K, Schulpher M. Representing uncertainty: The role of cost-effectiveness acceptability curves. Health Econ. 2001;10:779–787. doi: 10.1002/hec.635. [DOI] [PubMed] [Google Scholar]
- Gibbons C, Carroll KM, Ball SA, Nich C, Frankforter TL, Martino S. Community program therapist adherence and competence in a one-session motivational interviewing assessment. Am J Drug Alcohol Abuse. doi: 10.3109/00952990.2010.500437. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Giuseppe C, Clerici M. Dual diagnosis—policy and practice in Italy. Am J Addict. 2006;15(2):125–30. doi: 10.1080/10550490500528340. [DOI] [PubMed] [Google Scholar]
- Gold MR, Siegel JE, Russell LB, Weinstein MC. Cost-Effectiveness in Health and Medicine. Oxford University Press; Oxford, UK: 1996. [Google Scholar]
- Hawkins KA, Sinha R. Can line clinicians master the conceptual complexities of dialectical behavior therapy? An evaluation of a state department of mental health training program. J Psychiatr Res. 1998;32:379–384. doi: 10.1016/s0022-3956(98)00030-2. [DOI] [PubMed] [Google Scholar]
- Hein D, Litt LC, Cohen L, Miele GM, Campbell A. Trauma services for women in substance abuse: an integrated approach. Washington DC: American Psychological Association; 2009. Preparing, training, and integrating staff in providing integrated treatment; pp. 143–161. [Google Scholar]
- Hintz T, Mann K. Co-occurring disorders: policy and practice in Germany. Am J Addict. 2006;15(4):261–7. doi: 10.1080/10550490600754275. [DOI] [PubMed] [Google Scholar]
- Lundahl BW, Tollefson D, Gambles C, Brownell C, Burke BL. A meta-analysis of motivational interviewing: Twenty five years of research. Res Soc Work Pract. 2010;20:137–160. [Google Scholar]
- Madson MB, Loignon AC, Lane C. Training in motivational interviewing: A systematic review. J Subst Abuse Treat. 2009;36:101–109. doi: 10.1016/j.jsat.2008.05.005. [DOI] [PubMed] [Google Scholar]
- Martino S. Delivering evidence-based treatments: Strategies for counselor training. Addict Sci Clin Pract. in press. [PMC free article] [PubMed] [Google Scholar]
- Martino S, Ball SA, Nich C, Canning-Ball M, Rounsaville BJ, Carroll KM. Teaching community program clinicians motivational interviewing using expert and train-the-trainer strategies. Addict. doi: 10.1111/j.1360-0443.2010.03135.x. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug Alcohol Depend. 2008;96:37–48. doi: 10.1016/j.drugalcdep.2008.01.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Martino S, Giannini F, Ball SA, Carroll KM. A motivational interviewing manual for community treatment program clinicians. Unpublished psychotherapy manual 2004 [Google Scholar]
- Miller WR, Rollnick S. Motivational Interviewing: Preparing people for change. 2. Guilford Press; New York: 2002. [Google Scholar]
- Miller WR, Rollnick S, Moyers TB. Motivational interviewing (6-tape series) University of New Mexico 1998 [Google Scholar]
- Miller WR, Sorensen JL, Selzer JA, Brigham GS. Disseminating evidence-based practices in substance abuse treatment: a review with suggestions. J Subst Abuse Treat. 2006;31(1):25–39. doi: 10.1016/j.jsat.2006.03.005. [DOI] [PubMed] [Google Scholar]
- Miller WR, Yahne CE, Moyers TE, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivational interviewing. J Consult Clin Psych. 2004;72:1050–1062. doi: 10.1037/0022-006X.72.6.1050. [DOI] [PubMed] [Google Scholar]
- Miller WR, Zweben J, Johnsen WR. Evidence-based treatment: Why, what, where, when, and how? J Subst Abuse Treat. 2005;29:267–276. doi: 10.1016/j.jsat.2005.08.003. [DOI] [PubMed] [Google Scholar]
- Olmstead TA, Ostrow C, Carroll KM. Cost effectiveness of computer-assisted training in cognitive-behavioral therapy as an adjunct to standard care for addiction. Drug Alcohol Depend. 2010;110:200–207. doi: 10.1016/j.drugalcdep.2010.02.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Olmstead TA, Sindelar JL, Easton CJ, Carroll K. The cost-effectiveness of four treatments for marijuana dependence. Addiction 102. 2007;9:1443–1453. doi: 10.1111/j.1360-0443.2007.01909.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Reickmann TR, Kovas AE, Fussell HE, Stettler NM. Implementation of evidence-based practices for treatment of alcohol and drug disorders: the role of the state authority. J Beh Health Ser Res. 2009;36(4):407–19. doi: 10.1007/s11414-008-9122-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Santa Ana EJ, Carroll KM, Añez L, Paris M, Ball SA, Nich C, Frankforter TL, Suarez-Morales L, Szapocznik J, Martino S. Evaluating motivational enhancement therapy adherence and competence among Spanish-speaking therapists. Drug Alcohol Depend. 2009;103:44–51. doi: 10.1016/j.drugalcdep.2009.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive behavioral therapy. J Consult Clin Psych. 2005;73:106–115. doi: 10.1037/0022-006X.73.1.106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tober G, Godfrey C, Parrot S, Copello A, Farrin A, Hodgson R, Kenyon R, Morton V, Orford J, Russell I, Slegg G. Setting standards for training and competence: The UK alcohol treatment trial. Alcohol Alcohol. 2005;40:413–418. doi: 10.1093/alcalc/agh181. [DOI] [PubMed] [Google Scholar]
- Walters ST, Matson SA, Baer JS, Ziedonis DM. Effectiveness of workshop training for psychosocial addiction treatments: A systematic review. J Subst Abuse Treat. 2005;29(4):283–93. doi: 10.1016/j.jsat.2005.08.006. [DOI] [PubMed] [Google Scholar]