Abstract
Objective:
Costs of supporting prevention program implementation are not well known. This study estimates the societal costs of implementing CHOICE, a voluntary after-school alcohol and other drug prevention program for adolescents, in Boys and Girls Clubs (BGCs) across Southern California with and without an implementation support system called Getting To Outcomes© (GTO).
Method:
This article uses micro-costing methods to estimate the cost of the CHOICE program and GTO support. Labor and expense data were obtained from logs kept by the BGC staff and by the GTO technical assistance (TA) staff, and staff time was valued based on Bureau of Labor Statistics estimates.
Results:
From the societal perspective, the cost of implementing CHOICE at BGCs over the two-year study period was $27 per attendee when CHOICE was offered by itself (all costs incurred by the BGCs) and $177 per attendee when CHOICE was offered with GTO implementation support ($67 cost to the BGCs; $110 to the entity funding GTO). These results were most sensitive to assumptions as to the number of times CHOICE was offered per year.
Conclusions:
Adding GTO implementation support to CHOICE increased the cost per attendee by approximately $150. For this additional cost there was evidence that the CHOICE program was offered with more fidelity and offered more often after the two-year intervention ended. If the long-term benefits of this better and continued implementation are found to exceed these additional costs, GTO could be an attractive structure to support evidence-based substance misuse prevention programs.
Keywords: implementation support, evidence-based programs, adolescent prevention programs, substance use prevention, cost analysis
Introduction
The adoption and rigorous implementation of evidence based programs (EBPs) for youth substance misuse remains low even in the face of a wide array of available potentially effective EBPs (Ennett et al. 2003; Hallfors and Godette 2002; Kumar et al. 2013; Ringwalt et al. 2008; Ringwalt et al. 2009). Various agencies such as Blueprints for Healthy Youth Development (Institute of Behavioral Science 2019) and the Clearinghouse for Military Family Readiness at Penn State (Clearinghouse for Military Family Readiness 2018) offer lists of effective EBPs for preventing substance misuse. However, when EBPs are attempted in new contexts, fidelity often does not reach expectations set by program developers, leading to suboptimal outcomes (Wandersman and Florin 2003; Livet and Wandersman 2005). This gap between research and community practice exists because prevention researchers often do not consider the flexibilities needed in settings other than those where initial studies were conducted and often do not specify core components that need to be adhered to, EBP purveyors do not always provide sufficient guidance on all the implementation best practices needed to make an EBP successful (goal setting, tailoring programs to fit local structures, planning, evaluation and quality improvement), and community organizations often do not engage in these implementation best practices.
One method to address the multiple causes of this gap and thus improve the implementation and outcomes of EBPs is an implementation support intervention called Getting To Outcomes® (GTO) (Chinman et al. 2015). This intervention is specifically designed to help community organizations tailor EBPs to their own setting (i.e., account for the local context) and use best practices such as setting realistic goals, thoughtful planning, and carrying out ongoing evaluation, and quality improvement. Implementation support systems such as GTO have been found to improve the implementation quality of EBPs which in turn can improve outcomes (Chinman et al. 2016a; Hawkins et al. 2009; Durlak and DuPre 2008; Spoth et al. 2007). However, this support has a cost, which is in addition to the cost of the implementation of the EBP itself. To determine whether this additional cost is worthwhile, it must be quantified and compared to the benefits gained. Costs have important policy implications because federal (i.e., SAMHSA) and state governments spend millions each year on program funding and implementation support, and it is unclear whether these funds are a worthwhile investment.
Cost analyses have been published for two other implementation support interventions focused on youth substance misuse prevention: PROmoting School-Community-University Partnerships to Enhance Resilience (PROSPER) (Crowley et al. 2012) and Communities That Care (CTC) (Kuklinski et al. 2012). Both focused on providing broader community support for the implementation of EBPs and did not specify the EBP to use. The cost analysis of PROSPER (Crowley et al. 2012) did not compare its costs with its benefits. However, the analysis of the CTC program did compare its costs to monetized values for two of its outcomes: reductions in cigarette smoking and delinquency (Hawkins et al. 2009). The cost-benefit analysis of CTC reported a return over 4 years of $5.30 for every dollar spent on CTC (Kuklinski et al. 2012).
This cost analysis documents the costs of implementing a substance use EBP called CHOICE (D’Amico et al. 2012) in a sample of community providers–Boys and Girls Clubs (BGCs)–with and without GTO implementation support. These costs can then be compared to the outcomes seen to allow determination of whether the additional cost of adding GTO is worth its benefits (outcome analyses are presented in a separate report (Chinman et al. 2018); the issue of moderation of outcomes is being addressed in another report under review).
Methods
This cost analysis is for Preparing to Run Effective Prevention (PREP), a two-year, cluster-randomized controlled trial comparing CHOICE to CHOICE+GTO in a sample of BGCs in Southern California. The trial is described below. Another publication provides more detail on this trial, the GTO approach, and its outcomes (Chinman et al. 2018). This study was approved by RAND’s Human Subjects Protection Committee. All subjects provided consent to participate.
Setting and participants
This study estimates the cost of implementing CHOICE with and without GTO across 15 BGC clubs comprising 29 individual, independently functioning sites in Southern California. The BGCs were recruited at meetings of the Los Angeles Alliance for Boys and Girls Clubs and all area BGCs were eligible. One-to-one randomization using a random number generator resulted in 15 BGC sites in the CHOICE-only group and 14 BGC sites in the CHOICE+GTO group, and all were included in the final analysis (Figure 1). The study’s Principal Investigator notified each site as to its allocation. Despite some variability, each site had its own facility and a small number of full- and part-time staff (n = 7–10). A subset of staff (between 1 and 10; mean = 2.2, median = 3) at each site participated in the study. As described in our previous publication (Chinman et al. 2018), to assess baseline differences between groups, we compared the participating staff at each site by group on their demographic characteristics and on their attitudes and toward evidence-based practices (Aarons et al. 2010), which has been shown to influence the implementation of such practices. There were no differences between the groups on staff demographics or on attitudes toward evidence-based practices. Half (49%) of the staff were female; most (59%) were over 25 years old; half (50%) had a 4-year college degree or more; and 56% were Hispanic or Latinx, 22% were non-Hispanic African-American and 22% were non-Hispanic White, multiracial, or of other races.
Figure 1.
CONSORT Flow Diagram
Each BGC site was asked to implement CHOICE in different groups of adolescents each year over a two-year period from May 2014 to April 2016. Each site was asked to recruit 20 adolescents 11–14 years of age for the program each time. The sites sent information flyers to parents, approached parents at the site, and/or held special information sessions at the site to attract attendees. Some sites offered snacks or special events as incentives for attendance. Each BGC site received $2,000 a year to defray some costs of participating in the study.
CHOICE
CHOICE is a voluntary program targeting middle-school youth to prevent and reduce alcohol and drug use. It provides youth with normative information to help them better understand their peers’ use, helps youth examine the pros and cons of use, and provides skills training (D’Amico and Edelen 2007). Each cycle of CHOICE is made up of a set of five half-hour weekly sessions. Existing BGC staff at each site received a CHOICE curriculum manual and were trained in the curriculum and in motivational interviewing (Miller and Rollnick 2002; Rollnick et al. 2008), a key theory underlying the CHOICE program, to enable them to conduct each session. Two randomized trials have shown CHOICE to reduce alcohol and marijuana use (D’Amico et al. 2012; D’Amico and Edelen 2007). Two half-time, Masters-level technical assistance (TA) providers delivered standard training and CHOICE manuals to all sites.
Getting To Outcomes (GTO)
In addition to the CHOICE manuals and training above, those in the CHOICE+GTO group sites also received GTO manuals, which contain text and tools published by RAND Corporation (Chinman et al. 2016b); face-to-face training in GTO; and ongoing, onsite, proactive TA to support CHOICE implementation for two years The minimum time period for a full implementation of GTO is two years to allow first-year evaluation data to be used in quality improvement planning process for the second year. The GTO manual contains written guidance about how to complete GTO steps, with each step being a different set of implementation best practices important to successfully carrying out an EBP. Most GTO steps contain “tools” or worksheets that prompt practitioners to make, and then record, decisions about various practices.
In the first year of this study, both groups received training in CHOICE and then implemented the program. Therefore, no real difference in the costs or outcomes associated with CHOICE were expected in the first year. During this year, across all sites, about a third of CHOICE curriculum activities were observed and rated on their fidelity by trained observers. However, only the CHOICE+GTO group were provided the results of this observation and assistance by TA providers to conduct quality improvement activities on their program delivery, and additional training on CHOICE in year 2. The CHOICE-only group was asked to implement the program again in year 2 without this additional feedback, quality improvement, and training. Implementation of the CHOICE program beyond this second year was at discretion of the site.
Outcomes
Details on Year 1 and Year 2 outcomes for CHOICE+GTO and CHOICE are published elsewhere (Chinman et al. 2018). Briefly, the study assessed fidelity (curriculum adherence, quality of CHOICE delivery, dosage) and the alcohol and drug proximal outcomes of participating middle school youth—i.e., attitudes and intentions regarding cigarettes, alcohol, and marijuana use. Fidelity was assessed at all sites by observer ratings and attendance logs. Proximal outcomes were assessed via survey at baseline, 3 months, and 6 months. GTO support ended after 2 years, but data on fidelity were gathered in Year 3 for the sites that continued to offer the CHOICE program. Data were also gathered in Years 3 and 4 on the number of sites that offered CHOICE and attendance of youth, and in Year 4 sites were asked their plans regarding future implementation of CHOICE.
Costing
Our cost estimates used micro-costing methods (Barnett 2009; Frick 2009) and are presented in 2015 USD so that costs from three perspectives are possible: 1) that of the BGCs; 2) that of a future federal or state funding agency that would cover the cost of providing GTO; and 3) the societal perspective, which includes all non-zero costs no matter who pays for them and in this analysis is the sum of the other two.
We captured resource use using cost logs kept by GTO and BGC staff. Unit costs for time, mileage and photocopies, and the sources for those cost estimates are shown in Table 1. Transportation costs for GTO staff visits to sites and for BGC staff for GTO and CHOICE-related travel (i.e., not including their normal commutes) were estimated as their time plus miles driven. The cost of supplies was based on actual costs incurred and reported by the BGC sites.
Table 1.
Unit costs used in the cost analysis
| Cost category | Unit Cost | Source |
|---|---|---|
| GTO technical assistance supervisor time per hour | $46.68 | Estimated 2015 Employer Cost of Employee Compensation for a “Social and Community Service Managers” (11–9151)a |
| GTO technical assistance staff time per hour | $29.83 | Estimated 2015 Employer Cost of Employee Compensation for a “Community and Social Service Specialists, All Other” (21–1099)b |
| BGC staff time per hour | $20.97 | Estimated 2015 Employer Cost of Employee Compensation for a “Social and Human Services Assistant” (21–1093)c |
| Transportation cost per mile | $0.575 | US General Services Administration 2015 rates for privately owned automobilesd |
| Printing/copying cost per sheet | $0.10 | Rough estimate based on ranges available online |
GTO = Getting to Outcomes implementation support system; BGC = Boys and Girls Club
Found at: https://www.bls.gov/oes/2015/may/oes119151.htm. Accessed 11/27/17.
Found at: https://www.bls.gov/oes/2015/may/oes211099.htm. Accessed 11/27/17.
Found at: https://www.bls.gov/oes/2015/may/oes211093.htm. Accessed 11/27/17.
Cost per attendee was calculated using initial enrollment at each site since that is the number of students each site must prepare to serve for each set of CHOICE sessions. We assumed no (zero) opportunity cost to participant adolescent time or productivity, no parent time or productivity costs, no adolescent (or parent) transportation costs, and no facility costs as all participants were members of their BGCs and would have spent time there in any case during the CHOICE session periods. These costs likely cannot be assumed to be zero when considering the implementation of CHOICE and GTO in other settings. Research costs related to data collection and study support were excluded from this cost analysis.
Analyses and sensitivity analysis
Comparisons between groups utilized t tests for continuous variables and χ2 for frequencies. We used one-way sensitivity analyses to explore the variation in our cost estimates under alternative assumptions (Drummond et al. 2005). The cost estimates we present reflect CHOICE+GTO being offered to 14 BGC sites and CHOICE alone in 15 BGC sites in Southern California over a 2-year period, with each site providing one set of the CHOICE sessions per year. There were few, if any, costs involved in adding one more attendee to the CHOICE sessions. Given that, having more (or fewer) attendees per site will directly lower (or increase) the average cost per attendee. Therefore, we focus on cost per site. Certain costs will differ depending on the number of sites being trained at a time (i.e., CHOICE trainer costs) or being managed by TA at a time (i.e., TA supervision costs). Other costs will differ depending on the distance TA staff need to travel to BGC sites and the time that travel takes (i.e., TA travel time and mileage costs), and yet other costs will not need to be repeated (i.e., are possibly one-time costs per year) if additional sets of CHOICE sessions are provided each year (i.e., CHOICE training and all GTO costs). The marginal cost of adding an additional set of CHOICE sessions beyond the two-year GTO training period would be the same as adding an additional set of CHOICE sessions during a year.
Results
Enrollment by site in the two groups was very similar for both years. In the first year the CHOICE+GTO group (14 sites) averaged 16.9 (SD = 3.4) attendees per site and the CHOICE-only group (15 sites) averaged 16.0 (SD = 3.9) attendees (p = .533). In the second year there were 14.7 (SD = 3.9) attendees for CHOICE+GTO and 14.6 (SD = 5.1) for CHOICE (p = .946). The CHOICE-only attendees included more females (56%) than the CHOICE+GTO group (47%), p = .007. However, for both groups across both years the median age was 11 years, the median school grade was 7th, and two-thirds of both groups identified as Latinx.
Table 2 shows resource use in terms of hours and mileage for each group for each year. Although the BGC staff hours for CHOICE training and implementation were higher in year 1 for the CHOICE+GTO group than for the CHOICE group, they were not significantly higher (p = .584 and p = .538, respectively). However, in year 2, there was no CHOICE training in the CHOICE group, and the hours spent implementing CHOICE were significantly lower in the CHOICE group than in the CHOICE+GTO group (p = .011).
Table 2.
Resources used, Mean (SD)
| CHOICE+GTO Group (n=14 sites) | CHOICE Group (n=15 sites) | ||
|---|---|---|---|
| Year 1 Resource use - hours | GTO Staff Costs | BGC Costs | BGC Costs |
| CHOICE training - BGC staff | 14.3 (6.2) | 10.9 (4.5) | |
| CHOICE training - trainer | 2.6 (1.0) | 2.4 (0.7) | |
| CHOICE implementation* | 15.7 (14.6) | 12.9 (8.6) | |
| GTO training and TA at the site | 11.2 (3.4) | 17.5 (8.4) | |
| Travel time to sites for TA | 10.1 (4.2) | ||
| GTO training non-site, non-travel preparation† | 10.2 (--) | ||
| TA supervision - supervisor time† | 5.3 (--) | ||
| TA supervision - TA time† | 7.0 (--) | ||
|
Year 1 Resource use – miles and number of trips for TA | |||
| CHOICE implementation | 2.6 (5.4) | 23.2 (31.4) | |
| Mileage to sites for TA | 220.7 (92.3) | ||
| Number of round trips for TA | 4.9 (1.6) | ||
|
Year 2 Resource use - hours | |||
| CHOICE training - BGC staff | 6.0 (6.2) | ||
| CHOICE training - trainer | 2.6 (3.5) | ||
| CHOICE implementation‡ | 12.2 (4.7) | 7.7 (4.2) | |
| GTO training and TA at the site | 14.7 (3.9) | 19.2 (8.6) | |
| Travel time to sites for TA | 10.4 (4.6) | ||
| GTO training non-site, non-travel preparation† | 15.4 (--) | ||
| TA supervision - supervisor time† | 5.7 (--) | ||
| TA supervision - TA time† | 7.0 (--) | ||
|
Year 2 Resource use – miles and number of trips for TA | |||
| CHOICE implementation | 4.0 (8.6) | 1.4 (3.5) | |
| Mileage to sites for TA | 229.0 (112.2) | ||
| Number of round trips for TA | 5.2 (2.4) | ||
BGC = Boys and Girls Club; CHOICE = an evidence-based program to prevent substance misuse in youth; GTO = Getting to Outcomes implementation support system; TA = GTO technical assistance
The differences between groups in CHOICE training and implementation hours by site in year 1 were not statistically significant: p = .584 and p = .538, respectively.
The use of these resources was not tracked by site; thus, only site averages are available.
In year 2 BGC staff in the CHOICE+GTO group sites spent significantly more hours on CHOICE implementation than those in the CHOICE only group: p = .011.
Table 3 shows per site costs of each program component and totals per attendee for each study group per site by year. Except for other expenses which were entered as reported, all costs were calculated by multiplying the resource use from Table 2 by the unit costs in Table 1. As would be expected from Table 2, the cost to train for and implement CHOICE in each group is similar in year 1 (p=.141 per site, p=.215 per attendee). However, in year 2 the cost of CHOICE differed substantially and significantly between groups (p<.001 per site, p=.012 per attendee).
Table 3.
Resource costs by component and year, Mean (SD)
| CHOICE+GTO Group (n=14 sites) | CHOICE Group (n=15 sites) | ||
|---|---|---|---|
| Year 1 | GTO Staff Costs | BGC Costs | BGC Costs |
| CHOICE training - BGC staff | $300 ($129) | $229 ($95) | |
| CHOICE training - trainer | $77 ($29) | $72 ($20) | |
| CHOICE implementation | $329 ($307) | $270 ($179) | |
| CHOICE implementation – miles | $2 ($3) | $13 ($18) | |
| CHOICE implementation – other expenses | $75 ($106) | $36 ($54) | |
| Year 1 Cost of CHOICE by site | $782 ($330) | $621 ($227) | |
| Year 1 Cost of CHOICE by attendee | $47 ($17) | $40 ($12) | |
| GTO training and TA at the site | $333 ($100) | $367 ($176) | |
| Travel time to sites for TA | $302 ($125) | ||
| Mileage to sites for TA | $127 ($53) | ||
| GTO training non-site, non-travel preparation | $306 (--) | ||
| TA supervision - supervisor time | $247 (--) | ||
| TA supervision - TA time | $209 (--) | ||
| Year 1 cost of adding GTO to by site | $1523 ($239) | $367 ($176) | |
| Year 1 cost of adding GTO to by attendee | $96 ($35) | $23 ($11) | |
| Total year 1 costs by site | $1523 ($239) | $1149 ($483) | $621 ($227) |
| Total year 1 costs by attendee | $99 ($35) | $69 ($27) | $40 ($12) |
|
Year 2 | |||
| CHOICE training - BGC staff | $126 ($131) | ||
| CHOICE training - trainer | $77 ($104) | ||
| CHOICE implementation | $255 ($99) | $160 ($88) | |
| CHOICE implementation – miles | $2 ($5) | $1 ($2) | |
| CHOICE implementation – other expenses | $76 ($97) | $47 ($38) | |
| Year 2 Cost of CHOICE by site | $536 ($235) | $209 ($106) | |
| Year 2 Cost of CHOICE by attendee | $41 ($30) | $17 ($11) | |
| GTO training and TA at the site | $439 ($116) | $403 ($181) | |
| Travel time to sites for TA | $309 ($138) | ||
| Mileage to sites for TA | $132 ($65) | ||
| GTO training non-site, non-travel preparation | $458 (--) | ||
| TA supervision - supervisor time | $267 (--) | ||
| TA supervision - TA time | $209 (--) | ||
| Year 2 cost of adding GTO to by site | $1813 ($271) | $403 ($181) | |
| Year 2 cost of adding GTO by attendee | $140 ($82) | $30 ($18) | |
| Total year 2 costs by site | $1813 ($271) | $939 ($331) | $209 ($106) |
| Total year 2 costs by attendee | $140 ($82) | $71 ($44) | $17 ($11) |
| Total 2-year costs by site | $3336 ($345) | $2088 ($694) | $829 ($253) |
| Total 2-year costs by attendee | $110 ($30) | $67 ($24) | $27 ($6) |
BGC = Boys and Girls Club; CHOICE = evidence-based program to prevent substance misuse in youth; GTO = Getting to Outcomes implementation support; TA = GTO technical assistance
By definition, the CHOICE group did not have any costs for GTO either year. Over the 2-year study period the addition of GTO more than doubled the BGC cost per attendee of offering CHOICE from $27 (SD = $6) for CHOICE alone to $67 (SD = $24) for CHOICE+GTO. In addition, GTO staff costs (presumably covered by a future federal or state funding agency) were $3,336 (SD = $345) per site or $110 (SD = $30) per attendee. Total societal costs were then $829 (SD = $253) per site and $27 (SD = $6) per attendee for CHOICE alone and $5424 (SD = $863) per site and $177 (SD = $46) per attendee for CHOICE+GTO.
Table 4 shows results of sensitivity analyses examining the cost impact of travel distance and time, number of sites being trained, and adding a set of CHOICE sessions per year. TA mileage and travel time only affect GTO costs each year but have a substantial impact: doubling miles and travel time increased average GTO costs per site by roughly 25% and halving miles and travel time lowered costs by 12–14%. Increasing or decreasing the number of sites involved mainly affected CHOICE trainer costs (BGC costs for both groups in year 1 and only for the CHOICE+GTO group in year 2) and TA supervision costs (GTO costs both years). Finally, running CHOICE more than once per year substantially reduced the cost per site of adding GTO.
Table 4.
Sensitivity analysis, cost per site for each year, Mean (SD)
| Year 1 | Year 2 | |||||||
|---|---|---|---|---|---|---|---|---|
| CHOICE+GTO | CHOICE only | CHOICE+GTO | CHOICE only | |||||
| GTO costs | CHOICE costs | CHOICE costs | GTO costs | CHOICE costs | CHOICE costs | |||
| GTO staff | BGC staff | BGC staff | BGC staff | GTO staff | BGC staff | BGC staff | BGC staff | |
| Base case | $1523 ($239) | $367 ($176) | $782 ($330) | $621 ($227) | $1813 ($271) | $403 ($181) | $536 ($235) | $209 ($106) |
| Twice the miles | $1951 ($408) | $367 ($176) | $782 ($330) | $621 ($227) | $2253 ($463) | $403 ($181) | $536($235) | $209 ($106) |
| Half the miles | $1309 ($161) | $367 ($176) | $782 ($330) | $621 ($227) | $1593 ($182) | $403 ($181) | $536 ($235) | $209 ($106) |
| Twice the sites | $1295 ($239) | $367 ($176) | $744 ($333) | $585 ($223) | $1575 ($271) | $403 ($181) | $497 ($202) | $209 ($106) |
| Half the sites | $1978 ($239) | $367 ($176) | $859 ($325) | $692 ($237) | $2288 ($271) | $403 ($181) | $612 ($317) | $209 ($106) |
| Add 1 site | $1067 ($239) | $367 ($176) | $705 ($337) | $549 ($219) | $1337 ($271) | $403 ($181) | $459 ($178) | $209 ($106) |
| Add 1 more CHOICE set per year | $0 ($0) | $0 ($0) | $406 ($317) | $320 ($180) | $0 ($0) | $0 ($0) | $333 ($173) | $209 ($106) |
CHOICE = an evidence-based program to prevent substance misuse in youth; GTO = Getting to Outcomes implementation support system
Table 5 shows CHOICE implementation outcomes for Years 1 through 4. Fidelity and quality of delivery outcomes for Years 1 and 2 have been reported elsewhere (Chinman et al. 2018). As can be seen, even though implementation past Year 2 was not a condition of the study, more of the CHOICE+GTO group’s BGCs continued to implement the CHOICE program past Year 2, and more planned to implement CHOICE past Year 4.
Table 5.
Outcomes for years 1 through 4
| Outcome | Year 1 | Year 2 | Year 3 | Year 4 |
|---|---|---|---|---|
| CHOICE+GTO | ||||
| # Sites that implemented CHOICE | 14 (100%) | 14 (100%) | 13 (93%) | 6 (43%)1 |
| Year 5+ plans to implement CHOICE | 6=Yes; 2=No; 5=Don’t know | |||
| Attendees per session >10 | 13 | 13 | 10 | 2 |
| Attendees per session 5–10 | 1 | 1 | 3 | 4 |
| CHOICE | ||||
| # Sites that implemented CHOICE | 15 (100%) | 15 (100%) | 11 (73%) | 0 (0%) |
| Year 5+ plans to implement CHOICE | 2=Yes; 1=No; 10=Don’t know | |||
| Attendees per session >10 | 13 | 12 | 7 | 0 |
| Attendees per session 5–10 | 2 | 3 | 4 | 0 |
CHOICE = an evidence-based program to prevent substance misuse in youth; GTO = Getting to Outcomes implementation support system
Difference in number of sites continuing their offer of CHOICE in Year 4 was statistically significant at p = .003.
Discussion
Despite the availability of many substance misuse prevention EBPs for adolescents (Clearinghouse for Military Family Readiness 2018; Institute of Behavioral Science 2019), it is not clear how much these EBPs have affected overall use rates for the US. A recent literature review on EBP implementation in large samples of public schools from 1991 to 2013 found that few schools adopt EBPs and they tend to implement them poorly when they do (Chinman et al. 2019). This is often the result of unclear program component specification, and the need for local communities to tailor EBPs to their own context, which they often lack the capacity to do. State and federal agencies have tried to provide funding and support, but their current efforts (mostly information dissemination) have not had a big impact on substance abuse rates nationally. Information-only prevention programs of the 70s and 80s had to evolve into more intensive skill-building programs to truly become evidence-based. The same is true for implementation support strategies. PREP (Chinman et al. 2018) and other studies of GTO and other intensive implementation support strategies have demonstrated they can improve implementation and outcomes (Chinman et al. 2016a; Hawkins et al. 2009; Durlak and DuPre 2008; Spoth et al. 2007). However, it is important to understand whether the additional costs are worth the benefit.
To answer that question, we examined costs of implementing the CHOICE program with and without the addition of GTO services in a sample of 29 BGC sites in Southern California. In these settings, and as would be expected given our experimental design and the fact that GTO wasn’t expected to have an impact until year 2, the cost to the BGCs of one implementation of the five-session CHOICE program in year 1 was similar between groups: $40 to $47 per attendee. In year 2, however, the cost of CHOICE differed substantially and significantly between groups ($41 for the CHOICE+GTO group versus $17 per attendee for the CHOICE group). The higher costs in the CHOICE+GTO group were because 1) they spent more hours on implementation and 2) they were offered and received additional CHOICE training. As representatives of organizations without implementation support the CHOICE group was not offered more training. They could have obtained more training on their own but did not.
Adding GTO to CHOICE and the time needed for GTO TA also increased the cost per attendee to BGCs by $23 to $30 per year, and added GTO staff costs of $96 to $140 per attendee per year, with the larger increases for both costs occurring in year 2 when GTO quality improvement activities were initiated based on observations of the sites’ first-year program delivery. According to our sensitivity analysis the costs of CHOICE, and specifically the cost of CHOICE training, can vary by the number of sites trained, but not by much. The marginal cost of adding a site ($1067+$367+$705 or $2139 per site in year 1 for the CHOICE+GTO group; $549 per site in year 1 for the CHOICE group; $1337+$403+$459 or $2199 in year 2 for the CHOICE+GTO group; and $209 in year 2 for the CHOICE group, from Table 4) was not much lower than the average cost per site ($1523+$1149 or $2672 in year 1 for the CHOICE+GTO group; $621 in year 1 for the CHOICE group; $1813+$939 or $2752 in year 2 for the CHOICE+GTO group; and $209 in year 2 for the CHOICE group, from Table 3), so the cost reduction to BGCs for CHOICE from adding sites would be minimal. On the other hand, running CHOICE more than once per year should substantially lower the cost per attendee of adding GTO since additional training in CHOICE and GTO and additional TA were assumed to be unnecessary within a one-year period. Also, because GTO training and TA and any additional CHOICE training end after the second year, the cost of adding another set of CHOICE sessions in the third year should be similar. The distances traveled to participating sites also affected the non-BGC cost of GTO but are not likely modifiable.
Adding GTO to CHOICE has been shown to increase adherence to the curriculum and quality of classroom delivery in Year 2 over training in CHOICE alone (Chinman et al. 2018). This increased adherence and quality of delivery also aligns with the increased BGC staff hours spent implementing CHOICE seen in this cost analysis for the CHOICE+GTO group in the second year. Year 3 and 4 results also indicate that adding GTO for Years 1 and 2 may increase the tendency for BGC sites to continue to offer the CHOICE program past Year 2 without any continuing GTO support (and costs), which is an important outcome.
Are these differences in outcomes worth the extra cost of adding GTO? The CHOICE program has been found to reduce and delay alcohol and marijuana use (D’Amico et al. 2012; D’Amico and Edelen 2007). It has also been shown that each additional year delay in drinking reduces the likelihood of dependence by 14% (Grant and Dawson 1997), and that underage drinking predicts other negative consequences: unintentional injuries, motor vehicle crashes, physical fights, risky sexual behavior, mental health problems, and other violent and delinquent behaviors (Spoth et al. 2008; Berg et al. 2013; D’amico et al. 2016). Thus, there is a real benefit in delaying initiation of alcohol and marijuana use. Note that the CTC program only achieved its benefits after 4 years and that the CTC theory of change posits that it should take 2–5 years of community intervention to observe changes in risk factors and 5–10 years to observe changes in adolescent alcohol use behaviors (Hawkins et al. 2009). Therefore, the effects of CHOICE may only appear years into the future. The Washington State Institute for Public Policy estimates that preventing one adolescent who would otherwise initiate alcohol use in middle-school is worth about $30,200 in terms of reduced treatment, healthcare, criminal justice, and earnings (2016 USD; personal communication September 4, 2018).1 If this estimate is correct, the benefit of GTO and more sessions of CHOICE would more than offset its costs. Further, GTO could be used with any EBP. It is possible that its results could be stronger with a more potent EBP than CHOICE. Finally, it is possible that GTO may have a wider impact on the capacity of these organizations than just the CHOICE program. Staff trained to use GTO could apply those skills to other programs and in other organizations over the course of their careers. If true, GTO could be yielding additional monetary benefits beyond what was specifically studied here.
We know of one other cost analysis of the CHOICE program (Kilmer et al. 2011). This analysis also used micro-costing from a societal perspective but examined the implementation of CHOICE as an after-school program in middle schools. Comparing results of the PREP study to that of this after-school based program indicates that offering CHOICE in BGCs is less expensive. The average cost per unique attendee in the after-school based program in 2010 USD ranged from $121 to $305, and these estimates ignore CHOICE training costs. PREP was less expensive because the schools offered the program more often, spent more on advertising, and took account of displaced class time, which was not an issue for BGC members.
There are two cost studies of other implementation support programs: PROSPER (Crowley et al. 2012) and CTC (Kuklinski et al. 2012). Both focused on providing broader community support for the implementation of EBPs and did not specify the EBP to use. The PROSPER system linked stakeholders from the local Cooperative Extension offices with local schools to deliver school- and family-based programs to reduce substance use for 6th to 7th graders (Crowley et al. 2012). The University faculty provided training and TA through the extension services network. The CTC program built community coalitions to determine the types of programs their adolescents needed and then to deliver them faithfully (Kuklinski et al. 2012).
Because PROSPER and CTC offered implementation support other than GTO, we wouldn’t expect the costs to be exactly the same. However, comparison is still important. The PROSPER cost analysis showed costs of implementing their school-based EBPs (including training) were a bit lower than PREP, ranging from $9 to $27 per student in 2010 USD, and the average cost of implementation support per youth served was higher, ranging from $311 to $405.
The CTC study followed the same group of youth from 5th to 8th grade and did not present their cost estimates in terms of costs per adolescent. However, if we take the average of their first two years of EBP implementation costs and divide by the number of adolescents, we get an estimate of $29 per adolescent. Their average training, technical assistance, implementation monitoring, and coalition costs per community (implementation support) divided by the number of adolescents is $67, lower than the cost of GTO implementation support.
This cost analysis benefits from detailed data collection, including data from time, mileage and cost logs kept by each BGC site. However, there were some limitations. Two CHOICE BGC sites did not provide their cost logs in Year 1 and five others did not provide cost logs in Year 2. We used group averages for these missing BGC costs. This study presented costs for GTO and CHOICE when CHOICE was offered in BGCs, only conducted once a year, adolescents attended only one set of CHOICE sessions, and adolescents’ outcomes were only collected for the year in which they attended CHOICE. It is possible, and even likely, that the costs of GTO and CHOICE would be different if offered in a setting other than a BGC (or similar organization with regular youth after-school attendance)—e.g., more costs to recruit adolescents, non-zero opportunity costs for their and their parents’ time. It is also possible that the costs of GTO and CHOICE would be lower if CHOICE was conducted more often per year as shown in our sensitivity analysis. And it is possible that outcomes seen for CHOICE and CHOICE+GTO would have been better if adolescents were allowed to attend more than one set of sessions and if their outcomes were measured over a longer follow-up.
Conclusions
Adding GTO implementation support to CHOICE increased the cost per attendee to society over two years from $27 to $177. The long-term effects of the better and continued implementation of CHOICE found with GTO are unknown at this point but may be worth the additional costs. If so, GTO could be an attractive alternative to the current structure for US substance misuse prevention EBPs.
Acknowledgements.
The authors would like to acknowledge the contributions of Marylou Gilbert and Christian Lopez for their assistance with this project.
Funding. All the authors were funded by a grant from the National Institute on Alcohol Abuse and Alcoholism (NIAAA): Preparing to Run Effective Prevention (R01AA022353-01). NIAAA had no role in the study design; the collection, analysis, and interpretation of data; the writing of the report; and the decision to submit the manuscript for publication.
Footnotes
Publisher's Disclaimer: This Author Accepted Manuscript is a PDF file of a an unedited peer-reviewed manuscript that has been accepted for publication but has not been copyedited or corrected. The official version of record that is published in the journal is kept up to date and so may therefore differ from this version.
Conflicts of interest. The authors declare that they have no conflicts of interest.
Trial registration. This project is registered at ClinicalTrials.gov with number (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was registered May 12, 2014.
Ethical approval. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee (RAND Human Subjects Protection Committee FWA00003425) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent. Informed consent was obtained from all participants included in the study.
The Washington State Institute for Public Policy (WSIPP) notes that this estimate is not a formal or published finding. As such it has not undergone Monte-Carlo Analysis, nor typical WSIPP publishing checklists. Their benefit-cost methodology undergoes continuous improvement and this estimate may not match a future version of the estimate. The views, opinions, and findings expressed in reports citing this number are not endorsed by WSIPP and may not reflect the findings of WSIPP.
Contributor Information
Patricia M. Herman, RAND Corporation, 1776 Main Street, PO Box 2138, Santa Monica, CA 90403.
Matthew Chinman, RAND Corporation, 4570 Fifth Ave #600, Pittsburgh, PA 15213.
Jill Cannon, RAND Corporation, 1776 Main Street, PO Box 2138, Santa Monica, CA 90403.
Patricia Ebener, RAND Corporation, 1776 Main Street, PO Box 2138, Santa Monica, CA 90403.
Patrick S. Malone, Malone Quantitative, Durham, NC 27705.
Joie Acosta, RAND Corporation, 1200 S Hayes St, Arlington, VA 22202.
Elizabeth J. D’Amico, RAND Corporation, 1776 Main Street, PO Box 2138, Santa Monica, CA 90403.
References
- Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, & Cafri G (2010). Psychometric properties and US National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychological Assessment, 22(2), 356–365. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Barnett PG (2009). An improved set of standards for finding cost for cost-effectiveness analysis. Medical Care, 47(7_Supplement_1), S82–S88. [DOI] [PubMed] [Google Scholar]
- Berg N, Kiviruusu O, Karvonen S, Kestilä L, Lintonen T, Rahkonen O, et al. (2013). A 26-year follow-up study of heavy drinking trajectories from adolescence to mid-adulthood and adult disadvantage. Alcohol and Alcoholism, 48(4), 452–457. [DOI] [PubMed] [Google Scholar]
- Chinman M, Acosta J, Ebener P, Malone PS, & Slaughter ME (2016a). Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes®. Implementation Science, 11(1), 78. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chinman M, Acosta J, Hunter S, & Ebener P (2015). Getting To Outcomes®: Evidence of Empowerment Evaluation and Evaluation Capacity Building at Work In Fetterman DM, Kaftarian S, & Wandersman A (Eds.), Empowerment Evaluation: Knowledge and Tools for Self-assessment, Evaluation Capacity Building and Accountability (Second ed., pp. 316–336). Thousand Oaks, CA: Sage. [Google Scholar]
- Chinman M, Acosta JD, Ebener P, Sigel C, & Keith J (2016b). Getting To Outcomes: A guide for teen pregnancy prevention. Santa Monica, CA: RAND Corporation. [Google Scholar]
- Chinman M, Acosta JD, Ebener PA, Hunter SB, Imm PS, & Wandersman A (2019). Dissemination of Evidence-Based Prevention Interventions and Policies In Sloboda Z, Petras H, Robertson E, & Hingson R (Eds.), Advances in Prevention Science: Prevention of Substance Abuse (pp. 367–383). Heidelberg, Germany: Springer International Publishing. [Google Scholar]
- Chinman M, Ebener P, Malone PS, Cannon J, D’Amico EJ, & Acosta J (2018). Testing implementation support for evidence-based programs in community settings: A replication cluster-randomized trial of Getting To Outcomes®. Implementation Science, 13, Article number 131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Clearinghouse for Military Family Readiness (2018). https://militaryfamilies.psu.edu/. Accessed March 26 2019.
- Crowley DM, Jones DE, Greenberg MT, Feinberg ME, & Spoth RL (2012). Resource consumption of a diffusion model for prevention programs: The PROSPER delivery system. Journal of Adolescent Health, 50(3), 256–263. [DOI] [PMC free article] [PubMed] [Google Scholar]
- D’Amico EJ, & Edelen MO (2007). Pilot test of project choice: A voluntary afterschool intervention for middle school youth. Psychology of Addictive Behaviors, 21(4), 592. [DOI] [PubMed] [Google Scholar]
- D’amico EJ, Tucker JS, Miles JN, Ewing BA, Shih RA, & Pedersen ER (2016). Alcohol and marijuana use trajectories in a diverse longitudinal sample of adolescents: Examining use patterns from age 11 to 17 years. Addiction, 111(10), 1825–1835. [DOI] [PMC free article] [PubMed] [Google Scholar]
- D’Amico EJ, Tucker JS, Miles JN, Zhou AJ, Shih RA, & Green HD (2012). Preventing alcohol use with a voluntary after-school program for middle school students: Results from a cluster randomized controlled trial of CHOICE. Prevention Science, 13(4), 415–425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, & Stoddart GL (2005). Methods for the Economic Evaluation of Health Care Programmes (3rd ed.). Oxford: Oxford University Press. [Google Scholar]
- Durlak JA, & DuPre EP (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. [DOI] [PubMed] [Google Scholar]
- Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. (2003). A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science, 4(1), 1–14. [DOI] [PubMed] [Google Scholar]
- Frick KD (2009). Micro-costing quantity data collection methods. Medical Care, 47(7 Suppl 1), S76–S81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grant BF, & Dawson DA (1997). Age at onset of alcohol use and its association with DSM-IV alcohol abuse and dependence: results from the National Longitudinal Alcohol Epidemiologic Survey. Journal of Substance Abuse, 9, 103–110. [DOI] [PubMed] [Google Scholar]
- Hallfors D, & Godette D (2002). Will the principles of effectiveness’ improve prevention practice? Early findings from a diffusion study. Health Education Research, 17(4), 461–470. [DOI] [PubMed] [Google Scholar]
- Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, et al. (2009). Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: A test of Communities That Care. Archives of Pediatrics and Adolescent Medicine, 163(9), 789–798. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Institute of Behavioral Science (2019). Blueprints for Healthy Youth Development. https://www.blueprintsprograms.org/. Accessed March 26 2019.
- Kilmer B, Burgdorf JR, D’Amico EJ, Miles J, & Tucker J (2011). Multisite cost analysis of a school-based voluntary alcohol and drug prevention program. Journal of Studies on Alcohol and Drugs, 72(5), 823–831. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kuklinski MR, Briney JS, Hawkins JD, & Catalano RF (2012). Cost-benefit analysis of Communities That Care outcomes at eighth grade. Prevention Science, 13(2), 150–161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kumar R, O’Malley PM, Johnston LD, & Laetz VB (2013). Alcohol, tobacco, and other drug use prevention programs in US schools: a descriptive summary. Prevention Science, 14(6), 581–592. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Livet M, & Wandersman A (2005). Organizational functioning: facilitating effective interventions and increasing the odds of programming success In Fetterman DM, & Wandersman A (Eds.), Empowerment Evaluation in Practice (pp. 123–154). New York, NY: Guilford. [Google Scholar]
- Miller WR, & Rollnick S (2002). Motivational Interviewing: Preparing People for Change (2nd ed.). New York, NY: Guilford Press. [Google Scholar]
- Ringwalt C, Hanley S, Vincus AA, Ennett ST, Rohrbach LA, & Bowling JM (2008). The prevalence of effective substance use prevention curricula in the nation’s high schools. The Journal of Primary Prevention, 29(6), 479–488. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ringwalt C, Vincus AA, Hanley S, Ennett ST, Bowling JM, & Rohrbach LA (2009). The prevalence of evidence-based drug use prevention curricula in US middle schools in 2005. Prevention Science, 10(1), 33–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rollnick S, Miller WR, & Butler CC (2008). Motivational Interviewing in Health Care: Helping Patients Change Behavior. New York, NY: Guilford Press. [Google Scholar]
- Spoth R, Greenberg M, & Turrisi R (2008). Preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Pediatrics, 121(Supplement 4), S311–S336. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Spoth R, Redmond C, Shin C, Greenberg M, Clair S, & Feinberg M (2007). Substance-use outcomes at 18 months past baseline: The PROSPER community–university partnership trial. American Journal of Preventive Medicine, 32(5), 395–402. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wandersman A, & Florin P (2003). Community interventions and effective prevention. American Psychologist, 58(6–7), 441–448. [DOI] [PubMed] [Google Scholar]