Cost analysis of a randomized trial of Getting to Outcomes implementation support for a teen pregnancy prevention program offered in Boys and Girls Clubs in Alabama and Georgia

Abstract

Objective

Implementation support can improve outcomes of evidence-based programs (EBP) for adolescents, but with a cost. To assist in determining whether this cost is worthwhile this study estimated the cost of adding Getting To Outcomes© (GTO) implementation support to a teen pregnancy and sexually-transmitted infection prevention EBP called Making Proud Choices (MPC) in 32 Boys and Girls Clubs (BGCs) in Alabama and Georgia.

Methods

Enhancing Quality Interventions Promoting Healthy Sexuality was a 2-year, cluster-randomized controlled trial comparing MPC to MPC+GTO. We used micro-costing to estimate costs and captured MPC and GTO time from activity logs completed by GTO staff. Key resource use and cost components were compared between the randomized groups, years, and states (to capture different community site circumstances) using 2-sample t-tests.

Results

There were no significant differences between randomized groups in attendees per site, resource use, or costs for either year. However, there were significant differences between states. Adding GTO to MPC increased the societal costs per attendee from $67 to $144 (2015 US dollars) in Georgia and from $106 to $314 in Alabama. The higher Alabama cost was due to longer travel distances and to more BGC staff time spent on GTO in that state. GTO also improved adherence, classroom delivery, and condom-use intentions more in Alabama youth. Thus, Alabama’s GTO-related BGC staff time costs may be better estimates of effective GTO.

Conclusions

If teen childbearing costs taxpayers approximately $20,000 per teen birth, adding GTO to MPC would be worthwhile to society if it prevented one more teen birth per 140 attendees than MPC alone.

Background

A number of evidence-based programs (EBPs) to prevent teen pregnancy and sexually transmitted infections (STIs) have been identified (Goesling et al. 2014). However, typical youth-serving community-based organizations have difficulty implementing these programs with the fidelity needed to achieve the outcomes seen in the effectiveness trials (Kramer et al. 2005; Tibbits and Siahpush 2017). This gap in outcomes between research and practice (Wandersman and Florin 2003) results from several factors, including limited resources, EBP complexity, and a lack of staff capacity (knowledge, attitudes, and skills) required for successful implementation.

There are public health implications associated with this lack of strong EBP implementation. Although teen pregnancy and birth rates are declining (Hamilton and Mathews 2016), the US still has the highest rates among countries who capture these data (Sedgh et al. 2015). In 2015 there were 22 births per 1000 adolescent females ages 15–19 and this rate was substantially higher in teens who identified as non-Hispanic black (32), Hispanic (35) and American Indian or Alaska Native (26) than in those who identified as non-Hispanic white (16) (Hamilton and Mathews 2016). Sexually active teens, and their children, are also at high risk for poor outcomes such as worse health, school dropout, and poverty, which can perpetuate cycles of social, educational, and economic disadvantage (Jutte et al. 2010; Lee 2010; Hoffman and Maynard 2008). The cost to society of teen pregnancy was estimated in 2010 to be $9.4 billion per year from increased public assistance, public health, foster care, and criminal justice costs (Power to Decide 2013).

The Centers for Disease Control and Prevention named teen pregnancy as one public health issue where EBPs, with implementation support to ensure successful adoption and implementation, can help. Making Proud Choices (MPC) is a voluntary teen pregnancy/STI prevention EBP intended for African-American, Hispanic and White adolescents, ages 11–13. It uses social cognitive theory (Bandura 1992) and the theories of reasoned action (Fishbein and Ajzen 1975) and planned behavior (Ajzen 1991) to influence adolescents’ knowledge and beliefs about sex and contraception to reduce the frequency of sexual activities and to increase condom use (Jemmott III et al. 1998; Jemmott III et al. 1992). MPC consists of eight highly-scripted one-hour sessions designed to influence adolescents’ knowledge, beliefs, and intentions to increase condom use and reduce the frequency of sex (Jemmott III et al. 1998). MPC has been found effective in clinical trials (Johnson et al. 2003; Johnson et al. 2011; Mullen et al. 2002; Scher et al. 2006) and is one of the most commonly implemented EBPs for teen pregnancy/STI in the US (Chinman et al. 2016a).

Getting To Outcomes© (GTO) is a 10-step implementation support system that improves EBP implementation by strengthening the capacity (i.e., knowledge, attitudes and skills) needed to carry out practices needed to run any program well (Livet and Wandersman 2005), namely setting goals, planning, evaluating process and outcomes, and conducting quality improvement. GTO builds capacity through three types of support: (1) the GTO manual of text and tools adapted for teen pregnancy and STI prevention (Chinman et al. 2016b), (2) face-to-face training, and (3) ongoing, onsite technical assistance (TA). Vital to GTO’s capacity-building is engaging practitioners in active learning, setting expectations and providing the opportunity to conduct the GTO-specified key programming practices themselves. GTO has been applied to many different content domains including drug use prevention (Chinman et al. 2004), underage drinking prevention (Imm et al. 2007), and positive youth development (Fisher et al. 2006).

Enhancing Quality Interventions Promoting Healthy Sexuality (EQUIPS) was a cluster-randomized controlled study of MPC offered with and without two years of GTO implementation support to randomized groups of middle school youth in community-based organizations (Boys and Girls Clubs, BGCs) (Chinman et al. 2016a; Chinman et al. 2016b; Chinman et al. 2013).

A key component of GTO is to use year 1 data for quality improvement for year 2 implementation. Therefore, and as expected, there were no significant effects between randomized groups in year 1. However, in year 2 after GTO-stimulated quality improvement, the GTO group saw significantly improved implementation (adherence and classroom delivery) (Chinman et al. 2016a), and adolescent condom attitudes and use intentions (proximal outcomes) (Chinman et al. 2018a). Changes in sexual behaviors did not differ between groups in either year; however, this could be because baseline rates of these behaviors were low in the sample and changes were measured at 6 months (Chinman et al. 2018a).

Implementation support has been found to improve program fidelity in EQUIPS and in other studies of EBPs for youth (Chinman et al. 2016a; Chinman et al. 2018b; Fagan et al. 2008; Spoth et al. 2007). However, this support has a cost, which is in addition to the cost of implementing the EBP itself. Those who fund communities to carry out EBPs (e.g., the Office Adolescent Health, local foundations) would benefit from knowing whether this additional cost is worthwhile. Therefore, those costs must be quantified and compared to the benefits gained.

Few cost analyses of implementation support interventions have been published. Two that we know of were in support of youth substance misuse prevention programs: PROmoting School-Community-University Partnerships to Enhance Resilience (PROSPER) (Crowley et al. 2012) and Communities That Care (CTC) (Kuklinski et al. 2012). These studies both used forms of implementation support other than GTO. The PROSPER system linked shareholders from the local Cooperative Extension offices with schools to deliver school- and family-based programs to reduce substance abuse to 6^th and 7^th graders. The CTC program built community coalitions to first identify the types of programs needed and then to deliver these EBPs faithfully. The cost analyses of PROSPER (Crowley et al. 2012) left it to future policy makers to compare its costs with its benefits. However, the analysis of the CTC program compared its costs to the monetary value of two of its outcomes: reductions in cigarette smoking and in delinquency (Hawkins et al. 2009). This cost-benefit analysis of CTC reported a return over 4 years of $5.30 for every dollar spent on CTC (Kuklinski et al. 2012).

Adding GTO implementation support to MPC has already been shown to be effective (Chinman et al. 2016a; Chinman et al. 2016b; Chinman et al. 2013; Chinman et al. 2018a). This cost analysis documents the costs of MPC with and without GTO implementation support. Since the characteristics of BGCs in Georgia and Alabama varied widely in terms of numbers of youth per site, staff capabilities, and distance between BGCs, we calculated and compared costs between states to determine the sensitivity of cost estimates to these factors.

Methods

EQUIPS was a 2-year, cluster-randomized controlled trial comparing MPC to MPC+GTO in BGCs in Georgia and Alabama. The trial is described briefly below and more detail is available elsewhere on the trial (Chinman et al. 2016a; Chinman et al. 2016b; Chinman et al. 2013) and its outcomes (Chinman et al. 2018a; Chinman et al. 2016a). This study was approved by the RAND Human Subjects Protection Committee. All parents provided written informed consent and all youth provided assent to participate.

Participants were thirty-two BGC sites; 16 from metro Atlanta, Georgia, and another 16 from multiple locations in three small metro areas in Alabama (Montgomery, Huntsville, and Auburn-Opelika) each with a population of roughly 150,000 to 200,000. Eight sites in each state (16 sites in total) were randomized to MPC alone and 8 in each state (16 total) were randomized to MPC+GTO. BGC staff numbers and demographics (age, gender, race, education) were similar between states and groups (Chinman et al. 2018a). About two-thirds were female; they were equally split between ages 50–65 and 26–49 years; almost all had some college education or greater; and 81% were African-American and 19% were White. There were also no differences by group or state between staff survey responses at baseline regarding their organization’s support for EBPs (Aarons et al. 2009) and their individual capacity for quality prevention (Acosta et al. 2013) and attitudes toward EBPs (Aarons 2004).

Each BGC site was asked to implement MPC in two different groups of adolescents, once a year for two years staggered across a three-year period (2012–2014). Each site was asked to recruit at least 10 adolescents 11–14 years of age for the program each time. The sites sent information flyers to parents, approached parents at the site, and/or held special MPC information sessions at the site to attract attendees. Each BGC site received $3,000 a year to defray some costs of participating in the study.

Two half-time and Masters-level GTO TA providers (one based in Georgia and one based in Alabama) provided standard MPC training and manuals to all sites, and delivered the GTO manuals (Chinman et al. 2016b); provided face-to-face training on GTO; and provided onsite TA to the 8 MPC+GTO sites in each state. The GTO manual contained written guidance about how to complete GTO steps, with each step being a different set of implementation best practices important to successfully carrying out an EBP. GTO staff provided training and held bi-weekly one-on-one meetings with BGC staff on conducting GTO planning activities.

The study was designed to run over two years because the largest benefits of GTO come after the first year or first cycle of EBP implementation. In the first year of the study, both groups received training in MPC and then implemented the program. Therefore, no real difference in the costs or outcomes associated with MPC were expected in the first year. During this first year about a quarter of MPC curriculum activities across all sites were observed and rated on their fidelity by trained observers. However, only the MPC+GTO group were provided results of this observation along with TA by the GTO staff to conduct quality improvement activities on their program delivery. The MPC-only group implemented MPC in year 2 without this feedback.

Outcomes

Details on Year 1 and Year 2 outcomes for the MPC+GTO and MPC-only groups are published elsewhere (Chinman et al. 2018a; Chinman et al. 2016a). Briefly, the study assessed MPC fidelity, performance of key programming tasks targeted by GTO, and youth proximal and distal outcomes. Youth proximal outcomes were knowledge (e.g., HIV, Condoms) and attitudes (e.g., beliefs about prevention, hedonism, negotiation; self-efficacy) and intentions toward condom use and abstinence. These were assessed using the same 14 scales that Jemmott III et al. (1998) reported as important mediators of sexual behavior (see Table 1 in Chinman et al., 2018 for a listing of each scale). Youth distal outcomes included whether adolescents ever had sexual intercourse, frequency of intercourse, and days of unprotected sex when they had sex. MPC fidelity (curriculum adherence, quality of classroom delivery, dosage or attendance) and performance (staff ratings of how well they performed in eight implementation best practices targeted by GTO as key to program success—e.g., developing goals, planning) were assessed through staff interviews. Fidelity was assessed at all sites by observer ratings of 2–3 MPC sessions per site per year and attendance logs. Proximal outcomes and sexual behaviors were assessed via youth survey prior to and 6 months after MPC delivery.

Costing

Our cost estimates were made using the ingredients method (Crowley et al. 2018; Levin and Belfield 2015), also known as resource cost or micro-costing method (Barnett 2009; Frick 2009). Our estimates are presented in 2015 US dollars (USD) so that costs from three perspectives are possible: 1) that of the BGCs; 2) that of a future federal or state funding agency that would cover the cost of providing GTO; and 3) the societal perspective, which includes all costs no matter who pays and for this study, is the sum of the other two.

We captured MPC training time and time spent on GTO training and TA from activity logs completed by GTO staff. BGC staff time for training and TA was also captured from these logs because the BGC sites and staff members involved in each activity were recorded. The two GTO staff were supervised via weekly one-hour meetings with their supervisor for 45 weeks in each year. BGC staff time for MPC implementation was estimated as eight 1-hour sessions per year. Implementing MPC required a curriculum and teacher’s set of materials per site and a workbook per youth attendee. MPC training also utilized condom demonstration and contraception kits for each site, which were reused across years. MPC training and implementation costs constitute day of implementation costs and the additional costs of GTO constitute the intervention infrastructure costs (Crowley et al. 2018). Unit costs for time, mileage, over-night stays, and materials, and the sources for those cost estimates are shown in Table 1.

Table 1.

Unit costs used in the cost analysis, all in 2015 USD

Cost category	Unit Cost	Source
GTO technical assistance supervisor time per hour	$46.68	Estimated 2015 Employer Cost of Employee Compensation for a “Social and Community Service Managers” (11–9151)a
GTO technical assistance staff time per hour	$29.83	Estimated 2015 Employer Cost of Employee Compensation for a “Community and Social Service Specialists, All Other” (21–1099)b
BGC staff time per hour	$20.97	Estimated 2015 Employer Cost of Employee Compensation for a “Social and Human Services Assistant” (21–1093)c
Transportation cost per mile	$0.575	US General Services Administration 2015 rates for privately owned automobilesd
Lodging and per diem per overnight stay	$137.00	US General Services Administration 2015 rates for Huntsville, ALe
MPC materials needed at each site	$571.77	ETR website (MPC is an ETR product) for the curriculum and teacher setf,g
Condom demonstration (10) and contraception kits (1) for each site	$238.28	Total Accessg,h for the condom demonstration models and Planned Parenthoodg,i for the contraception kits
MPC workbooks for each student	$3.07	ETR website (MPC is an ETR product)f,g

GTO = Getting to Outcomes implementation support system; BGC = Boys and Girls Club; MPC = an evidence-based program to prevent teen pregnancy and sexually transmitted infections in youth.

^aFound at: https://www.bls.gov/oes/2015/may/oes119151.htm. Accessed 11/27/17.

^bFound at: https://www.bls.gov/oes/2015/may/oes211099.htm. Accessed 11/27/17.

^cFound at: https://www.bls.gov/oes/2015/may/oes211093.htm. Accessed 11/27/17.

^dFound at: https://www.gsa.gov/travel/plan-book/transportation-airfare-rates-pov-rates-etc/privately-owned-vehicle-pov-rates/pov-mileage-rates-archived. Accessed 11/27/17.

^eFound at: https://www.gsa.gov/travel/plan-book/per-diem-rates/per-diem-rates-lookup/?action=perdiems_report&state=AL&fiscal_year=2015&zip=&city=Huntsville. Accessed 5/23/19.

^fFound at: https://www.etr.org/ebi/programs/making-proud-choices/#calculator. Accessed 6/18/19.

^gThe 2019 prices available on the website were adjusted to 2015 prices based on Consumer Price Index: https://data.bls.gov. Accessed 6/19/19.

^hFound at: http://www.totalaccessgroup.com/wood_condom_demonstrator.html. Accessed 6/25/19.

ⁱFound at: https://www.plannedparenthood.org/files/2314/0612/7399/Contraception_Kit_Order_Form.pdf. Accessed 6/18/19.

Average cost per attendee was calculated using initial enrollment each year. We assumed no opportunity cost to participant adolescent time, no adolescent (or parent) transportation costs, and no facility costs as all participants were members of their BGCs and would have already spent time there during the MPC session periods. Research-related costs were excluded.

Analyses

Average resource use and costs across sites in each group and in each state were reported along with their standard deviations. Comparisons of key resource use and cost components were made between the randomized groups, between years, and between states utilizing 2-sample t tests assuming unequal variances. Reported p-values for each set of comparisons were adjusted following the Bonferroni-Holm method (Holm 1979; Chen et al. 2017). We report and discuss the costs per site and per attendee for each state separately to examine the sensitivity of cost estimates to different community site circumstances (e.g., travel distances, differences in the number of youth attendees). We also reanalyzed and report the implementation (Chinman et al. 2016a) and proximal youth outcomes (Chinman et al. 2018a) found for MPC+GTO by state to allow a comparison of resource use to outcomes achieved. Outcomes by state were estimated using the same methods used in the original articles (Chinman et al. 2018a; Chinman et al. 2016a).

Results

Numbers of youth that attended MPC by site for the two randomized groups were very similar for both years. In the first year the MPC+GTO group (16 sites) averaged 15.4 (SD:4.0) attendees per site and the MPC-only group (16 sites) averaged 14.7 (SD:5.2) attendees (t₂₈=0.46, p = .650). In the second year there were 13.9 (SD:5.6) attendees for MPC+GTO and 13.1 (SD:4.7) for MPC alone (t₂₉=0.40, p=.694). However, the Georgia sites enrolled significantly more attendees than the Alabama sites in each year (Table 2). Gender of enrolled youth was similar between states and groups (just over half were female), but states differed significantly on attendees’ race and school grade. Significantly more identified as African-American in Georgia (91.5%) than in Alabama (87.1%), χ²=10.5, p=.001; and significantly more youth in Alabama than in Georgia were in grades 4–6 versus grades 7–12 (62.1% versus 51.6%, respectively), χ²=23.2, p<.001.

Table 2.

Resources used by group and year for the sites in each state

	Alabama sites			Georgia sites
	MPC+GTO (n=8 sites)		MPC (n=8 sites)	MPC+GTO (n=8 sites)		MPC (n=8 sites)
Year 1 Resource use – hours	GTO Staff	BGC	BGC	GTO Staff	BGC	BGC
MPC training - BGC staff timea		26.8 (7.1)	23.4 (6.7)		33.6 (20.2)	26.1 (21.0)
MPC training – trainer time		4.1 (1.7)	3.3 (1.1)		4.2 (1.3)	2.7 (1.3)
MPC trainer travel time		1.0 (0.4)	0.9 (0.5)		1.0 (1.0)	0.4 (0.2)
MPC implementation timeb		8.0 (--)	8.0 (--)		8.0 (--)	8.0 (--)
Hours spent on GTO training & TA at sitec	15.4 (6.6)	30.6 (15.3)		11.7 (5.3)	29.2 (7.0)
Travel time to sites for TA	10.5 (4.9)	2.0 (3.2)		4.3 (2.6)	2.0 (1.2)
TA supervision - supervisor timeb	1.4 (--)			1.4 (--)
TA supervision - TA timeb	2.8 (--)			2.8 (--)
Year 1 Resource use – MPC workbooks for attendees, miles and hotel nights for MPC trainer and TA
MPC workbooks (1 per attendee)d		14.0 (3.7)	11.4 (5.0)		16.9 (4.0)	18.0 (2.8)
MPC trainer miles		57.4 (22.3)	52.7 (30.2)		25.9 (23.0)	11.2 (7.3)
MPC trainer hotel nights		0.2 (0.2)	0.3 (0.2)		0.0 (0.0)	0.0 (0.0)
GTO and BGC staff TA miles	626.9 (304.1)	77.3 (86.5)		114.5 (97.5)	47.2 (22.1)
GTO TA hotel nights	0.1 (0.1)			0.0 (0.0)
Year 2 Resource use – hourse
MPC training - BGC staff timef		6.4 (3.2)	5.7 (1.0)		13.8 (2.7)	13.5 (2.8)
MPC training – trainer time		1.2 (0.7)	1.0 (0.3)		1.5 (0.7)	1.4 (0.7)
MPC trainer travel time		0.9 (0.5)	0.8 (0.4)		0.3 (0.3)	0.2 (0.0)
MPC implementation timeb		8.0 (--)	8.0 (--)		8.0 (--)	8.0 (--)
Hours spent on GTO training & TA at siteg	18.9 (6.4)	47.4 (6.4)		9.9 (6.3)	20.5 (11.9)
Travel time to sites for TA	11.3 (4.8)	6.7 (6.4)		4.0 (2.5)	0.8 (0.6)
TA supervision - supervisor timeb	1.4 (--)			1.4 (--)
TA supervision - TA timeb	2.8 (--)			2.8 (--)
Year 2 Resource use – MPC workbooks for attendees, miles and hotel nights for MPC trainer and TAb
MPC workbooks (1 per attendee)h		9.8 (4.2)	10.7 (5.3)		18.0 (3.4)	15.3 (3.0)
MPC trainer miles		50.5 (26.9)	41.5 (23.0)		8.6 (4.9)	7.1 (1.5)
MPC trainer hotel nights		0.0 (0.0)	0.0 (0.0)		0.0 (0.0)	0.0 (0.0)
GTO and BGC staff TA miles	669.9 (301.8)	304.8 (126.5)		102.4 (88.0)	19.9 (18.4)
GTO TA hotel nights	0.1 (0.1)			0.0 (0.0)

Note: Table values are means and standard deviations. BGC = Boys and Girls Club; MPC = Making Proud Choices an evidence-based program to prevent teen pregnancy and sexually transmitted infections in youth; GTO = Getting to Outcomes implementation support system; TA = GTO technical assistance.

^aIn year 1 across groups average hours of BGC staff time for MPC training for Alabama (M:25.1, SD:6.9) was not significantly different than for Georgia (M:29.8, SD:20.3), t₁₈=0.89, p=0.774.

^bThe use of these resources was not tracked by site. We estimated MPC implementation time based on 8 sessions of 1 hour each, and we averaged total TA supervision time across sites.

^cIn year 1 the Alabama MPC+GTO group had higher GTO staff and BGC hours than the Georgia group, but this difference was not significant (GTO staff t₁₃=1.24, p=0.705; BGC t₁₀=0.23, p=0.823).

^dIn year 1 across groups the Georgia sites had more attendees per site (M:17.4, SD:3.4) than the Alabama sites (M:12.7, SD:4.4), t₂₈=3.41, p=.010.

^eOne Alabama control site dropped out in Year 2 so the MPC group averages are across the remaining 7 sites.

^fIn year 2 across groups average hours of BGC staff time for Alabama (M:6.1, SD:2.4) was significantly lower than for Georgia (M:13.6, SD:2.7), t₂₉=8.30, p<.001.

^gIn year 2 the Alabama MPC+GtO group had higher GTO staff and BGC hours than the Georgia group, but only the higher BGC staff hours reached the threshold of statistical significance (GTO staff t₁₄=2.85, p=0.052; BGC t₁₁=5.61, p=0.001).

^hIn year 2 across groups the Georgia sites had more attendees per site (M:16.6, SD:3.4) than the Alabama sites (M:10.2, SD:4.6), t₂₆=4.38, p=0.001.

Table 2 shows resource use in terms of hours, mileage and overnight stays by state for each group for each year, and Table 3 shows key resource use by group and year for both states combined. The BGC staff hours spent in MPC training were higher in the MPC+GTO group than the MPC-only group in both years, but not significantly so (Table 3). As would be expected, the BGC staff hours spent in MPC training in the first year were significantly higher than the hours spent retraining in the second year for both the MPC+GTO and MPC-only groups (Table 3). Comparing states, the BGC staff hours spent in MPC training for the Georgia sites were higher than for the Alabama sites both years, but significantly higher in year 2 (Table 2). These higher year 2 Georgia MPC training hours may be related to the wide variation seen in their year 1 hours.

Table 3.

By site key resources used and cost totals by group and year across states, 2015 USD

	MPC+GTO (n=16 sites)		MPC (n=16 sites)
Year 1 Resource use	GTO Staff Costs	BGC Costs	BGC Costs
MPC training - BGC staff time (hours)a		30.2 (15.0)	24.7 (15.1)
Hours spent on GTO training & TA at site	13.5 (6.1)	29.9 (11.5)
GTO and BGC staff TA miles	370.7 (342.9)	62.3 (62.9)
Year 1 Costs
Year 1 Cost of MPC/siteb		$1850 ($315)	$1688 ($346)
Year 1 Cost of MPC/attendeeb		$128 ($43)	$140 ($86)
Year 1 cost of adding GTO to MPC/site	$996 ($418)	$705 ($272)
Year 1 cost of adding GTO to MPC/attendee	$70 ($38)	$47 ($18)
Total year 1 costs/site	$996 ($418)	$2554 ($458)	$1688 ($346)
Total year 1 costs/attendee	$70 ($38)	$175 ($41)	$140 ($86)
Year 2 Resource usec
MPC training - BGC staff time (hours)d,e		10.1 (4.8)	9.9 (4.5)
Hours spent on GTO training & TA at sitef	14.4 (7.7)	33.9 (16.7)
GTO and BGC staff TA milesg	386.1 (363.3)	162.3 (171.1)
Year 2 Costsc
Year 2 Cost of MPC/siteh,i		$483 ($121)	$478 ($98)
Year 2 Cost of MPC/attendeeh,i		$40 ($16)	$43 ($22)
Year 2 cost of adding GTO to MPC/site	$1037 ($510)	$883 ($495)
Year 2 cost of adding GTO to MPC/attendee	$107 ($99)	$91 ($81)
Total year 2 costs/site	$1037 ($510)	$1366 ($572)	$478 ($98)
Total year 2 costs/attendee	$107 ($99)	$130 ($79)	$43 ($22)
2-Year cost of MPC/sitej		$2333 ($331)	$2136 ($353)
2-Year cost of MPC/attendeej		$86 ($26)	$87 ($31)
2-Year cost of adding GTO to MPC/site	$2033 ($858)	$1588 ($615)
2-Year cost of adding GTO to MPC/attendee	$82 ($54)	$61 ($34)
Total 2-year costs/site	$2033 ($858)	$3921 ($797)	$2136 ($353)
Total 2-year costs/attendee	$82 ($54)	$149 ($61)	$87 ($31)

Note: Table values are means and standard deviations. BGC = Boys and Girls Club; MPC = Making Proud Choices an evidence-based program to prevent teen pregnancy and sexually transmitted infections in youth; GTO = Getting to Outcomes implementation support system; TA = GTO technical assistance

^aBGC staff hours for MPC training did not differ significantly between groups in year 1 (t₃₀=1.03, p=0.313).

^bThe total cost of MPC was not significantly different between groups on either a per site or per attendee basis in year 1 (per site t₃₀=1.39, p=0.176; per attendee t₂₂=0.50, p=0.621).

^cOne Alabama control site dropped out in Year 2 so the MPC group averages are across the remaining 15 sites.

^dBGC staff hours for MPC training did not differ significantly between groups in year 2 (t₂₉=0.15, p=0.885).

^eBCG staff hours for MPC training were significantly lower for each group in year 2 compared to year 1 (MPC+GTO t₁₈=5.10, p<.001; MPC t₁₈=3.76, p=.005).

^fGTO staff and BGC time for GTO training and TA were both higher in year 2 than year 1, but not significantly so (GTO staff t₂₈=0.35, p=0.731; BGC t₂₇=0.79, p=0.435).

^gMiles traveled by GTO and BGC staff were both higher in year 2, but only BGC miles were significantly higher in year 2 (GTO staff t₃₀=0.12, p=0.903; BGC t₁₉=2.19, p=0.164).

^hThe total cost of MPC was not significantly different between groups on either a per site or per attendee basis in year 2 (per site t₂₈=0.13, p=0.896; per attendee t₂₆=0.48, p=0.633).

ⁱThe total cost of MPC was significantly higher in year 1 than year 2 for both groups on either a per site or per attendee basis (MPC+GTO per site t₁₉=16.18, p<.001; per attendee t₁₉=7.66, p<.001; MPC per site t₁₈=13.44, p<.001; per attendee t₁₇=4.38, p<.001).

^jThe total cost of MPC across 2 years was not significantly different between groups on either a per site or per attendee basis (per site t₃₀=1.63, p=0.113; per attendee t₂₉=0.10, p=0.920).

Only the MPC+GTO group received GTO training and TA. BGC sites in Alabama had more of these GTO hours than sites in Georgia both in terms of GTO and BGC staff time. These differences were not statistically significant in the first year but were for BGC staff time in year 2 (Table 2).

Because the sites in Alabama were widely dispersed (the furthest sites were 190 miles from the GTO staff office), the MPC trainers and GTO staff in that state spent more time in travel, traveled more miles, and incurred lodging expenses when multi-day events occurred at distant sites. GTO staff in Alabama combined meetings and trainings with distant sites where possible, but still drove an average of 627 miles per site in year 1 and 670 miles in year 2. In contrast, the site furthest from the TA office in Georgia was 43 miles away making combined meetings much easier across those sites. GTO staff there drove an average of 114 miles per site in year 1 and 102 miles in year 2 with no overnight stays required.

Table 3 also shows total costs per site and per attendee by group and year for major program components, and Table 4 shows similar total costs by state and for each individual program component. All costs are calculated by multiplying the resource use from Tables 2 and 3 by the unit costs in Table 1. Similar to what was seen for MPC resource use, the costs of MPC (including training, materials and implementation) did not differ significantly between groups. However, because of the ~$850 per site required to buy MPC materials and demonstration kits in year 1 and because BGC staff did not need full MPC training in the second year, the costs for MPC training in the first year were significantly higher than in the second year for both groups.

Table 4.

Resource costs by component and year for the Alabama and Georgia sites, all in 2015 USD

	Alabama sites			Georgia sites
	MPC+GTO (n=8 sites)		MPC (n=8 sites)	MPC+GTO (n=8 sites)		MPC (n=8 sites)
Year 1	GTO Staff Costs	BGC Costs	BGC Costs	GTO Staff Costs	BGC Costs	BGC Costs
MPC training - BGC staff time		$562 ($148)	$490 ($141)		$704 ($423)	$547 ($440)
MPC training – trainer time		$122 ($51)	$99 ($32)		$124 ($39)	$81 ($40)
MPC trainer travel time		$29 ($11)	$27 ($14)		$29 ($31)	$11 ($6)
MPC trainer miles		$33 ($13)	$30 ($17)		$15 ($13)	$6 ($4)
MPC trainer hotel stays		$31 ($26)	$38 ($24)		$0 ($0)	$0 ($0)
MPC materials		$853 ($11)	$845 ($15)		$862 ($12)	$866 ($9)
MPC implementation time		$168 (--)	$168 (--)		$168 (--)	$168 (--)
Year 1 Cost of MPC/sitea		$1798 ($163)	$1697 ($127)		$1902 ($425)	$1678 ($489)
Year 1 Cost of MPC/attendeea		$140 ($51)	$186 ($101)		$117 ($33)	$95 ($29)
Time spent on GTO training & TA at site	$459 ($196)	$641 ($322)		$348 ($159)	$613 ($146)
Travel time to sites for TA	$314 ($146)	$41 ($67)		$128 ($79)	$42 ($25)
Mileage to sites for TA	$360 ($175)	$44 ($50)		$66 ($56)	$27 ($13)
Hotel stays for TA trips to sites	$17 ($18)			$0 ($0)
TA supervision - supervisor time	$66 (--)			$66 (--)
TA supervision - TA time	$84 (--)			$89 (--)
Year 1 cost of adding GTO to MPC/siteb	$1300 ($297)	$727 ($365)		$691 ($272)	$682 ($155)
Year 1 cost of adding GTO to MPC/attendeeb	$98 ($30)	$53 ($22)		$42 ($17)	$41 ($11)
Total year 1 costs/site	$1300 ($297)	$2525 ($448)	$1697 ($127)	$691 ($272)	$2584 ($545)	$1678 ($489)
Total year 1 costs/attendee	$98 ($30)	$192 ($61)	$186 ($101)	$42 ($17)	$158 ($43)	$95 ($29)
Year 2
MPC training - BGC staff time		$135 ($67)	$120 ($22)		$289 ($57)	$283 ($58)
MPC training – trainer time		$37 ($21)	$29 ($9)		$44 ($21)	$42 ($22)
MPC trainer travel time		$27 ($14)	$22 ($12)		$10 ($8)	$7 ($1)
MPC trainer miles		$29 ($15)	$24 ($13)		$5 ($3)	$4 ($1)
MPC trainer hotel stays		$0 ($0)	$0 ($0)		$0 ($0)	$0 ($0)
MPC materials		$30 ($13)	$33 ($16)		$55 ($11)	$47 ($9)
MPC implementation time		$168 (--)	$168 (--)		$168 (--)	$168 (--)
Year 2 Cost of MPC/sitec		$426 ($83)	$396 ($20)		$571 ($82)	$550 ($77)
Year 2 Cost of MPC/attendeec		$47 ($20)	$49 ($30)		$32 ($6)	$37 ($9)
Time spent on GTO training & TA at site	$564 ($191)	$993 ($135)		$294 ($188)	$429 ($250)
Travel time to sites for TA	$337 ($143)	$140 ($135)		$119 ($74)	$17 ($12)
Mileage to sites for TA	$385 ($174)	$175 ($73)		$59 ($51)	$11 ($11)
Hotel stays for TA trips to sites	$17 ($14)			$0 ($0)
TA supervision - supervisor time	$66 (--)			$66 (--)
TA supervision - TA time	$84 (--)			$84 (--)
Year 2 cost of adding GTO to MPC/sited	$1453 ($285)	$1309 ($205)		$621 ($283)	$457 ($261)
Year 2 cost of adding GTO to MPC/attendeed	$178 ($95)	$156 ($64)		$35 ($17)	$26 ($16)
Total year 2 costs/site	$1453 ($285)	$1734 ($245)	$396 ($20)	$621 ($283)	$1028 ($327)	$550 ($77)
Total year 2 costs/attendee	$178 ($95)	$202 ($82)	$49 ($30)	$35 ($17)	$58 ($21)	$37 ($9)
Total 2-year costs/site	$2753 ($559)	$4260 ($666)	$2108 ($133)	$1312 ($279)	$3612 ($426)	$2229 ($446)
Total 2-year costs/attendee	$125 ($43)	$189 ($47)	$106 ($34)	$38 ($9)	$106 ($19)	$67 ($12)

Note: Table values are means and standard deviations. BGC = Boys and Girls Club; MPC = Making Proud Choices an evidence-based program to prevent teen pregnancy and sexually transmitted infections in youth; GTO = Getting to Outcomes implementation support system; TA = GTO technical assistance

^aIn year 1 the total cost of MPC across groups per site in Georgia was higher (M: $1790, SD: $457) than in Alabama (M: $1747, SD: $151), but not significantly so (t₁₈=0.35, p=0.728). The opposite was true for MPC cost per attendee in year 1: Georgia - M: $106, SD: $32; Alabama - M: $163, SD: $81; t₁₆=2.62, p=0.080.

^bIn year 1 GTO staff costs were significantly higher in Alabama than Georgia on both a per site and per attendee basis (per site t₁₄=4.28, p=0.005; per attendee t₁₁=4.70, p=0.005). Also, the BGC costs of GTO training and TA were higher in Alabama than Georgia on both a per site and per attendee basis, but not significantly so (per site t₉=0.32, p=0.756; per attendee t₁₀=1.32, p=0.645).

^cIn year 2 the total cost of MPC across groups per site in Georgia was significantly higher (M: $561, SD: $77) than in Alabama (M: $396, SD: $60), t₂₈=6.65, p=<.001). The cost of MPC per attendee in year 2 was higher in Alabama (M: $48, SD: $25) than Georgia (M: $35, SD: $8), but not significantly so (t₁₇=1.96, p=0.264).

^dIn year 2 GTO staff costs were significantly higher in Alabama than Georgia on both a per site and per attendee basis (per site t₁₄=5.85, p<.001; per attendee t₇=4.18, p=0.024). Also, the BGC costs of GTO training and TA were significantly higher in Alabama than Georgia on both a per site and per attendee basis (per site t₉=7.27, p<.001; per attendee t₁₀=5.59, p<. 001).

Parallel to what was seen in Table 2, the per-site MPC training and implementation costs for the Georgia sites were higher than for the Alabama sites in both years, and significantly so in year 2 (Table 4). This was despite the higher travel costs experienced in Alabama. However, because of the higher number of attendees in Georgia, the opposite was true for per-attendee costs.

By definition, the MPC-only group did not have any costs for GTO either year. Consistent with what was seen in Table 2, the per-site and per-attendee costs to the BGCs for GTO training and TA were higher for Alabama than Georgia sites both years, but only significantly higher in year 2: almost three times as high on a per site basis and almost 6 times as high on a per attendee basis (Table 4). Alabama costs were also substantially and significantly higher for GTO staff time and expenses in both years: again, over twice as high on a per site basis and in year 2 about 5 times higher on a per attendee basis.

Table 5 shows the results for the implementation and proximal youth outcomes for MPC+GTO group compared to the MPC group for the study as a whole and by state. As can be seen, and as reported in previously, significant improvements in MPC implementation (adherence and classroom delivery)(Chinman et al. 2016a), and in the overall measure of condom attitudes and intentions and three of its eight subscales (Chinman et al. 2018a) were found to be significantly improved for the MPC+GTO group compared to the MPC-only group. However, analyses of these results by state indicate that most of the outcome gains in the MPC+GTO group were seen only in the Alabama youth.

Table 5.

Multivariate fixed effects for implementation and proximal outcomes in Year 2 of EQUIPS

	States combineda	Alabama	Georgia
Implementation
Adherence	11.81 (4.12, 33.80)	77.95 (3.21, >99)	6.15 (2.24, 16.89)
Classroom delivery – control	0.93 (0.29, 1.56)	1.48 (0.42, 2.54)	0.57 (−0.10, 1.25)
Classroom delivery – student interest	0.73 (0.17, 1.29)	0.91 (0.03, 1.79)	0.68 (−0.03, 1.38)
Classroom delivery - enthusiasm	0.59 (0.11, 1.06)	0.36 (−0.31, 1.02)	0.84 (0.15, 1.54)
Classroom delivery - objectives	1.27 (0.67, 1.87)	2.07 (1.31, 2.84)	0.71 (−0.01, 1.42)
Proximal outcomes
Abstinence attitudes/intentions	0.04 (−0.12, 0.19)	0.07 (−0.17, 0.31)	0.04 (−0.13, 0.21)
Knowledge of HIV/condoms	0.03 (−0.06, 0.12)	0.09 (−0.04, 0.23)	−0.02 (−0.12, 0.09)
Condom attitudes/intentions	0.23 (0.02, 0.44)	0.37 (0.07, 0.68)	0.10 (−0.13, 0.33)
Prevention beliefs	0.35 (−0.08, 0.78)	0.45 (−0.18, 1.08)	0.27 (−0.31, 0.85)
Hedonistic beliefs	0.02 (−0.22, 0.27)	0.21 (−0.13, 0.56)	−0.14 (−0.45, 0.18)
Availability control	0.14 (−0.22, 0.50)	0.48 (−0.02, 0.98)	−0.15 (−0.61, 0.31)
Impulse control	0.29 (0.02, 0.57)	0.26 (−0.14, 0.67)	0.32 (−0.05, 0.70)
Negotiation beliefs	0.09 (−0.26, 0.44)	0.08 (−0.42, 0.58)	0.10 (−0.36, 0.56)
Technical skills beliefs	0.24 (−0.13, 0.61)	0.15 (−0.38, 0.69)	0.31 (−0.18, 0.81)
Self-efficacy to use condoms	0.48 (0.04, 0.92)	0.79 (0.16, 1.42)	0.23 (−0.36, 0.81)
Intentions to use condoms	0.29 (0.04, 0.55)	0.44 (0.07, 0.81)	0.17 (−0.17, 0.51)

Note: k = 31 sites. Tabled values for adherence are odds ratios with 95% confidence intervals (N = 278 ratings). Tabled values for classroom delivery are mean differences between conditions (and 95% confidence limits) on original metrics (1–7); N = 53 ratings. Tabled values for proximal outcomes are differences and confidence limits at immediate youth post-test on original metrics: 1–5 for abstinence and condom attitudes/intentions and 0–1 (proportion correct) for knowledge. N (youth) = 750 Bold type indicates statistically significant difference, p < .05.

^aSource: For Implementation: Table 4 of Chinman et al, 2016. For Proximal outcomes: Tables 3 and 4 of Chinman et al, 2018.

Discussion

Implementation support has been shown effective in studies of many EBPs for youth (Chinman et al. 2016a; Chinman et al. 2018b; Fagan et al. 2008; Spoth et al. 2007). However, the addition of this support increases program costs over the costs of EBP implementation alone. Policy makers need information on these additional costs to make informed decisions as to whether these costs are worth the improvement in outcomes. This study shows that the total 2-year costs of adding GTO to a teen pregnancy and STI prevention EBP called Making Proud Choices (MPC) at BGCs in Alabama and in Georgia increased costs to the BGCs by an average of almost $1600 per site and $61 per attendee (i.e., by almost 70%) over offering MPC alone. In addition, adding GTO would also require funds from a federal or state funding agency of just over $2000 per site and $82 per attendee to cover GTO staff costs. Total costs to society were then about $3600 per site, $143 per attendee and $57,932 overall. The only cost that was variable by number of attendees was the $3.07 cost of the MPC manual given to each attendee, making it the marginal cost of adding one more attendee to either group.

This cost analysis found that the resource use and costs of providing MPC (training and implementation) did not differ significantly in either year between those randomized to MPC+GTO versus MPC alone. Therefore, any differences in outcomes seen across these groups in the EQUIPS study were likely because of the addition of GTO and not because of different average levels of effort in providing the MPC sessions.

In the first year the extra costs to the BGC of adding GTO were remarkably similar between states. However, in year 2 the per-site BGC costs of adding GTO were more than twice as high in Alabama as they were in Georgia. GTO staff costs were also twice as high in Alabama than Georgia in both years, but travel costs made up most of this difference in both years: 79% of the difference in year 1 and 65% in year 2. However, travel costs made little difference in BGC costs of GTO in year 1 and made up only one-third of the difference in year 2. The substantial difference between states in the year 2 BGC costs of GTO was that BGC staff in Alabama spent significantly more time in GTO training and engaging with GTO TA, where a good deal of GTO-based planning occurs. If BGC effort involved in GTO made a difference to outcomes we would expect better outcomes in Alabama.

Our analyses of year 2 outcomes by state indicate that the Alabama sites experienced the largest improvement in outcomes. Therefore, it is possible that more BGC staff time in GTO training and engaging with GTO TA contributed to Alabama’s better outcomes—i.e., this could be a dose-response relationship, one indication of causality (Hill 1965). The Alabama MPC+GTO sites not only had better proximal outcomes than the Georgia sites, they also had better implementation both in terms of adherence and classroom delivery; additional support for GTO being responsible. However, it is also possible that Alabama youth were simply more open to MPC for some other reason, which may have to do with their coming from more rural areas or earlier grades in school and/or being less likely to identify as African-American (e.g., African-American teens have a higher birth rate than Whites (Sedgh et al. 2015; National Center for Health Statistics 2019)).

The Power to Decide (formerly The National Campaign to Prevent Teen and Unplanned Pregnancy) has estimated the average annual cost to taxpayers of teen childbearing as $1682 for every year from birth to 15 years of age (Power to Decide 2013). Discounting future costs at 3% results in a present value cost to taxpayers over the 15 years of about $20,000. Adding GTO costs society (BGCs plus the GTO funders) an average over 2 years of $143 ($82 + $61) per attendee across states more than MPC alone. If we only count the benefits of fewer teen births, adding GTO to MPC would be worthwhile to society if it prevented one more teen birth per 140 attendees ($20,000/$143) than MPC alone. This impact of GTO is distinctly possible given that attitudes and intentions toward condoms (the outcome found to favor the MPC+GTO group in EQUIPS) are a predictor of actual condom use according to a meta-analysis of 28 prospective studies of condom use (Sheeran and Orbell 1998). Condom use plays an important role in reducing unwanted pregnancies in adolescents according to a meta-analysis of 41 randomized trials of about 95,000 adolescents (Oringanje et al. 2016).

This study used actual trainer time and expenses to estimate the MPC trainer costs. Those who want to offer MPC can also get training from program developers. Their cost for a two-day training for BGC staff to offer MPC is a flat fee of $975 per trainee (personal communication from MPC vendor) and does not include training materials. Our actual MPC trainer time and expenses were much lower. They totaled between $227 and $308 per site over 2 years depending on the state. Note that this is a cost of MPC and would not affect the cost of adding GTO.

The cost of implementation support in EQUIPS was between the cost estimates of two other programs: PROSPER (Crowley et al. 2012) and CTC (Kuklinski et al. 2012). The PROSPER system cost of implementation support per attendee ranged from $311 to $405 (2010 USD), higher than EQUIPS ($77 per attendee in Georgia and $208 in Alabama). The CTC program average training, TA, implementation monitoring, and coalition costs of $67 (2004 USD) per adolescent were lower than seen in EQUIPS.

This cost analysis benefits from detailed data collection by the GTO staff regarding time spent in MPC and GTO training and in TA with BGC site. However, there were limitations. We did not have the BGC site’s actual MPC implementation time and expenses and may have underestimated these costs by assuming MPC implementation required only 8 hours to cover 8 one-hour sessions. This study presented costs for GTO and MPC when MPC was only conducted once a year, youth attended only one set of MPC sessions, and their outcomes were collected at 6 months for the year in which they attended MPC. It is likely that the costs per site and attendee of GTO and MPC would be lower if MPC was conducted more than once per year, and it is possible that outcomes seen for MPC and MPC+GTO would have been better if youth attended more than one set of sessions and/or if their outcomes were measured over a longer follow-up. Also, because we only had two years of data, our cost estimates do not likely represent those of steady state program operation.

Conclusions

The cost of adding GTO implementation support to MPC offered in BGCs differed substantially between community organizations in two states with different distances between sites (one more urban and densely populated and one more rural and sparsely populated), different numbers of adolescents, and different training and support needs. Adding GTO to MPC increased the cost per attendee to society over two years from $67 to $144 in Georgia and from $106 to $314 in Alabama. Whether these additional costs are worthwhile depend on the value of the outcomes seen. The long-term effects of the better implementation of MPC with GTO are unknown at this point. However, one proximal outcome (condom attitudes and intentions) seems to have improved more in Alabama—where more time was spent by BGC staff in GTO training and engaging in GTO TA and where clubs served mostly teens in rural areas. Finding ways to improve teen pregnancy prevention in rural areas is critical since the teen birth rate is much higher in rural areas than in urban ones (Hamilton et al. 2016). In general, adding GTO to MPC would be worthwhile to society if it prevented one more teen birth per 140 attendees than MPC alone, a distinct possibility given condom intentions can predict condom use and condom use can prevent pregnancy. If this and other possible long-term effects are worth the additional cost of adding GTO, this form of implementation support could be an attractive addition to US EBPs to prevent teen pregnancies and sexually transmitted diseases, especially in rural areas like Alabama that are looking for evidence-based options to address their high teen birth rates.