Long-term Impacts and Benefit–cost Analysis of the Communities That Care Prevention System at Age 23, Twelve Years After Baseline

Abstract

This study estimated sustained impacts and long-term benefits and costs of the Communities That Care (CTC) prevention system, implemented and evaluated in a longitudinal cluster-randomized trial involving 24 communities in seven states. Analyses utilized reports from a longitudinal panel of 4,407 participants, followed since the study’s baseline in Grade 5, with most recent follow-up 12 years later at age 23. Impacts on lifetime abstinence from primary outcomes of substance use and antisocial behavior were estimated using generalized linear mixed Poisson regression analysis, adjusted for individual and community-level covariates. Possible cascading effects on 4-year college completion, major depressive disorder, and generalized anxiety disorder through age 23 were evaluated as secondary outcomes. CTC had a statistically significant global effect on primary outcomes and also on combined primary and secondary outcomes. Among primary outcomes, point estimates suggested absolute improvements in lifetime abstinence of 3.5% to 6.1% in the intervention arm, and relative improvements of 13% to 55%; 95% confidence intervals revealed some uncertainty in estimates. Among secondary outcomes, 4-year college completion was 1.9% greater among young adults from intervention communities, a 20% relative improvement. Mental health outcomes were approximately the same across trial arms. Although CTC had small sustained effects through age 23, benefit-cost analyses indicated CTC was reliably cost beneficial, with a net present value of $7,152 (95% credible interval: $1,253 to $15,268) per participant from primary impacts and $17,919 ($306 to $39,186) when secondary impacts were also included. It remained cost beneficial even when impacts were adjusted downward due to the involvement of CTC’s developer in the trial. Findings suggest that broader dissemination of CTC could improve public health and individual lives in the long term and generate positive net benefits to society.

The development and testing of interventions to promote positive youth development and prevent problems like substance use and delinquency has resulted in effective preventive interventions with potential to improve public health through large-scale implementation (Fagan, Bumbarger, et al., 2019; Gottfredson et al., 2015). Evidence of sustained impact, along with technical assistance to support high-quality implementation, can help make the case for broader dissemination, particularly when information on intervention costs and benefits is available to stimulate stakeholder investment. In this paper we report long-term impacts of Communities That Care (CTC), a community-based prevention system tested in the Community Youth Development Study (CYDS), a longitudinal randomized trial involving 24 communities in 7 states (Hawkins, Catalano, et al., 2008). Using data from a panel of youth followed from Grade 5 baseline, we assessed CTC’s impact on primary and secondary behavioral outcomes 12 years later when the panel were, on average, age 23. We also updated CTC’s expected net benefit. Last reported when participants were in Grade 12 (Kuklinski, Fagan, Hawkins, Briney, & Catalano, 2015), new estimates reflect sustained impacts and evolution in benefit-cost models used to evaluate CTC.

The CTC Prevention System and Efficacy Trial

CTC is a data-driven prevention system designed to assist communities in selecting and implementing evidence-based prevention programs aligned with their strategic prevention priorities for improving the health of youth community wide (Fagan, Hawkins, Catalano, & Farrington, 2019). Communities implementing CTC first complete a five-phase training and installation process over 12 to 18 months: Get Started, Get Organized, Develop a Community Profile, Create a Community Action Plan, and Implement and Evaluate. Support from key community leaders (e.g., mayor, police chief, school superintendent) who have local influence and also control community resources is a prerequisite and occurs in the Get Started phase. At this time communities also assess readiness for CTC, identify a local champion, and invite diverse stakeholders to join the effort. In phase 2, Get Organized, interested community stakeholders (teachers, parents, faith leaders, business representatives) form or work with an existing community coalition, learn about prevention science, develop a vision statement and timeline for installing CTC, and form workgroups. The major tasks of phase 3, Develop a Community Profile, are to examine data from surveys of local youth about risk and protective factors that predict health and behavior problems, identify elevated risk factors and depressed protective factors, and prioritize two to five for intervention. The coalition also determines which factors are already being addressed and where gaps exist. In phase 4, the coalition creates a Community Action Plan guiding local efforts to reduce prioritized risks and strengthen protection, including measurable outcomes to be achieved through implementation of evidence-based policies and programs. In phase 5, the coalition implements the selected programs and policies, monitors and evaluates implementation, adjusts implementation where needed, and compares results to the outcomes identified in the Community Action Plan. The process is expected to yield population-level improvements in risk and protection in 2 to 5 years and improvements in outcomes 4 to 7 years following initial installation.

After they completed training, the twelve CYDS communities randomly assigned to the intervention condition implemented 2–5 evidence-based prevention programs targeting youth in Grades 6 – 9 in each remaining year of the initial five-year efficacy trial. During this time, intervention communities also received some financial support and ongoing technical assistance (e.g., weekly phone calls, emails, and annual site visits) to support their CTC efforts. Control communities carried out “prevention as usual” during the course of the trial. Although they were provided data from the bi-annual CTC Youth Survey administered in their schools, they did not receive technical assistance in how to use the data or financial support for prevention.

A series of studies evaluating implementation outcomes in intervention and control communities has led to several important findings and highlights differences in science-based prevention activity in CTC and control communities throughout the efficacy trial. First, the CTC system was implemented with high fidelity over time. Intervention communities carried out over 90% of the key steps and procedures, or milestones and benchmarks, that indicate completion of CTC’s five phases, and they maintained high levels of system implementation throughout the efficacy trial (Fagan, Hanson, Hawkins, & Arthur, 2009; Quinby et al., 2008). Prevention programs were delivered with the recommended dosage and high rates of adherence to core components. Independent observers also rated the quality of delivery as high (Fagan, Arthur, Hanson, Briney, & Hawkins, 2011). Though they were at similarly low levels of adoption of a science-based approach to prevention at the start of the trial, CTC communities progressed further than control communities in adopting a science-based approach. Over time CTC communities were signficantly more likely than control communities to understand prevention science concepts, use risk and protective factor data in their prevention planning approach, select tested and effective interventions to address prevention goals, and implement, monitor, and adjust them when needed (Gloppen et al., 2016; Rhew, Brown, Hawkins, & Briney, 2013). CTC communities also implemented greater numbers of evidence-based prevention programs and reached more children and families compared to communities in the control arm (Fagan, Arthur, Hanson, Briney, & Hawkins, 2011; Fagan, Hanson, Briney, & Hawkins, 2012; Gloppen et al., 2016). Together these findings suggest that CTC provided a structure and process for data-driven selection and high-quality implementation of effective prevention programs aimed at local priorities. Control communities, who were not exposed to the CTC system and conducted their prevention activities “as usual” ultimately implemented fewer evidence-based programs and reached fewer youth and families with effective preventive interventions.

CTC’s theory of change suggests that implementation of the CTC system, including evidence-based prevention programs, will reduce risk, enhance protection, and lead to positive youth outcomes. In the CYDS, significant CTC-related improvements in risk and protection were observed among panel youth beginning in Grades 7 and 8 (Hawkins, Brown, et al., 2008; Hawkins et al., 2009). Lower lifetime incidence and recent (past-year, past month) prevalence of substance use and antisocial behavior were also first observed in Grade 8. These effects, along with impacts on lifetime incidence and recent prevalence of violence, continued through Grade 10 (Hawkins et al., 2012), though very few communities implemented prevention programs after Grade 9. Although effects on prevalence were not sustained beyond Grade 10, effects on substance use and antisocial behavior initiation endured through age 21 At this age, greater lifetime abstinence from antisocial behavior and gateway drug use were observed among participants from CTC compared to control communities (Hawkins, Oesterle, Brown, Abbott, & Catalano, 2014; Oesterle et al., 2015; Oesterle et al., 2018). A study of the mechanisms by which CTC led to positive youth outcomes suggests that community adoption of a science-based approach to prevention fully mediated CTC’s effect on youth outcomes (Brown et al., 2014).

Long-term Prevention Effects into Young Adulthood

The present paper reports intervention effect analyses through age 23, nearly a decade after participants from CTC communities were last exposed to evidence-based prevention programs implemented in the trial. Two years older than in the last published study (Oesterle et al., 2018), participants continued to move into more adult roles. For example, more were working full time, more were married or living with a partner, and more had biological children than at age 21. Age 23 also represented the first data collection wave in which all panel members had reached legal age. Along with the passing of time, normative developmental changes in the legality of substance use and involvement in antisocial behavior and violence raised questions about whether we would continue to find intervention effects on primary outcomes. Cascading or spillover effects on developmental outcomes not directly targeted by an intervention have been observed in other long-term studies (Kosterman et al., 2019; Lynne-Landsman, Bradshaw, & Ialongo, 2010). We wondered if CTC could lead to cascading effects on young adult outcomes that share risk factors with primary outcomes. CTC’s earlier and sustained effects on primary outcomes could also set participants on a healthier life course manifested in effects on secondary outcomes.

Sustained impacts on young adult substance use and antisocial behavior.

Although continued intervention differences in lifetime abstinence from substance use through age 23 were plausible, they might also have diminished because all participants could now legally use alcohol, cigarettes, and, in some states, marijuana. We still expected to find greater lifetime abstinence from illicit drugs among young adults from CTC communities because non-medically prescribed use of such drugs continued to be illegal.

Analyses of CTC’s long-term impact on lifetime participation in antisocial behavior through age 23 included a composite measure of delinquent, criminal, and violent behavior. Because involvement in these behaviors tends to decrease in young adulthood as individuals take on adult roles and responsibilities like work, marriage, and parenthood (Bersani & Doherty, 2018; Sweeten, Piquero, & Steinberg, 2013), we expected little additional initiation of antisocial behavior at age 23 in either experimental condition. We predicted that intervention differences in lifetime abstinence from antisocial behavior would, thus, be similar to what was observed previously.

Possible cascading impacts: educational attainment and mental health.

We hypothesized cascading impacts on associated outcomes that become salient in young adulthood, including educational attainment and also mental health concerns like depression and anxiety (Institute of Medicine and National Research Council, 2015) that share risk factors with substance use and delinquency (Jones et al., 2016). Although we did not find such evidence at age 21 (Oesterle, Hawkins, Fagan, Abbott, & Catalano, 2010), it may have been developmentally too early to observe effects on these secondary outcomes. For example, most college attenders had not been in college for 4 years when age 21 data were collected. Age 23 might be a better time to assess secondary impacts.

Benefit-cost Analysis of Interventions to Promote Positive Development

Evidence from program cost, benefit-cost, and cost-effective analyses is an important adjunct to evidence of program impact and has a long history of use by the federal government, in health and medicine, and, more recently, state-level policymaking (Enthoven, 2019; Neumann et al., 2018; White & VanLandingham, 2015). Benefit-cost analysis (BCA), which compares monetized impacts (i.e., avoided costs and/or increased revenues) to program investments, is generally better suited for prevention because multiple impacts can be included in the analysis even when measured differently. The time horizon over which benefits are expected to occur poses challenges for BCAs of youth prevention programs (Crowley et al., 2018; National Academies of Sciences, 2016). Investments may yield some short-term economic returns due, for example, to reductions in healthcare system utilization, grade retention, or delinquency, but the compounding effects of better health and fewer behavior problems can accrue for decades. For example, benefits from prevention-related improvements in workforce participation may not be seen until years after youth prevention program implementation but are experienced throughout life. Some prevention trials have followed participants over long periods of time (Kosterman et al., 2019; Reynolds, Ou, & Temple, 2018; Schweinhart, 2019), but most do not have extensive longitudinal follow-up.

Simulation approach to assessing long-term benefits and costs.

To overcome these problems, the present and prior BCAs of CTC (Kuklinski, Briney, Hawkins, & Catalano, 2012; Kuklinski et al., 2015) have used a simulation model developed by the Washington State Institute for Public Policy (WSIPP) (2019), which estimates long-term economic benefits of programs and policies in 11 different areas, including prevention and public health. Strengths include a consistent approach across areas, benefits by beneficiary (e.g., participants, taxpayers) and source (e.g., healthcare cost savings, earnings increases), procedures to guard against overestimating benefits, and a Monte Carlo feature to assess the impact of various sources of uncertainty (e.g., effect sizes, inflation rates). The model also is not static. Since our first CTC BCA (Kuklinski et al., 2012), it has expanded to include more policy areas and more monetizable outcomes. Parameters and computations for estimating benefits have been adjusted as new research becomes available, and program analyses have been updated periodically when new studies come to light. To assesses validity, WSIPP regularly compares model estimates with those produced independently and conducts external expert reviews.

Our first CTC BCA examined early effects on the incidence of antisocial behavior and alcohol use through Grade 8 (Kuklinski et al., 2012) and indicated CTC returned $5,250 per participant (2004 USD). Four years later when participants were in Grade 12, we were able to incorporate all significant primary impacts on the lifetime incidence of antisocial behavior and alcohol and cigarette use (Kuklinski et al., 2015) using WSIPP model developments. At $3,920 per participant (2011 USD), the estimated return was somewhat lower due to updates to benefit-cost model inputs and changes in our method for assessing CTC impact (e.g., hazard rates in Grade 8 versus cumulative initiation through Grade 12), but CTC remained cost beneficial.

CTC BCA at Age 23.

In the 5 years since our last BCA, the WSIPP model has continued to evolve. Changes that could impact CTC’s BCA include a stronger negative causal relationship between early antisocial behavior and high school graduation, the ability to monetize outcomes from 4-year college completion, a steeper effect size adjustment when the program developer is involved in the trial, and standard practice of including all measured impacts at a significance level of p < .20 or better (WSIPP, 2019). Changes to the model, additional follow-up data, and developmental changes in participants (e.g., now 5 years older with some completing college) suggested a revised CTC BCA was warranted.

Method

Data are from the Community Youth Development Study (CYDS, NCT01088542), a longitudinal community-randomized trial testing the efficacy of CTC (Brown et al., 2009; Hawkins, Catalano, et al., 2008) in 24 communities in seven states (Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington). All communities were small- to moderate-sized towns with their own governmental, educational, and law enforcement structures (population: 1,500 – 41,000, M = 15,633, SD 10,147). Prior to the trial (2003 – 2008), communities were matched in pairs within state on population, racial diversity, economic indicators, and crime rates, and assigned randomly to intervention or control condition. The study’s consort flow diagram is shown in online Figure S1. Training, support, and prevention program implementation are summarized in several prior publications (Fagan et al., 2009; Hawkins et al., 2014; Oesterle et al., 2018).

Sample

In fall 2004, all fifth graders in the 24 communities were invited to participate in the trial; recruitment continued into Grade 6 to increase participation. Of the 4,420 youth who received parental consent (76.1% in CTC, 76.7% in control communities) 4,407 completed baseline surveys in Grades 5 or 6 and remained in their communities for at least one semester. They comprise the study’s longitudinal panel. The 10th wave of data collection was completed in 2016 when panel members were, on average, age 23. Participants were blind to study assignment and were followed even if they subsequently left their original communities. The sample is 50% male, 20% Hispanic/Latino, 64% non-Hispanic White, 3% non-Hispanic African American, 5% non-Hispanic Native American, 1% non-Hispanic Asian American, and 6% other. At age 23, 3,850 (88.1%) of the active living sample (N = 4,368) completed data collection (CTC: 88.9%, control: 87.2%).

Measures

Data are from the Youth Developmental Survey (Arthur, Hawkins, Pollard, Catalano, & Baglioni Jr, 2002; Briney, Brown, Hawkins, & Arthur, 2012; Glaser, Van Horn, Arthur, Hawkins, & Catalano, 2005; Social Development Research Group, 2005–2016), which asked the panel about their perceptions of risk and protection in multiple domains (community, school, family, peer, partner, individual) and involvement in health-risking behavior at each data collection wave. After Grade 12, most respondents completed the survey online and the rest on paper (age 23: 88.2% online, 11.8% paper). Participants received incentive payments of $5 - $10 through Grade 12 and $25 - $50 thereafter when they completed the survey. The study protocol was approved by the University of Washington Institutional Review Board.

Primary outcomes: substance use and antisocial behavior.

Participants reported lifetime use of alcohol, tobacco, and marijuana (e.g., On how many occasions (if any) have you used marijuana in your lifetime?) at each wave. Lifetime use of other drugs was a composite of lifetime use of up to eight other substances (cocaine, LSD, stimulants, opioids, other prescription drugs, heroin, MDMA, inhalants). Reports across all 10 waves allowed us to examine cumulative initiation of each substance (coded 0 for no lifetime use at any wave, 1 for lifetime use in at least one wave). Cumulative initiation of antisocial behavior was based on prospective participant reports of past-year participation in any of seven behaviors (stealing, damaging property, shoplifting, attacking someone with intent to harm, carrying a handgun [other than while hunting or as part of their job], being arrested, and beating up someone so badly that they probably needed medical attention), It was coded 0 for no participation in any behavior at any wave, coded 1 for participation in at least one behavior in at least one wave.

Secondary outcomes: Educational attainment, major depressive disorder, generalized anxiety disorder.

Participants reported their highest level of educational attainment at ages 19, 21, and 23 (highest degree received), which allowed us to assess whether they had completed 4-year college by age 23 (coded 0 for not completed, 1 for completed). At these same ages, the PHQ-9 (Kroenke, Spitzer, & Williams, 2001) was used to assess depressive symptoms in the past 2 weeks. Participants met criteria for major depressive disorder (MDD) if they reported experiencing at least five of nine symptoms in the past 2 weeks for more than half the days or nearly every day, including feeling down, depressed, or hopeless and/or little interest or pleasure in doing things. At ages 21 and 23, the GAD-7 (Spitzer, Kroenke, Williams, & Löwe, 2006) assessed generalized anxiety disorder (GAD) symptoms in the past 2 weeks (e.g., feeling nervous, anxious, or on edge; not being able to stop or control worrying; 0 = not at all, 1 = several days, 2 = more than half the days, and 3 = nearly every day). Item scores were summed to yield a total GAD-7 score at each age. A total score greater than 10 indicated meeting criteria for GAD. These prospective reports allowed us to evaluate diagnosis of MDD by age 23 (coded 0 for no MDD at age 19, 21, or 23; 1 for MDD at least once at age 19, 21, or 23) and diagnosis of GAD by age 23 (coded 0 for no GAD at age 21 or 23; 1 for GAD at least once at ages 21 or 23).

Analysis Strategy

CTC impacts through age 23

We used multilevel regression analysis to evaluate CTC impacts on primary and secondary outcomes. Generalized linear mixed models with random intercepts that accounted for clustering of participants in 24 communities and 12 community pairs were estimated using HLM version 7 (Raudenbush, Bryk, Cheong, Congdon, & Du Toit, 2011). Poisson regression with a log link, binomial error distribution, and overdispersion was used to estimate adjusted risk ratios (ARRs) and their confidence intervals. All analyses included individual and community level covariates as detailed in the results section. Intent-to-treat analyses were conducted with 40 imputed datasets and results were averaged using Rubin’s rules to account for missing data that could bias estimates (Rubin, 1987). To evaluate the overall effect of CTC across outcomes, we calculated the global test statistic (Feng & Thompson, 2002). This test adjusts for the increased risk of Type I error when multiple statistical tests are performed. Because the sustained BCA analysis used Grade 12 as the starting point, we re-analyzed Grade 12 outcomes data using Poisson regression to match our approach at age 23. Findings reported in online Table S1 are slightly but not substantively different from prior logistic regression results (Hawkins et al., 2014; Kuklinski et al., 2015).

BCA

Consistent with best practices (Levin & McEwan, 2001; National Academies of Sciences, 2016), we conducted the CTC BCA from a societal perspective. All intervention resources were valued regardless of whether financial costs were incurred, and benefits estimates included a variety of stakeholder perspectives (participants, taxpayers, others). The time horizon for costs was the 5-year efficacy trial (2008 – 2013), but benefits were estimated over participant lifetimes. Benefits were for a general population and adjusted for the deadweight costs of taxation, or economic loss to society per dollar of taxes incurred or gained from taxes avoided. Costs and benefits were reported in 2017 USD, adjusted for inflation using the Implicit Price Deflator for Personal Consumption Expenditures, and discounted to intervention start at 3.5% per year.

CTC costs.

CTC costs included community coalition, intervention programs, and training, technical assistance, and implementation monitoring, as well as donations and in-kind contributions. Weighted average cost per youth was $602 in 2017 USD, with a range of ±35% to reflect differences observed across intervention communities during the trial.

CTC benefits.

Lifecycle benefits from primary outcomes were based on impacts at Grade 12 and age 23 so the trend over time could be incorporated into long-term estimates. Benefits from secondary outcomes were based only on age 23, as Grade 12 outcomes were not relevant (college completion) or not available (major depressive disorder, generalized anxiety disorder). Unlike our previous BCA, which included impacts judged statistically significant at a Type I error rate of .05 (Kuklinski et al., 2015), the current BCA included all impacts estimated at p < .20, consistent with WSIPP’s approach and best practices (National Academies of Sciences, 2016).

Benefits reflected long-term avoided costs or increased revenues expected to accrue over the lifetime of participants due to CTC. Table 1 describes the types of benefits expected for each impact, to whom they flow, and for how long (e.g., healthcare costs through the model’s age 100 maximum, earnings through age 65, criminal justice system costs through age 60). In general, for each year in which benefit was expected to occur, the amount of benefit was the product of the intervention effect size, the magnitude of the causal relationship between the impact and the source of the avoided cost or increased revenue, and the unit cost or revenue. For example, avoided direct costs from reductions in crime in each year through age 60 were the product of the likelihood of crime occurring in that year, expected criminal justice and victimization costs per crime, and the unit change in crime indicated by the CTC effect on antisocial behavior. Causal relationship magnitudes and standard errors were from WSIPP’s meta-analyses of relevant research literature (e.g., between substance use initiation and lifetime disorder). Unit values for avoided costs and increased revenues were from national or Washington State databases (e.g., National Survey on Drug Use and Health, Centers for Disease Control and Prevention). Total benefits were the sum of benefits across impacts, after adjusting for possible double counting arising from correlations among benefits from different outcomes; refer to Tables S2 and S3 for further information. Benefits by source and beneficiary were also generated. All reported values are after discounting to Grade 5 baseline (average age 11), to yield present value benefits per participant. For further detail, consult WSIPP’s technical documentation (WSIPP 2019).

Table 1.

Communities that Care: Benefits from primary and secondary impacts

	Beneficiary
Impact and benefit source a	Participant	Tax-payer	Other or indirect b
Primary impacts
Antisocial behavior c
Avoided criminal justice system costs: police/sheriff, court, county prosecutor, corrections facilities		✓
Avoided victimization costs
Tangible: Medical and mental health		✓	✓
Intangible: Pain, suffering, diminished quality of life			✓
Indirect benefits from increased high school graduation
Increased earnings and tax revenue	✓	✓	✓
Avoided healthcare costs due to better health	✓	✓	✓
Substance use (alcohol, tobacco, marijuana, illicit drugs) c
Indirect benefits from disordered substance use
Avoided healthcare costs, e.g., treatment, emergency department visits	✓	✓	✓
Reduced premature mortality (alcohol, cigarettes only)			✓
Avoided traffic and property loss costs (alcohol only)	✓		✓
Indirect benefits from increased high school graduation
Increased earnings and tax revenue	✓	✓	✓
Avoided healthcare costs due to better health	✓	✓	✓
Indirect benefits from reduced crime (alcohol, marijuana only)		✓	✓
Secondary impacts
College completion c
Increased earnings and tax revenue	✓	✓	✓
Avoided healthcare costs due to better health	✓	✓	✓
Major depressive disorder, generalized anxiety disorder c
Avoided healthcare costs, e.g., treatment, emergency department visits	✓	✓	✓
Increased earnings and tax revenue	✓	✓	✓
Reduced premature mortality			✓
Deadweight costs of taxation d			✓

^aBenefits are due to intervention-attributable avoided costs and increased revenues estimated to accrue over the lifetimes of participants in the CTC arm relative to the control arm. WSIPP’s technical documentation provides detailed information about computational routines, unit costs, causal relationship magnitudes, and other parameters affecting benefits estimates (Washington State Institute for Public Policy, 2019).

^bOthers benefit as crime victims from lower tangible and intangible crime victimization costs, as coworkers from educational externalities (i.e., better educated employees increase others’ productivity), as private insurers from lower healthcare costs, as household members when other members live longer.

^cTime horizon for estimating benefits: Criminal justice system and victimization: age 60. Healthcare, reduced mortality, property losses: participants’ lifetimes, modelled through age 100. Earnings and tax revenues: age 65.

^dDeadweight costs: Net welfare change from taxes that pay for a program and taxpayer benefits incurred.

Net present value and benefit-cost ratio.

CTC’s net present value per participant (NPV) was estimated by subtracting present value cost per participant from present value benefit per participant. CTC’s benefit-cost ratio (BCR) was the ratio of the two. An NPV greater than zero and a BCR greater than 1 indicated favorable cost-benefit, showing that CTC yielded more in benefits than it costs. Because of uncertainty in various inputs and parameters driving benefits estimates, we used Monte Carlo analysis to estimate the likelihood that CTC would produce a positive NPV in 1,000 simulations involving random draws from the distributions around various parameter estimates (e.g., CTC impacts and costs, discount rates, healthcare cost escalation). We calculated the proportion of simulations that yielded a positive NPV and estimated credible intervals around expected values. Results from analyses in which primary and secondary impacts were multiplied by a factor of .38 were also reported; this factor is the coefficient from WSIPP’s meta-regression of evidence about the impact of developer involvement in research trials on intervention impacts, as occurred in the CYDS.

Results

Sustained CTC Effects on Substance Use, Antisocial Behavior

At age 23, CTC’s global effect on primary outcomes was statistically significant (global t = −3.29, p < .01), as was its overall effect on primary and secondary outcomes combined (global t = −3.47, p < .01). All primary effects were in the expected direction (Table 2), indicating that sustained abstinence from substance use and antisocial behavior was greater among panel participants from CTC communities compared to participants from control communities through age 23. Differences in sustained abstinence from alcohol and illicit drugs were statistically significant at a Type I error rate of 0.05. As expected, lifetime abstinence from substance use and antisocial behavior had decreased by age 23 compared to Grade 12 (Table S1), but point estimates suggested abstinence was still 3.5% to 6.1% greater in absolute terms among CTC participants compared to controls, depending on the outcome. Relative improvements ranged from 12.9% to 56.5%. Table 2 includes 95% confidence intervals around point estimates. These intervals suggest some uncertainty in CTC impact. For example, CTC was expected to increase abstinence from alcohol by 3.9%, but improvements of 0.9% to 6.8% were also plausible.

Table 2.

Effects of Communities That Care on primary and secondary outcomes through age 23

	Adjusted prevalence a					Adjusted risk ratios and 95% CIs	Absolute risk differences and 95% CIs
	Control	CTC b	95% CI b	t-value	p
Primary impacts on sustained abstinence c
Alcohol use	6.2%	9.7%	(7.1%, 13.0%)	3.34	0.01
Cigarette use	27.9%	32.0%	(26.8%, 38.5%)	1.67	0.13
Marijuana use	28.6%	32.3%	(28.3%, 37.2%)	2.02	0.07
Illicit drug use	35.6%	41.7%	(37.4%. 46.6%)	3.22	0.01
Antisocial behavior	28.1%	33.0%	(28.1%, 38.8%)	2.27	0.05
Secondary impacts on incidence d
Four-year college completion	9.1%	11.0%	(9.0%, 13.3%)	2.15	0.06
Major depressive disorder	16.0%	15.7%	(13.1%, 18.7%)	−0.26	0.80
Generalized anxiety disorder	19.9%	19.7%	(16.5%, 23.3%)	−0.16	0.88

^aPopulation average results from generalized linear mixed Poisson regression analyses of those who had not initiated the behavior at baseline. Baseline non-initiators ranged from n = 3,450 for alcohol use to n = 4,383 for marijuana use. Regression models adjusted for individual (gender, age, race, Latino ethnicity, parental education; and baseline rebelliousness, attendance at religious services; and, for college completion analyses only, risk for academic failure) and community characteristics (baseline school district enrollment, percentage of students eligible for free or reduced-price school lunch).

^bCTC = Communities That Care. ARR = adjusted risk ratio. CI = confidence interval.

^cSustained abstinence: Percentages reflect the proportion of baseline non-initiators who never reported the behavior in any data collection wave between Grade 5 and age 23.

^dIncidence through age 23: Percentages reflect the proportion of the sample who (a) completed 4-year college (had earned a bachelor’s degree or advanced degree); (b) met criteria for major depressive disorder at age 19, 21, and/or 23; and (c) met criteria for generalized anxiety disorder at age 21 and/or age 23.

Among secondary outcomes, CTC improved college completion by 1.9%, a relative improvement of 20.9%. The predicted prevalence of major depressive disorder and generalized anxiety disorder through age 23 were about 16% and 20%, respectively, and were approximately the same among CTC and control participants. As with primary impacts, 95% confidence intervals indicate some uncertainty in CTC’s impacts on secondary outcomes.

Benefit-cost Analysis of Sustained CTC Effects

Table 3 summarizes the standardized mean difference effect sizes and standard errors used in the BCA. Negative effect sizes reflect lower lifetime involvement in substance use and antisocial behavior among CTC compared to control participants. The positive effect size on 4-year college completion indicates greater completion among CTC participants compared to controls. CTC was reliably cost beneficial (Table 4), producing a positive NPV at least 77% of the time, exceeding the WSIPP’s reliability standard of 75%. Analyses involving the trial’s primary outcomes indicated that CTC returned $7,754 in benefits per participant, $7,152 more per participant than it cost, and returned $12.88 per dollar invested. Approximately 20% of estimated benefits were expected to accrue to participants, 27% to taxpayers, and 53% to others. About half flowed from reductions in crime, 35% from improved earnings, and 16% from healthcare cost savings.

Table 3.

Communities That Care effect size inputs to benefit-cost analysis

	Adjusted prevalence a		Impacts as estimated in CTC trial		Adjusted impacts–due to involvement of developer in trial b
	Control	CTC	Effect size c	Standard error d	Effect size b	Standard error c
Primary impacts
Initiation through Grade 12
Alcohol use	81.4%	75.7%	−0.208	0.051	−0.079	0.051
Cigarette use	60.7%	55.7%	−0.125	0.039	−0.048	0.039
Marijuana use	55.1%	52.0%	−0.077	0.037	−0.029	0.037
Illicit drug use	52.7%	47.4%	−0.127	0.037	−0.048	0.037
Antisocial behavior	67.6%	61.6%	−0.158	0.044	−0.060	0.044
Initiation through age 23
Alcohol use	93.8%	90.3%	−0.290	0.079	−0.110	0.079
Cigarette use	72.1%	68.0%	−0.118	0.042	−0.045	0.042
Marijuana use	71.4%	67.7%	−0.107	0.040	−0.041	0.040
Illicit drug use	64.4%	58.3%	−0.156	0.038	−0.059	0.038
Antisocial behavior	71.9%	67.0%	−0.141	0.045	−0.053	0.045
Secondary impacts d, e
Four-year college completion through age 23	9.1%	11.0%	0.125	0.062	0.047	0.062

^aFor primary impacts, adjusted prevalence is estimated as 1 minus the proportion who abstained, as reported in Table 2 (age 23) and Table S1 (Grade 12). For the secondary impact, adjusted prevalence is as reported in Table 2.

^bEffect size = (ln(AdjPrev_CTC × (1-AdjPrev_Control)/((1-AdjPrev_CTC)×AdjPrev_Control)) / 1.65. AdjPrev = Adjusted Prevalence (Sánchez-Meca, Marín-Martínez, & Chacón-Moscoso, 2003).

^cStandard errors = square root of (0.367 × (1/I_s + 1/I_f + 1/C_s + 1/C_f)), where I_s, I_f, C_s, and C_f are the number of successes (e.g., no initiation of marijuana use by age 23, no MDD by age 23) and failures (e.g., initiation of marijuana use by age 23, MDD by age 23) in the intervention and control conditions (Washington State Institute for Public Policy, 2019).

^dWSIPP adjusts effect sizes by a factor of .38 for prevention trials in which the prevention program developer was involved, but keeps standard error estimates the same. Adjusted impacts = Effect size_{CYDS Tnal} × 0.38.

^eCTC’s impacts on the incidence Major Depressive Disorder and Generalized Anxiety Disorder through age 23 did not meet the threshold for being included in the BCA as p values reported in Table 2 were not less than .20.

Table 4.

Long-term benefits and costs of Communities That Care (CTC)

	Impacts as estimated in the CTC trial a				Adjusted impacts – due to developer involvement in trial a
	Benefits from primary impacts only b		Benefits from primary and secondary impacts b		Benefits from primary impacts only b		Benefits from primary and secondary impacts b
	2017 USD e	%	2017 USD	%	2017 USD	%	2017 USD	%
Costs c, d	$602		$605		$600		$603
(CrI) e	($434, $764)		($441, $772)		($437, $769)		($440, $772)
Benefits d	$7,754		$18,524		$2,700		$6,713
(CrI)	($1,988, $15,996)		($907, $39,794)		(−$2,169, $8,685)		(−$8,424, $23,877)
NPV d, e	$7,152		$17,919		$2,101		$6,110
(CrI)	($1,253, $15,268)		($306, $39,186)		(−$2,805, $8,088)		(−$8,917, $23,292)
BCR d, e	$12.88		$30.62		$4.50		$11.14
Investment risk f	100%		98%		79%		77%
Disaggregated benefits d
By beneficiary
Participant	$1,537	19.8%	$6,997	37.8%	$579	21.4%	$2,553	38.0%
Taxpayer	$2,098	27.1%	$4,841	26.1%	$774	28.7%	$1,811	27.0%
Other	$4,119	53.1%	$6,686	36.1%	$1,347	49.9%	$2,350	35.0%
By source
Earnings	$2,713	35.0%	$13,477	72.8%	$1,027	38.0%	$4,902	73.0%
Crime	$3,969	51.2%	$3,984	21.5%	$1,490	55.2%	$1,564	23.3%
Health care	$1,258	16.2%	$1,256	6.8%	$452	16.7%	$508	7.6%
Mortality	$97	1.3%	$96	0.5%	$35	1.3%	$37	0.5%
Property loss	$17	0.2%	$17	0.1%	$6	0.2%	$6	0.1%
Deadweight loss	−$299	−3.9%	−$305	−1.6%	−$309	−11.4%	−$304	−4.5%

^aAdjusted impacts were calculated by multiplying effect sizes from the CTC trial by a factor of .38.

^bPrimary outcomes: Lifetime initiation of alcohol use, cigarette use, marijuana use, other illicit drug use, and antisocial behavior. Secondary outcome: 4-year college completion. Major depressive disorder and generalized anxiety disorder did not meet the threshold for inclusion in the benefit-cost analysis.

^cCosts are from Kuklinski et al. (2015), converted from 2014 USD to 2017 USD using the implicit price deflator for personal consumption expenditures.

^dExpected values from 1,000 Monte Carlo simulations. Credible intervals represent the 25th lowest and 25th highest values from the Monte Carlo simulations.

^eUSD = United States dollars. CrI = credible interval. NPV = net present value. BCR = benefit-cost ratio.

^fInvestment risk: Percentages reflect the proportion of Monte Carlo simulations that produced a positive NPV.

When college completion was included in the BCA, the NPV rose significantly to $17,919, due largely to the substantial earnings benefits that followed from college completion and accrued primarily to participants. Investment risk remained low, with 98% of Monte Carlo simulations yielding a positive NPV. Even when trial effect sizes were multiplied by .38 due to developer involvement in the trial, CTC was still estimated to be reliably cost beneficial. Expected values were approximately one third of those reported for the trial effect sizes, but the distribution of benefits by beneficiary and source was the same as in the unadjusted impact analyses.

Discussion

Using data collected from CYDS participants through age 23, 12 years after baseline, this study provided further evidence of CTC’s sustained impact on health-risking behaviors and continued positive cost benefit. Young adults from CTC communities continued to report greater abstinence from alcohol, cigarette, marijuana, and other drug use, and antisocial behavior. Abstinence improved in relative terms by about 15% (though alcohol abstinence improved by 55%) and in absolute terms by 3.5% – 6.1%. This study did not provide strong evidence of cascading effects. CTC led to a 20% relative improvement in college completion, but the absolute gain was roughly 2%. With only 11% of CTC and 9% of control panel members having completed 4-year college, it will be important to re-evaluate impacts in a few years when more young adults are expected to have completed college, especially given postsecondary education’s strong overall impact on health and wellbeing throughout the life course. As at age 21, analyses not reported in this paper did not suggest CTC effects on recent substance use and antisocial behavior at age 23.

Although effect sizes at age 23 were small and estimated with some uncertainty, analyses continued to show that implementing the CTC system and evidence-based prevention programs in middle and early high school reduced lifetime involvement in health-risking behaviors more than a decade after Grade 5 baseline. Greater abstinence among baseline non-initiators in early young adulthood when substance use normally peaks is important, as substance use initiation among young adults from CTC communities might have caught up with peers from control communities. Findings suggest that CTC implementation put community youth on healthier life trajectories that lasted into young adulthood, even though evidence-based prevention programs in the trial focused on the late elementary and middle school ages, with few study youths exposed to prevention after Grade 9. Had communities offered programs to the sample in high school, continued prevention exposure may have boosted the impact of CTC on these youths’ health-risking behaviors, including on recent substance use and antisocial behavior in the past month or past year (Hawkins et al., 2012). Notably, findings are consistent with those from a quasi-experimental evaluation of statewide implementation of CTC in Pennsylvania showing sustained effects over sixteen years on substance use, delinquency, and also depression in school districts where CTC was implemented, compared to non-CTC districts (Chilenski, Frank, Summers, & Lew, 2019). Economic benefits expected to accrue over participants’ lifetimes because of CTC are one way of illustrating the significance of sustained albeit modest improvements in young adults’ lifetime involvement in substance use and antisocial behavior. The net present value and return per dollar invested were $7,754 per participant and $12.88 per dollar invested, respectively, from primary impacts and were far higher when CTC’s effect on educational attainment was monetized. Returns were expected to be reliably positive even when estimated using sharply reduced effect sizes due to the developer’s involvement in the trial.

Compared to our prior analysis at Grade 12, the age 23 BCA using WSIPP’s latest model indicated roughly $3,000 more in benefits per participant from primary impacts (Table S4). Gains were distributed across all beneficiaries ($500 to participants, $1,000 to taxpayers, $1,500 to others) and reflected approximately $800 more in earnings benefits, $1,000 more in avoided crime costs (largely to crime victims), and $1,200 more in avoided healthcare costs (largely due to impacts on illicit drug use). With a stronger developer involvement adjustment (.38 versus .50 in the Grade 12 analysis), adjusted benefits and NPV were largely unchanged since the prior report.

Limitations

CYDS trial communities are small and medium-sized rural towns. Findings may not generalize to urban cities and other states and regions not represented. Rural youth are, however, a large population who are more likely to lack access to addiction and mental health prevention and treatment services. Results suggest CTC can help small towns improve the behavioral health of young adults, including those who stay in the community and those who move away.

BCA results are expected values from a complex simulation model developed primarily for use in Washington State policymaking, not actual benefits. The model’s regular expert review, comparability with benefits produced by other studies, and inclusion of national studies and databases in computational routines and causal relationships lend confidence to conclusions reached. Though caution is in order when projecting lifecycle benefits and using results to guide public investments, the BCA model’s adjustments for possible double counting lead to conservative estimates. Tables available online (Tables S2 and S3) document the many benefits excluded from the BCA along with their rationale. Research that documents the covariance among substance use outcomes and among different categories of impact over time is needed to provide an empirical basis for excluding benefits. With respect to the steep adjustment due to the CTC developers’ involvement in the trial, strong training and technical assistance protocols should make it more likely that trial impacts will be replicated, reducing the need for such a strong discount.

The CYDS trial focused on evaluating CTC’s effects on youth substance use and delinquency as primary outcomes. Future research could address whether CTC is an effective and cost-beneficial approach for addressing other primary youth outcomes, like depression or suicide, or for community prevention efforts that extends beyond grade 9. Research aimed at optimizing the five-phase training process to reduce the time and costs of getting to high-quality CTC implementation could also facilitate greater uptake by communities and stronger public health impacts.

Conclusions

This long-term follow-up study adds to the evidence (Chilenski et al., 2019; Feinberg, Jones, Greenberg, Osgood, & Bontempo, 2010; Toumbourou, Rowland, Williams, Smith, & Patton, 2019) that CTC, which builds capacity in local stakeholder coalitions to develop and implement data-driven prevention plans addressing community-specific priorities with tested and effective programs, leads to better outcomes for youth into young adulthood, compared to communities who may implement some evidence-based programs but do not use CTC’s systemic and comprehensive approach. Broader dissemination of CTC, which has a well-developed program of technical assistance to support implementation in communities, regionally, or as a statewide scale-up system for evidence-based prevention programs, could improve public health and individual lives in the long term, with positive net benefits to society.