Sustained Effects of the Communities That Care System on Prevention Service System Transformation

Abstract

Objectives. We examined whether the Communities That Care (CTC) system sustained effects 1.5 years after study funding ended on prevention system constructs expected to be important for community-level reductions in drug use and antisocial behaviors among youths.

Methods. Data were from a community trial of 24 towns in the United States randomized to either the CTC intervention or control conditions. Participants were 928 community key leaders interviewed at 1 to 4 waves from 2001 to 2009. Intervention activities, including training and technical assistance, were conducted between 2003 and 2008 in the CTC communities.

Results. Leaders from CTC communities reported higher levels of adoption of a science-based approach to prevention and a higher percentage of funding desired for prevention activities in 2009 than did leaders in control communities. CTC communities showed a higher increase over time in community norms against adolescent drug use as well as adoption of a science-based approach compared with control communities.

Conclusions. These findings indicated that CTC implementation produced enduring transformation of important prevention system constructs in intervention communities, which might, in turn, produce long-term reductions in youth problem behaviors.

Despite evidence for effectiveness of preventive interventions to reduce the occurrence of drug use and delinquent behaviors among youths,^1,2 few communities have implemented such interventions as part of community-wide strategies. Approaches that address prevention of youth problem behaviors through community system-level changes have the potential for significant public health impact.^3–5 Consistent with this, the federal government identified systems thinking and evidence-based practices, research, and evaluation as 2 priorities for public health improvement for the next decade.⁶

A recent review identified that a common feature of successful strategies for reducing alcohol use or availability of alcohol to adolescents is reliance on local coalitions to develop and implement prevention plans.⁵ For example, Communities Mobilizing for Change on Alcohol (a program using coalition-based efforts to change community policies, practices, and norms related to underage alcohol use) showed a reduction of alcohol provision to minors and arrests for drunk driving among 18- to 20-year-old drivers.^7,8 The Midwestern Prevention Project, which combined coalition-led strategies with school-based prevention activities, demonstrated reductions in past-month cigarette and alcohol use among middle school students.⁹ Not all coalition-based efforts have proven successful, however. For example, the Fighting Back¹⁰ and Community Partnership¹¹ initiatives did not yield reductions in youth alcohol use. Insufficient guidance on implementing prevention strategies and reliance on locally created prevention programs not tested for effectiveness likely played roles in the lack of effects.

A coalition-based community strategy that showed significant effects on youth outcomes is Communities That Care (CTC).^12,13 CTC is a manualized system to mobilize communities to develop and transform their prevention systems to address elevated risk and depressed protective factors for youth problem behaviors through the appropriate selection, installation, and monitoring of tested and effective preventive interventions.^14,15 According to the CTC theory of change, CTC implementation leads to reduced problem behavior and positive development among youths through activating and reinforcing 5 prevention system constructs:

1. adoption of a science-based approach to prevention,¹⁶

2. community support for prevention,^17,18

3. community norms against adolescent drug use,¹⁹

4. collaboration across community service sectors for prevention,^20,21 and

5. use of the social development strategy.²²

Although each construct is theorized to be important,^23–26 adoption of a science-based approach to prevention, which refers to community leaders’ understanding and use of a prevention science framework to plan and implement programs to prevent youth problem behaviors,¹⁶ is the primary mechanism through which CTC is expected to produce positive changes in outcomes.²⁷

Findings from the Community Youth Development Study (CYDS), a community-randomized controlled trial designed to test the efficacy of CTC,^12,13 were consistent with this theory of change. Previous studies found positive effects of CTC on prevention system constructs. Earlier CYDS findings showed that CTC compared with control community leaders reported higher levels of adoption of a science-based approach to prevention in 2004 and 2007, 1.5 and 4.5 years after initial CTC implementation, respectively.^28,29 Also, at 4.5 years after implementation, leaders from CTC compared with control communities reported a higher percentage of funding desired for prevention activities and, among communities with higher proportions of residents in poverty, higher levels of community norms against adolescent drug use.

As CTC communities transformed their prevention systems, risk factors for problem behaviors that were selected by CTC communities to be targeted for intervention were also affected.^30,31 Using data from a sample of youths followed longitudinally from fifth through eighth grades over a corresponding time period (2004–2007) from the same randomized CYDS communities, studies demonstrated a slower growth of levels of targeted risk factors among youths from CTC compared with control communities.^30,31 Finally, data from these longitudinal studies showed that positive youth outcomes were achieved. Compared with those from control communities, CTC youths were less likely to initiate delinquent behavior and use of alcohol, cigarettes, and smokeless tobacco; were less likely to use alcohol and smokeless tobacco in the past month and to binge drink in the past 2 weeks; and reported fewer past-year delinquent behaviors.^30,31

Recent research from the CYDS longitudinal sample assessed during 10th grade, 1 year after the removal of study funding and resources for intervention to CTC communities, showed that youths from CTC compared with youths from control communities continued to have a reduced risk of initiation of alcohol and cigarette use and delinquent behavior, lower prevalence of current smoking and past-year delinquent behavior, and lower levels of risk factors targeted by communities.³² These findings indicated that the effects of interventions implemented through CTC during the study period were sustained for a year beyond funding.

An important next question was whether CTC’s effects on prevention systems were also sustained. Data from previous CYDS analyses of prevention system characteristics were collected while CTC communities were receiving study-funded intervention resources. The time shortly after removal of funding was an important period in the long-term viability of this intervention. Because of challenges with competing demands and sustaining momentum, interventions often cease after withdrawal of study funding.^33,34 Without study resources, will CTC communities abandon the use of science to guide prevention strategies or decrease their commitment to prevention efforts? Is an initial round of CTC training and implementation sufficient to generate sustained system-wide changes in prevention systems that have potential for continued long-term reductions in youth problem behaviors?

As a follow-up to previous studies and as part of the ongoing CYDS, this study examined whether CTC’s effects on prevention system constructs were sustained through 2009—approximately 6.5 years after initial implementation, 2 years after the previous data collection, and 1.5 years after study resources for CTC implementation were withdrawn. This study also examined whether effects of CTC on prevention system constructs differed by community-level characteristics, including percentage of residents living in poverty, percentage of non-White residents, and population size.

METHODS

Participating communities were 24 geographically distinct, incorporated towns in 7 US states: Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington. Population sizes of participating communities ranged from 1400 to 45 000 according to the 2000 US Census. Communities were selected from a larger pool of communities participating in a naturalistic study of the diffusion of prevention science.²⁷ Twelve pairs of communities not showing evidence of implementing a science-based approach to prevention in 2002 were selected to participate in CYDS. One community from each pair was assigned randomly to the CTC intervention or control condition. CTC intervention activities, which began in the summer of 2003, included trainings for community stakeholders from certified trainers and technical assistance, such as regular correspondence and community site visits from CYDS staff. Study funding to CTC communities for intervention resources ended in the spring of 2008.

Participants

Participants for the present study were community key leaders interviewed in 2001, 2004, 2007, and 2009. A 2-stage approach was used to sample participants at each wave. First, community leaders across 10 service sectors were identified and selected for participation. These positional leaders included mayors, school superintendents, business leaders, etc. For the second sampling stage, positional leaders named 2 individuals knowledgeable about the community’s prevention efforts. Of the referred individuals, the 5 most frequently recommended from each community were interviewed. Figure 1 shows the flow of communities and participants through the study. Between 336 and 354 leaders responded at each wave. A total of 928 leaders were surveyed at 1 or more waves

FIGURE 1—

Flow of communities and participants in the randomized trial: Community Youth Development Study, United States, 2009.

Note. CTC = Communities That Care.

Primary measures assessed were from the Community Key Informant interview,³⁵ a 1-hour telephone survey about a community’s characteristics and approach to prevention.

Adoption of a science-based approach to prevention.

This measure consisted of 21 items asking leaders about their knowledge of prevention science concepts and their community’s use of epidemiological data, use of tested and effective prevention programs, and system monitoring.²⁷ Based on a prevention science framework and Roger’s theory of innovation diffusion,³⁶ stage of adoption was coded into 1 of 6 discrete levels from 0 to 5: (0) little or no awareness of prevention science concepts, (1) awareness of prevention science terminology and concepts, (2) attention to risk and protective factors in the community’s prevention planning, (3) collection of epidemiological data on risk and protective factors to guide prevention planning, (4) selection and implementation of tested and effective preventive interventions to address prioritized risk and protective factors, and (5) continued collection of epidemiological data for program evaluation, system monitoring, and adjustment of prevention programming. Each higher stage incorporated the elements of the lower stages (e.g., a respondent coded as stage 5 also met the criteria for stages 1–4). Using a multilevel analysis of leaders nested within communities across time, inter-rater reliability (i.e., the ratio of true score variance to total variance) was found to be high (0.81).²⁹

General community support for prevention.

This construct was measured as a second-order latent variable derived from 2 first-order latent variables: community member support and community leader support. To assess community member support, leaders rated the degree (strongly disagree, somewhat disagree, somewhat agree, strongly agree) to which community members believed in prevention effectiveness, knew about prevention efforts in the community, and were willing to pay higher taxes to support prevention activities. To assess community leader support, leaders rated these same items in reference to themselves rather than their community as a whole. Internal consistency of community support items was high (α ≥ 0.77).²⁹

Desired prevention funding.

This measure was derived from 1 question that asked, “If you were deciding how to spend money for reducing substance abuse, what percentage would you allocate to each of the following approaches? Law enforcement, treatment, and prevention.” The percentage reported for prevention was used in analyses.

Community norms against adolescent drug use.

This measure was derived as a first-order factor using 6 items: “In this community, how wrong do most adults think it is for adolescents to (1) drink alcohol, (2) smoke cigarettes, and (3) use marijuana (not wrong at all, a little wrong, wrong, very wrong),” and “Adults in [community] think that using (4) alcohol, (5) tobacco, and (6) marijuana are a normal part of growing up (strongly disagree, somewhat disagree, somewhat agree, strongly agree).” Confirmatory factor analyses suggested that these items reflected a unitary construct, and multilevel analyses indicated that community leader reports showed significant agreement with student reports of community norms.³⁷

Community collaboration for prevention.

This measure was derived as a second-order factor from 2 first-order components: sectoral collaboration and prevention collaboration. Sectoral collaboration was indicated using 7 items assessing frequency (none, some, a little, a lot) of collaboration with members from each of 7 other service sectors on prevention issues. Prevention collaboration was indicated by the degree of agreement (strongly disagree, somewhat disagree, somewhat agree, strongly agree) on 9 items about community leaders’ engagement in specific prevention collaborative activities (e.g., sharing resources, coordinating activities). This measure showed high construct validity according to multilevel confirmatory factor analyses.³⁸

Use of the social development strategy.

Based on core elements of the social development model,³⁹ this was measured as a first-order latent variable indicated by agreement (strongly disagree, somewhat disagree, somewhat agree, strongly agree) on 5 items concerning community leaders’ work to (1) increase opportunities for youth involvement in prosocial activities; (2) help adolescents learn new skills; (3) recognize and compliment youths for positive effort, improvement, and accomplishments; (4) promote bonding between youths and prosocial members of the community; and (5) ensure clear standards for youth behavior. This measure was assessed in 2007 and 2009 only. This measure showed high internal consistency (α = 0.91).²⁹

Demographic information included leader’s gender, age, level of education, years resided in the community, positional versus referred status, and number of waves responded. Table 1 presents the distribution of these characteristics in the sample. Using data from the 2000 US Census, community characteristics were also assessed: total population size, percentage of non-White residents, and percentage of residents living in poverty.

TABLE 1—

Community Key Leader Characteristics by Intervention Condition: Community Youth Development Study, United States, 2009

Characteristic	Control, No. (%) or Mean ±SD	CTC, No. (%) or Mean ±SD
Participants	463 (49.9)	465 (50.1)
No. of waves responded
1	325 (70.2)	300 (64.5)
2	83 (17.9)	114 (24.5)
3	29 (6.3)	36 (7.7)
4	26 (5.6)	15 (3.2)
Y participated
2002	176 (38.0)	178 (38.3)
2004	169 (36.5)	171 (36.8)
2007	164 (35.4)	172 (37.0)
2009	173 (37.4)	175 (37.6)
Referred	177 (38.2)	174 (37.4)
Female	197 (42.6)	186 (40.2)
Age, y	49.8 ±10.0	49.3 ±10.7
Y lived in community	16.8 ±17.7	16.9 ±17.4
≥ bachelor’s degree	215 (46.4)	205 (44.4)

Note. CTC = Communities That Care.

Statistical Analyses

Analyses were conducted in the fall of 2010. Multilevel models were used to compare levels of prevention system constructs between CTC and control community leaders using HLM 6.0 software (Scientific Software International, Skokie, IL).⁴⁰ Survey wave was specified at level 1; leader characteristics at level 2; and community characteristics, including intervention condition (0 = control, 1 = CTC) and US Census variables, at level 3. This model partitioned the variance in outcomes across the 3 levels and examined the effects of CTC at the appropriate unit of randomization (i.e., communities) with appropriate degrees of freedom (i.e., df = 20). With the exception of adoption of a science-based approach, prevention system outcomes were modeled as continuous. Because of its ordinal stages, adoption was analyzed using a cumulative probability model, in which odds ratios were estimated describing the odds of being in a higher stage of adoption according to intervention condition. To examine mean differences in constructs in 2009 between CTC and control community leaders, a fully multivariate model was estimated,⁴¹ in which each of the 4 waves was incorporated into the level 1 equation with no intercept term; the 2009 wave was coded as 1, and other waves were coded as 0. Additional models included interaction terms between CTC condition and population size, percentage living in poverty, and percentage of non-White residents. To examine differences in change over time between CTC and control communities in prevention system constructs, survey wave was specified as a linear slope (coded 0–3), and effects of time were allowed to differ by intervention condition. Models were adjusted for characteristics at the individual level (gender, age, level of education, years resided in the community, positional vs referred status, and number of waves responded) and community level (total population size, percentage of non-White residents, and percentage of residents living in poverty).

RESULTS

Table 2 presents regression coefficients from multilevel models describing differences in prevention system constructs between CTC and control community leaders. The difference in stage of adoption of a science-based approach to prevention in 2009 between CTC and control community leaders after adjusting for covariates was statistically significant (P < .001). The adjusted odds ratio indicated that CTC compared with control community leaders had 4.34 times higher odds of reporting a higher stage of adoption in 2009 (95% CI = 2.76, 6.83). Model-implied probabilities were 0.36 for CTC communities and 0.12 for control communities for being in the highest stage of adoption, and 0.22 for CTC communities and 0.55 for control communities for being in the lowest stage (Figure 2). There was no evidence of a moderating effect of population size, percentage of non-White residents, or the percentage of residents living in poverty on the CTC intervention. Comparing slopes, the change in log odds of stage of adoption per wave from 2001 to 2009 was 0.56 higher among leaders in CTC communities compared with leaders in control communities (P = .001), where stage of adoption increased during this period among leaders in CTC communities but remained fairly stable in control communities.

TABLE 2—

Multilevel Model Parameter Estimates for Communities That Care (CTC) Compared with Control Conditions: Community Youth Development Study, United States, 2009

	Mean in 2009		Slope Over Timea
Prevention System Construct	b (SE)	P	b (SE)	P
Adoption of a science-based approachb	1.470c (0.22)	< .001	0.56 (0.14)	.001
General community support for prevention	0.080 (0.07)	.25	0.04 (0.03)	.25
Desired prevention funding	3.510 (1.52)	.032	1.06 (0.63)	.11
Community norms against adolescent drug use	0.130 (0.14)	.38	0.06 (0.03)	.044
Community collaboration for prevention	−0.006 (0.05)	.9	0.03 (0.02)	.13
Use of the social development strategy	0.050 (0.13)	.72	NA	NA

Note. NA = not available.

^aTime specified as survey wave, 0 (2001) to 3 (2009).

^bAnalyzed using cumulative probability model to describe difference in log-odds of being in 1 stage higher of adoption.

^cOdds ratio = 4.34; 95% confidence interval = 2.76, 6.83.

FIGURE 2—

Predicted probability of reporting each stage of adoption of a science-based approach to prevention among Communities That Care (CTC) and control communities: Community Youth Development Study, United States, 2009.

The adjusted model-estimated percentages for desired prevention funding in 2009 were 46.1% among community leaders in CTC communities and 42.6% among control community leaders. This difference was statistically significant (b = 3.51; P = .032). A statistically significant interaction between CTC and percentage of non-White residents was also observed (P = .038), where the effect of CTC was restricted to communities with a higher proportion of non-White residents. There was no significant difference between CTC and control community leaders in change in desired funding over time.

The mean level of community norms against adolescent drug use in 2009 did not differ significantly between CTC communities and controls. However, CTC community leaders showed an increase over time in norms against adolescent drug use from 2001 to 2009 that was significantly greater than that reported by control community leaders (b = 0.059; P = .044). There were no statistically significant interactions between CTC and any of the community characteristics on community norms in 2009.

For general community support for prevention, community collaboration, and use of the social development strategy, there were no statistically significant differences in mean levels in 2009 between CTC and control community leaders and no significant interactions between CTC and community characteristics. Furthermore, there was no difference in change over time in community support or community collaboration reported by leaders from CTC versus control communities. No tests for differences in slopes were conducted for use of the social development strategy because only 2 waves of data were available for this construct.

DISCUSSION

Previous research found that CTC had positive effects on reducing levels and delaying the initiation of youth problem behaviors both during the time CTC communities were receiving study funded resources for intervention activities,³¹ as well as 1 year after removal of funding.³² Consistent with the CTC theory of change, previous studies showed that CTC also had effects on prevention system characteristics.^28,29 The present study provided evidence that effects on prevention system characteristics were sustained 2 years after the previous wave of interviews and 1.5 years after study funded resources for CTC ended. Furthermore, this study found a greater increase in norms against adolescent drug use from 2001 to 2009 as reported by leaders in CTC compared with control communities. Similar to previous findings,²⁹ there was no evidence of a main effect of CTC on community collaboration for prevention, general community support for prevention, or use of the social development strategy.

The findings of sustainability in prevention system changes are consistent with other research showing sustainability of other aspects of the CTC system. In a separate quasi-experimental study of CTC communities conducted in Pennsylvania, of 110 communities that received a 3-year grant to implement CTC, 90% continued to have functioning CTC coalitions after the initial funding period.⁴² Furthermore, many of these sites were able to secure funding at levels higher than the initial 3-year grant funding. Because of the integral role of CTC coalitions in implementing tested and effective prevention programs and monitoring the effectiveness of the system, sustained CTC coalition functioning is likely to help maintain community-wide changes prevention system changes.

It was noteworthy that of the constructs examined, adoption of a science-based approach to prevention showed the strongest difference between leaders from CTC compared with control communities in 2009. Although other system constructs were expected to be important, adoption of a science-based approach to prevention was the primary mechanism through which CTC was theorized to yield positive youth outcomes.²⁷ The fact that CTC community leaders continued to report that their community sought to address elevated risk and depressed protective factors through the installation and implementation of evidence-based preventive interventions and ongoing monitoring of prevention system effectiveness suggested that fundamental change in community systems was being achieved and sustained.

Although no effect of CTC on leader-reported general support for prevention was observed, there was a positive effect of CTC on desired prevention funding in 2009. Without adequate funding, efforts to transform prevention systems are likely to stall.⁴³ Furthermore, obtaining local funding is a key factor for sustaining prevention programs after external funding has ended.³⁴ Thus, if community leaders and stakeholders demonstrate strong support for prevention through desiring greater funding allocation toward prevention activities, this would increase the likelihood of local funding and maintenance of effective prevention strategies.

In contrast to earlier waves,²⁹ change over time in community norms against drug use between 2001 and 2009 was significantly higher among CTC compared with control community leaders. CTC might have a slower effect in shifting community norms as perceived by community leaders. It was interesting to note that although an effect on change over time was observed, there was no difference between CTC and control leaders’ reports of the mean level of community norms in 2009. This might be related to increased power to detect differences in change in this measure over multiple time points rather than at 1 specific time point.

The sustained effects of CTC on prevention systems might be explained in part by aspects of CTC implementation that built community capacity for social interventions that are commonly accepted as important for sustainability.⁴⁴ For example, CTC actively engaged and activated a diverse set community leaders and stakeholders through trainings and coalition building, and CTC coalitions developed prevention strategies that were uniquely tailored to the community’s specific risk and protection profile and demographic characteristics.¹⁵ Furthermore, technical assistance was available to communities for a relatively long period (5 years), which should have allowed sufficient time for installation of the CTC system with fidelity. These capacity-building elements might help CTC communities make long-term changes to their prevention systems.

As observed earlier,²⁹ this study found no difference in levels of prevention collaboration between CTC and control community leaders. This might be surprising given the theoretical importance of collaboration in transforming community prevention systems.²⁵ However, because of the emphasis on collaboration in federal and state programs that provide funding for prevention to communities,^26,45 virtually all communities might have understood the need for collaboration. It appeared that high levels of collaboration on prevention activities became normative in both CTC and control communities. Earlier studies of efforts to prevent substance abuse through community collaboration showed that collaboration alone was not sufficient to produce changes in drug use.⁴⁶ Thus, collaboration might be a necessary, but insufficient, condition to reduce adolescent substance use community wide.

There were limitations to this study. First, participating communities were small- to-moderate-sized incorporated towns. Thus, these findings might be limited in their generalizability to urban or suburban communities. Furthermore, because data were reported by community leaders and referred individuals, some of whom were members of CTC coalitions in intervention communities, social desirability might have influenced reporting. However, the finding that certain prevention system constructs differed across CTC and control communities whereas others did not suggests there was no overall “halo” effect of CTC on leaders’ responses. Furthermore, the fact that differences were observed 18 months after funding ended suggested reported differences between leaders of CTC and control communities reflected actual sustained differences in the perceptions, norms, and intentions of key leaders.

This study suggested that CTC had sustained effects on prevention system characteristics beyond the withdrawal of study support. Thus, the study funded resources and support provided for CTC implementation in intervention communities might be adequate to produce sustained change to prevention systems characteristics, particularly the adoption of a science-based approach to prevention. According to the CTC theory of change, this should lead to long-term public health benefits in the form of reduced occurrence of problem behaviors among youths.