Sustainability of the Communities That Care Prevention System by Coalitions Participating in the Community Youth Development Study

Abstract

Purpose

Community prevention coalitions are a common strategy to mobilize stakeholders to implement tested and effective prevention programs to promote adolescent health and well-being. This paper examines the sustainability of Communities That Care (CTC) coalitions approximately 20 months after study support for the intervention ended.

Methods

The Community Youth Development Study (CYDS) is a community-randomized trial of the CTC prevention system. Using data from 2007 and 2009 coalition leader interviews, this study reports changes in coalition activities from a period of study support for CTC (2007) to 20 months following the end of study support for CTC (2009), measured by the extent to which coalitions continued to meet specific benchmarks.

Results

Twenty months after study support for CTC implementation ended, 11 of 12 CTC coalitions in the CYDS still existed. The 11 remaining coalitions continued to report significantly higher scores on the benchmarks of phases 2 through 5 of the CTC system than did prevention coalitions in the control communities. At the 20-month follow-up, two-thirds of the CTC coalitions reported having a paid staff person.

Conclusions

This study found that the CTC coalitions maintained a relatively high level of implementation fidelity to the CTC system 20 months after the study support for the intervention ended. However, the downward trend in some of the measured benchmarks indicates that continued high-quality training and technical assistance may be important to ensure that CTC coalitions maintain a science-based approach to prevention, and continue to achieve public health impacts on adolescent health and behavior outcomes.

Introduction

Adolescent health promotion is a national priority in the United States and is a key component of Healthy People 2020, a 10-year federal agenda for improving the health of all Americans [1]. Community-level efforts to prevent substance use, delinquency, violence, and other problem behaviors are an essential component of a national strategy for promoting health during adolescence as well as throughout the life span [1, 2]. Over the past 20 years there has been significant progress in the development of effective preventive interventions, providing evidence that quality prevention programs can reduce adolescent health risk behaviors and promote healthy adolescent development [3-5]. While governmental and other funding sources increasingly require that communities implement programs with evidence of effectiveness, communities continue to struggle in selecting, implementing, and sustaining effective prevention programs with fidelity.

Community prevention coalitions are a popular way to mobilize stakeholders to address adolescent health promotion. There is increasing evidence that well-functioning coalitions, implementing tested and effective prevention programs, can create population-level change in adolescent health behaviors, including substance use and delinquency [6-8]. Communities That Care (CTC) is a prevention system that empowers community stakeholders to collaborate through a coalition to develop and implement a science-based community prevention system [9]. A series of trainings in the CTC system guides community coalitions through a process of identifying and prioritizing specific risk and protective factors for substance use and delinquency among local youth; selecting and implementing tested and effective prevention programs that address the targeted risks and protections; and then monitoring the community's prevention system to ensure implementation fidelity of selected programs and determine whether the intended health outcomes are obtained [10]. The CTC prevention system addresses several of the barriers community coalitions face by helping coalitions to set clear, measurable, and achievable goals, implement tested and effective programs, monitor implementation fidelity, and evaluate program impacts in relation to the community's goals [11, 12].

Findings from the Community Youth Development Study (CYDS), a community-randomized trial of the CTC system, revealed that CTC coalitions, with high-quality training from CTC Master Trainers and ongoing, weekly technical assistance from study staff, implemented the CTC prevention system with high fidelity [13]. In fact, CTC coalitions were significantly more likely to implement 15 key components of a science-based prevention system (e.g., assess and prioritize risk and protective factors; select and implement tested, effective interventions to address the prioritized factors; monitor implementation fidelity and intervention impacts) than prevention coalitions in control communities [14]. CTC communities also showed significant community-level reductions in the initiation and prevalence of substance use and delinquent behaviors among adolescents in comparison to the control communities [7]. Despite this evidence for the effects of CTC on community health-promotion goals, sustaining community coalitions and their impact on public health, especially after the initial funding for implementation is gone, remains a challenge for many communities [15, 16].

This paper examines the sustainability of CTC coalitions 20 months after CYDS study support ended. Prior research studying state-supported CTC coalitions in Pennsylvania found that about 10% of those coalitions did not survive for 1 year after state funding ended, with an additional 3%-8% of the coalitions discontinuing each subsequent year [15]. The current study extends these findings by assessing the sustainability of CTC coalitions in 7 additional states. For the purposes of this paper, we defined sustainability as the extent to which the coalitions maintained their structure and fidelity to the CTC prevention system, and were able to obtain funding to sustain the coalition's staff and prevention initiatives after study funding ended [16, 17].

Through CYDS participation, 12 intervention communities received funding for a paid staff person to facilitate the work of the coalition and to install tested and effective preventive interventions to address community needs. The ability of a coalition to obtain continued funding for a staff person to facilitate coalition work and to support preventive interventions after the CYDS funding period ended is hypothesized to have an impact on how well the coalition maintained implementation fidelity to the CTC prevention system beyond the period of support from CYDS. This paper addresses three research questions. 1) To what extent did the CTC coalitions in the intervention communities sustain their structure and fidelity to the CTC system 20 months after support from the CYDS ended? 2) Were the CTC coalitions in intervention communities able to obtain funding for a paid staff person and the interventions implemented during the CYDS after support from the CYDS ended? 3) Were the differences in the implementation of a scientific approach to prevention between CTC coalitions in the intervention communities and prevention coalitions in the control communities maintained 20 months after CYDS funding, training, and technical assistance was withdrawn?

Method

Community Sample and Randomization

The CYDS is a randomized controlled trial of the CTC prevention system. The study includes 12 pairs of communities from the states of Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington matched with regard to population size, poverty, ethnic and racial diversity, and crime indices in an earlier study. These 12 pairs of matched communities (24 communities in total) were recruited in the fall of 2002 to participate in the study. Each pair of eligible communities was randomized to intervention or control condition by a coin toss [18]. This study was approved by the University of Washington Institutional Review Board.

CTC Implementation

As described by Arthur and colleagues [14], the CTC intervention began in the summer of 2003. The 12 communities that were randomly assigned to the intervention condition were instructed to identify an existing prevention coalition in their community or to form a new coalition to implement the CTC system. These 12 coalitions were provided tailored training and technical assistance to help them use student survey data collected in 1998, 2000, and 2002 [18, 19] to prioritize specific risk and protective factors to target in their communities. The training and technical assistance also helped communities select appropriate tested and effective prevention programs, and to implement with fidelity and evaluate the selected programs [13]. By April of 2004, intervention communities had selected preventive interventions to address their prioritized risk and protective factors and had created strategic community plans to implement these interventions. The selected programs were implemented in intervention communities with financial support from the study beginning in the fall of 2004 and continuing through the spring of 2008.

The 12 intervention communities received study funding to hire a full-time coalition coordinator and to support the initial implementation of the tested and effective preventive interventions selected by the communities in the second through fifth years of the study. Control communities did not receive study funding, training, or technical assistance.

Coalition Leader Interviews

Telephone interviews with 43 prevention coalition leaders across the 24 CYDS communities were conducted in the winter and early spring of 2002 to create a baseline. Sixty-two prevention coalition leaders were interviewed in the winter and early spring of 2007 during the 4^th year of CTC intervention, and 58 prevention coalition leaders were interviewed in fall and winter of 2009, approximately 20 months after the end of the CYDS-supported intervention. Coalition leaders in this study were defined as the chairs of the coalitions, and were identified using a snowball sampling approach through telephone interviews conducted with community leaders and directors of community agencies and organizations providing prevention services to youth and their families [14, 20]. These coalition chairs were then interviewed to assess the focus and nature of the prevention activities in which the coalitions were engaged. One respondent from each identified coalition was interviewed, and these interviews averaged 45 minutes in length.

Measures

The measures used to assess the sustained implementation of the CTC system are questions asking about coalition structure, funding, and specific benchmarks associated with fidelity of implementation of CTC Phases 2 through 5, the community action phases of CTC [13, 14]. Although these actions correspond to specific benchmarks in the CTC system, the actions are generic enough that the questions were asked in terms that could apply to any prevention coalition implementing a science-based approach to prevention. For example, “Has your coalition ever been to a training to learn about risk- and protective-focused prevention?” “Has your coalition ever used an assessment of risk and protective factors in your community for prevention planning?” “Did your coalition develop a specific, written action plan as part of planning to implement prevention programs for substance abuse?” “Thinking about (Coalition Name)'s comprehensive plan or activities, were there any specific risk or protective factors that you have been focusing on?” “Were specific programs or activities implemented to address these factors?” Each benchmark was coded based on the responses of the coalition chair as either (0) Not accomplished, or (1) Accomplished by the coalition. Index scores for each of the 4 phases of CTC implementation were created by summing the number of benchmarks accomplished by the coalition within each phase, and a total score was created indicating the total number of benchmarks completed by the coalition across all 4 phases.

Analysis

By 2009, one CTC coalition had dissolved, rendering data from that coalition unavailable for this study. The following analyses are based on the 11 remaining CTC coalitions that participated in the coalition interviews conducted in 2009. Analyses first examined structure and funding characteristics of the 11 coalitions in 2009. Next, analyses compared the implementation scores of these 11 CTC coalitions assessed in 2009 with their implementation scores assessed in 2007. Cochrane's Q and paired sample t tests were used to test for changes in implementation fidelity among the CTC coalitions from 2007 to 2009. Comparisons were conducted on the completion of each individual benchmark, then on the mean number of benchmarks completed within each CTC phase, and finally on the total mean number of benchmarks completed across all phases. Cochrane's Q was also used to test whether the number of coalitions that had a paid staff person changed significantly from 2007 to 2009. The second set of analyses examined whether the CTC coalitions maintained a significantly higher level of implementation of a science-based prevention system 20 months after support from the CYDS ended compared to the 23 prevention coalitions identified in the control communities in 2009. Chi-squared tests and independent sample t tests were used to test for significant differences between the 2 sets of coalitions.

Results

By the fall of 2009, approximately 20 months after the end of project support, 1 of the 12 CYDS-supported CTC coalitions had disbanded. This coalition was in a small town, unable to secure funding for the coalition coordinator after the CYDS funding ended. However, some of the CTC-initiated prevention programming continued in this community through support from a county-level coalition. Each of the remaining 11 coalitions continued to meet regularly; 5 of the 10 who responded to the question meeting at least once a month, 3 meeting bi-monthly, and 2 quarterly. All but 1 of the 11 coalitions continued to set the agenda and take notes during the meeting, and all but 2 had elected someone to facilitate the meetings.

In terms of funding, 7 of the 11 coalition chairs reported that the level of their coalition's funding had changed from the previous year, with 3 saying it had decreased somewhat and 4 saying it had decreased by more than half. Nine of the 11 coalition chairs provided information about the total amount and sources of their coalition's funding. Of these nine, all but one reported they had been able to raise some funds for the coalition's activities, ranging from $5,800 to $240, 000 (Mean $104,533, SD $83,675). The most common sources of funding were from the federal and state government, followed by charitable sources and local government. Seven of the 11 coalitions reported they continued to support a paid staff person, and all but 1 of the 11 continued to support at least 1 of the tested and effective programs they had been supporting when study funding ended. Twenty-two of the 33 programs supported by the 11 coalitions at the end of study funding were still being supported, and 5 coalitions had added a new tested, effective program by the 20-month follow up.

Table 1 shows the proportions of the CTC coalitions that attained each benchmark in 2007, during the 4^th year of the intervention, and in 2009, 20 months after the CYDS support of the coalitions ended. Although coalition leaders were less likely to report that the coalition met some of the benchmarks in 2009 than in 2007, none of the changes in the proportions of coalitions meeting the benchmarks were significant—i.e., we were not able to reject the hypothesis that the proportion of coalitions meeting each benchmark was equivalent in both years. This suggests that the level of implementation of the CTC system did not degrade significantly among the 11 remaining CTC coalitions for 20 months after study support for CTC ended. The greatest change observed was in reports that the coalition had an explicit, written action plan. In 2007, 92% of CTC coalition leaders reported that their coalition had developed an explicit, written action plan; in 2009 that percentage was reduced to 55%. This decline approaches, but does not achieve, statistical significance (p=.102). Several of the benchmarks were met by 100% of the CTC coalitions in both 2007 and 2009, including assessing the risk and protective factors in the community using student surveys, and focusing on specific risk or protective factors prioritized by the coalition. There was no significant change in how the 11 CTC coalitions scored on the benchmarks of each of CTC phases 2 through 5.

Table 1.

Percent of CTC coalition leaders reporting that CTC Milestones & Benchmarks were achieved in 2007, the 4^th year of the intervention, and in 2009, 20 months after study support ended.

	CTC Coalitions
CTC Stage	2007	2009+
CTC Stage 2: Organize, Introduce, & Involve (Scale 0-3)	2.82	2.55
Does the coalition use a risk/protective framework?	100%	91%
Has the coalition received training in the risk/protective framework?	100%	100%
Have 75% or more coalition members been trained in the risk/protective framework?	83%	64%
CTC Stage 3: Develop a Community Profile (Scale 0-5)	4.82	4.73
Has the coalition assessed risk and protective factors in the community?	100%	100%
...using student surveys?	100%	100%
...using archival indicators?	92%	73%
Has the coalition focused on any specific risk or protective factors?	100%	100%
Has the coalition assessed prevention resources in the community?	92%	100%
CTC Stage 4: Create Community Action Plan (Scale 0-1)	0.91	0.55
Did the coalition develop an explicit, written action plan?	92%	55%
CTC Stage 5: Implement & Evaluate the Action Plan (Scale 0-6)	4.55	4.18
Were specific programs implemented to address these factors?	100%	91%
Did the coalition sponsor at least two TEPs?	100%	82%
Has the coalition evaluated or monitored the results of these programs?	100%	91%
...using pre/post surveys?	100%	82%
...using changes in participant outcomes (e.g., delinquency, school performance)	92%	82%
...using changes in participant risk and protective factors?	67%	73%
CTC Milestones & Benchmarks Total (Scale 0-15)	13.10	12.00

*p<.05 **p<.01

⁺20 months after study funding ended

Table 2 shows that in 2009, coalition chairs from the 11 remaining CTC coalitions reported significantly higher levels of implementation of a science-based prevention system than prevention coalitions in the control communities as described by the proportion of coalitions that met the CTC benchmarks. For example, CTC coalition members were significantly more likely than members of the control coalitions to have received training in the risk/protective framework, to have assessed risk and protective factors in the community using student surveys and archival indicators, and to have developed an explicit, written action plan. CTC coalitions were also more likely to report focusing on specific risk and protective factors, implementing specific programs to address these factors (compared to the control coalitions), and to have evaluated and monitored the results of the programs. Looking at the average scores on each of the CTC stages, the CTC coalitions scored significantly higher on all of the stages compared to coalitions in the control communities. On average, the CTC coalitions completed 13 of the 15 benchmarks of a science-based prevention system, while the coalitions in the control communities completed about 6 of the benchmarks.

Table 2.

Communities That Care (CTC) Benchmarks completed by coalitions in intervention and control communities in 2009, 20 months after study support ended

CTC Stage	CTC coalition N=11	Control coalition N=23
CTC Stage 2: Organize, Introduce, and Involve (Scale 0-3)	2.55*	1.39
Average # of prevention coalitions per community	1.0	2.6
Does the coalition use a risk/protective framework?	91%	65%
Has the coalition received training in the risk/protective framework?	100%**	52%
Have 75% or more coalition members been trained in the risk/protective framework?	64%**	22%
CTC Stage 3: Develop a Community Profile (Scale 0-5)	4.73**	2.22
Has the coalition assessed risk and protective factors in the community?	100%**	52%
...using student surveys?	100%**	52%
...using archival indicators?	73%*	35%
Has the coalition focused on any specific risk or protective factors?	100%*	70%
Has the coalition assessed prevention resources in the community?	100%**	48%
CTC Stage 4: Create Community Action Plan (Scale 0-1)	0.55*	0.22
Did the coalition develop an explicit, written action plan?	55%*	22%
CTC Stage 5: Implement and Evaluate the Action Plan (Scale 0-6)	5.18**	2.30
Were specific programs implemented to address these factors?	91%*	52%
Did the coalition implement at least 2 tested and effective programs?	82%**	4%
Has the coalition evaluated or monitored the results of these programs?	91%*	48%
...using pre/post surveys?	82%**	44%
...using changes in participant outcomes (e.g., delinquency, school performance)	82%**	44%
...using changes in participant risk and protective factors?	73%**	44%
CTC Milestones and Benchmarks Total (Scale 0-15)	13.00**	6.13

^*p<.05

^**p<.01

Discussion

Community coalitions have become a widespread strategy for mobilizing community stakeholders to implement tested and effective prevention programs designed to promote adolescent health and prevent risk behaviors such as substance use and delinquency. An increasing number of studies provide evidence that well-functioning community prevention coalitions focused on science-based prevention can have an impact on public health [6-8]. However there remain several challenges to understanding what factors influence coalition success and whether and how coalitions can sustain successes over time, especially once the original funding for launching a coalition is exhausted. This study reports findings from a randomized controlled trial of the CTC prevention system on the sustained implementation of a science-based approach to prevention 20 months after study funded ended.

The present study found that 20 months after support from the CYDS for CTC implementation ended, 11 of 12 CTC coalitions still existed. In large part, these 11 CTC coalitions were able to sustain the coalition's structure and prevention activities. Furthermore, in 2009, 20 months after the CYDS ended, the CTC coalitions continued to report significantly higher scores on the benchmarks of phases 2 through 5 of the CTC system than did the prevention coalitions in the control communities, indicating that CTC coalitions maintained a more scientific approach to prevention than coalitions that did not receive CTC training and technical assistance.

As noted above, 1 of the 12 CTC coalitions had dissolved by the time coalition leader interviews were conducted in the winter of 2009. Thus, a coalition leader could not be interviewed. Community residents reported that the coalition was dissolved after CYDS funding ended in part because the community was not able to support a paid staff person to facilitate coalition activities. Further, while 10 of the 11 existing coalitions had been able to obtain funding after study funding ended and had been able to continue supporting the use of tested, effective programs as initiated during the trial, 7 of the 11 coalition leaders interviewed said that their coalition's funding had declined during the past year and 4 of the 11 coalitions had not been able to support paid staff. It appears that the 4 CTC coalitions that did not continue to employ a paid staff person found ways to sustain the coalition without paid staff. It will be useful to learn how coalitions without paid staff maintain their coalition's activities given the hypothesis that maintaining a paid staff person to facilitate the coalition's activities is critical.

Early results from a study of the sustainability of the CTC prevention system in Pennsylvania suggested that the ability to maintain funding was a predictor of sustaining coalitions [16]. However, later findings from the same study found that continued funding does not fully account for coalition survival. A higher level of coalition board functioning predicts better capability to obtain continued funding [15]. According to these results, coalition sustainability is at least in part predicted by board functioning independent of funding. Studying the role of board functioning in the sustainability of CTC coalitions in the CYDS, and understanding how some coalitions were able to maintain a high level of functioning while others were not, will be a next step in our research.

Other factors could also play a role in sustaining the CTC coalitions and prevention system in these communities and will be examined in further research. Brown and colleagues [21, 22] reported several aspects of prevention system transformation in the intervention communities relative to control communities, including community leaders’ reports of adoption of a science-based approach to prevention and greater support for prevention, that might influence sustainability of the CTC coalitions. In addition, Fagan and colleagues (2008) found that using the CTC system, the intervention communities were able to implement tested and effective programs with high levels of fidelity as assessed by adherence to the core components of the program, adequate dosage, [23] quality of implementation, and participant engagement [24, 25]. Intervention communities were able to sustain significantly more tested and effective programs a year and a half following the end of study funding than control communities [26]. High fidelity implementation and greater perceived effectiveness of the coalition and its programs might also predict sustainability.

This study has several limitations. First, the interview data were obtained from a single respondent, typically the coalition's chairperson, from each coalition. It is possible that their perceptions of the coalition's activity were influenced by their high level of involvement and personal investment in the coalition. However, the congruence of these data on the CTC coalitions’ activities and those obtained from multiple raters’ perceptions of coalition activity suggests that these data were not biased by self-report [13]. Second, the sample of community prevention coalitions is relatively small, and does not allow for more complex analyses of the interactions among coalition and community characteristics that may impact CTC implementation fidelity and sustainability.

Previous analyses found that the coalitions that received training and technical assistance through the CYDS implemented the CTC system with a high level of fidelity, evidenced by the high level of benchmarks met in 2007, the 4^th year of the CTC intervention. The current study shows that the CTC coalitions were able to maintain a relatively high level of implementation fidelity to the CTC system 20 months after study funding for the CTC intervention ended. However, while the changes in the proportion of CTC coalitions that reported meeting the benchmarks and CTC stages 20 months after study funding ended were not statistically significant, a downward trend in the proportion of coalitions meeting certain benchmarks was observed, most obviously the proportion of coalitions that reported having a written, explicit action plan and maintained a paid staff person. Maintaining funding, especially during economic times when many states and communities are struggling, appears to be a significant challenge for prevention coalitions. Future research that looks more closely at how some of the CTC coalitions were able to sustain high levels of functioning after funding ended while others were not would be instructive. In addition, further follow-up of the CTC coalitions will illuminate whether this really is a downward trend in implementation fidelity in CTC coalitions over the long term, and whether the CTC coalitions are able to sustain themselves and maintain their activities without further assistance. It is possible that continued training and technical assistance will be important to ensure that CTC coalitions maintain implementation fidelity and continue to achieve community-level impacts on adolescent health and behavior outcomes.