Abstract
This study examined factors associated with the predicted and actual post-funding sustainability of evidence-based interventions implemented as part of the Pennsylvania Commission on Crime and Delinquency’s Research-Based Delinquency and Violence Prevention Initiative. Correlates of predicted post-funding sustainability included program staff, overall school support, and school administrator support. Additionally, predicted post-funding sustainability was strongly associated with actual post-funding sustainability. Other correlates of actual post-funding sustainability included financial sustainability planning and aligning the intervention with the goals of the agency/school. Five years post-funding 33% of the interventions were no longer operating, 22% were operating at a reduced level, and 45% were operating at the same level or a higher level than the final year of funding. These findings are discussed in terms of implications for increasing intervention sustainability, as well as implications for future research on intervention sustainability.
Keywords: Diffusion, Evidence-based intervention, Schools, Sustainability
The evidence base for interventions targeting the prevention and treatment of delinquency, substance abuse, and violence has grown dramatically in the past 20 years. As a result of public policies favoring their use, universal, selective, and indicated evidence-based interventions (i.e., those proven efficacious in well-designed studies) are increasingly adopted by schools and communities (Rohrbach et al. 2005). The potential public health impact of these interventions has not been realized, however, because they are rarely sustained in the long term (Gottfredson and Gottfredson 2002; Rohrbach et al. 2006).
Theories abound concerning the factors that are associated with intervention sustainability, yet few studies have assessed these relationships empirically. Therefore, research is needed that examines the factors associated with sustainability in order to inform the successful long-term implementation of interventions. In this study we use Scheirer’s (2005) definition of sustainability, “the program components developed and implemented in earlier stages are maintained after the initial funding or other impetus is removed” (p. 322).
In the present study, we examined the long-term post-funding sustainability of interventions implemented by community agencies and schools participating in the Pennsylvania Commission on Crime and Delinquency’s (PCCD) Research-Based Delinquency and Violence Prevention Initiative. The PCCD initiative, which was modeled after the U.S. Department of Justice’s “Blueprints” Initiative (Elliott and Mihalic 2004), began in 1998 with the goal of supporting the diffusion of evidence-based delinquency, substance abuse, and violence preventive interventions in Pennsylvania. Over the past 11 years, PCCD has provided more than $60 million to implement approximately 170 model interventions in over 120 Pennsylvania communities. Interested agencies/schools submitted grant applications outlining their programmatic needs, and the agencies/ schools selected to participate received 4 years of funding and technical assistance. Agencies/schools were responsible for selecting interventions based on the local collaborative board’s community risk and resource assessment, obtaining training, implementing interventions with fidelity, and planning for sustainability. These conditions are similar to what other agencies/schools experience when they adopt evidence-based interventions.
In addition to assessing the post-funding sustainability of the interventions, we also aimed to identify factors associated with predicted post-funding sustainability, the relationship between predicted post-funding sustainability and actual post-funding sustainability, and the factors associated with actual post-funding sustainability. Although empirical sustainability research is limited, current theory implicates several factors in the sustainability of evidence-based interventions. These factors include characteristics of the implementing site (e.g., community readiness), key relationships (e.g., collaboration with community coalitions) and the intervention itself (e.g., availability and quality of training and technical assistance; Adelman and Taylor 2000; Altman 1995; Elliott and Mihalic 2004; Johnson et al. 2004; Mancini and Marek 2004; Rohrbach et al. 2006), as well as financial sustainability planning (Adelman and Taylor 2000; Johnson et al. 2004). In the following sections, we provide a brief review of theory and research on the relationships between these factors and intervention sustainability.
Community Readiness and Sustainability
A growing body of research suggests that community readiness to adopt and implement evidence-based interventions is essential for sustainability. Although several conceptualizations of community readiness have been developed, common components include motivation for change, organizational capacity to implement change, and the support of influential leaders (Adelman and Taylor 2000; Elliott and Mihalic 2004; Johnson et al. 2004).
At the most basic level, agencies/schools that are unmotivated to make the changes and commit the time and energy required to implement an evidence-based intervention with fidelity are unlikely to sustain the intervention. This situation is not uncommon when communities adopt programs simply because there is grant funding being offered. Research suggests that lack of motivation may stem from several sources, including community perceptions about the need for the intervention, frontline implementers perception of the intervention’s potential effectiveness, and the alignment of intervention with the goals of the agency/school (Altman 1995; Chilenski et al. 2007; Scheirer 2005).
Even when motivated to implement, however, agencies/schools that lack sufficient organizational capacity may be unable to sustain interventions (Elliott and Mihalic 2004; Johnson et al. 2004; Rohrbach et al. 2006; Swisher 2000). Common components of organizational capacity include the ability of an agency/school to maintain trained and appropriate staffing levels (e.g., implementers and support staff), effectively manage funding, and work toward shared goals. Despite the consensus that organizational capacity is associated with program sustainability, it is important to note that few studies have empirically examined this relationship, indicating a need for further research in this area (Rohrbach et al. 2006).
In addition to motivation and organizational capacity, garnering the support of influential community/agency/school leaders is also considered by many to be an important aspect of community readiness (Adelman and Taylor 2000; Altman 1995; Elliott and Mihalic 2004; Johnson et al. 2004; Mancini and Marek 2004; Swisher 2000). Gingiss et al. (2006) found that the factor school-based leadership, which encompassed items such as principal’s involvement and importance of the program to the principal, predicted level and quality of school-based program implementation 2 years later.
Other studies also have found a positive relationship between principal and teacher support and implementation quality of school-based programs (Gager and Elias 1997; Payne et al. 2006). Additionally, in regard to both school-and community-based interventions research and theory suggest that key community leaders may play a vital role in sustainability (Altman 1995; Elliott and Mihalic 2004; Johnson et al. 2004; Scheirer 2005). It is important to note that interventions that align with the goals and needs of the agency/school are more likely to receive broad support from administrators and staff, and thus more likely to be properly implemented and sustained (Adelman and Taylor 2000; Altman 1995; Gager and Elias 1997; Gottfredson and Gottfredson 2002; Greenberg 2004; Johnson et al. 2004; Payne et al. 2006). As a whole, these findings suggest that multiple indicators of community readiness should be taken into account when examining sustainability.
Collaborations with Coalitions and Sustainability
In addition to community readiness, the extant literature suggests that collaborations with local coalitions also may impact intervention sustainability (Altman 1995; Greenberg 2004; Hawkins et al. 2002; Spoth et al. 2004). Research on Communities That Care (CTC; Hawkins et al. 2002), a model that utilizes community coalitions in order to assess community intervention needs and implement appropriate evidence-based interventions, indicates that CTC coalitions are sustainable and increase the adoption and implementation of evidence-based programs (CTC; Gomez et al. 2005; Hawkins et al. 2002).
There is a dearth of research, however, on the role of coalitions in the long-term sustainability of evidence-based interventions. Further, even less is known about the role of pre-existing coalitions with specific goals (e.g., substance abuse prevention and treatment) in intervention sustainability. Collaboration with community coalitions may result in increased funding, support, and integration of interventions in schools (Domitrovich and Greenberg 2000) and the community (Pentz 2000). For example, collaboration with coalitions may lead to broad community policy changes, which in turn may reinforce the values of the intervention (e.g., limiting minors’ access to alcohol and tobacco may work to enhance the effects of a substance abuse prevention program; Pentz 2000). The effectiveness of coalitions at supporting intervention activities, however, may vary based on the coalitions’ perceived and actual effectiveness and the strength of the connection between the coalition and program providers (Feinberg et al. 2004). Additional research clearly is needed in order to understand the ability of local coalitions to support or hinder the sustainability of evidence-based interventions implemented by community agencies and schools.
Intervention Characteristics and Sustainability
Aside from community readiness and collaboration with coalitions, research suggests that characteristics of the intervention and intervention support system, such as the quality of the available training and technical assistance, may also influence sustainability (Rohrbach et al. 2006). Although few studies have empirically examined the role of training and technical assistance in program sustainability, it is commonly believed that program implementers who are provided with sufficient training and technical assistance are more likely to understand the intervention logic model and lessons, thus improving implementation quality (Dusenbury et al. 2003; Gottfredson and Gottfredson 2002) and in turn, sustainability (Johnson et al. 2004). After the initial training period, technical assistance is believed to support continued implementation quality and sustainability.
Even in cases in which the implementers perceive training to be of high quality, however, they often still feel unprepared to effectively implement interventions (Elliott and Mihalic 2004). For example, Ennett and colleagues (2003) found that teachers who had been recently trained and were more comfortable with program material were more likely to implement interventions with fidelity. Although additional research is clearly needed, these findings indicate that implementers may be more likely to sustain interventions if they feel comfortable with the training they receive and have access to high-quality technical assistance.
Financial Sustainability Planning
Another key factor in the sustainability of evidence-based interventions is the procurement of ongoing financial support. Sustainability planning models suggest that implementing agencies and schools should discuss the financial future of interventions early in the implementation process, and that this type of planning is highly related to community/agency/school support of the intervention and perceived need for the intervention (Adelman and Taylor 2000; Johnson et al. 2004). Few studies, however, have examined the prevalence and/or process of agencies’ and schools’ financial sustainability planning. Thus, research is needed that determines how differences in financial sustainability planning relate to intervention sustainability.
The Current Study
This study was designed to contribute to the literature on the sustainability of evidence-based interventions implemented outside of the context of traditional research studies (i.e., tightly monitored and controlled conditions). The first aim of the study was to examine (a) the factors associated with predicted post-funding sustainability of interventions implemented as part of the PCCD Research-Based Delinquency and Violence Prevention Initiative and (b) the relationship between predicted post-funding sustainability and actual sustainability 1–3 years post-PCCD funding. Based on past research, we hypothesized that predicted post-funding sustainability would be associated with (a) indicators of community readiness; (b) strength of collaboration with local coalitions; and (c) the perceived quality of training and technical assistance.
The second aim of the study was to examine the actual post-funding sustainability of the interventions participating in the PCCD initiative. We first established overall levels of sustainability based on the number of years post-PCCD funding. As there were four different types of interventions (i.e., school-based prevention; community-based prevention; family-based prevention; family-based treatment), we next examined whether sustainability differed by intervention type. Additionally, given that the ability to secure funding for continued intervention implementation likely plays an important role in the sustainability, we also explored the relationship between current funding status and sustainability.
The third aim of this study was to examine the relationships between the hypothesized correlates of sustainability and actual post-PCCD funding sustainability. The hypothesized correlates included (a) indicators of community readiness; (b) strength of collaboration with local coalitions; (c) the perceived quality of training; and (d) sustainability planning.
Method
Data Collection
The data for this study were collected on three occasions between 2001 and 2007 in order to monitor the implementation and sustainability of the interventions delivered by agencies and schools participating in the Pennsylvania Commission on Crime and Delinquency’s (PCCD) Research-Based Delinquency and Violence Prevention. Given that data were collected to meet the needs of the funding agency rather than as part of a traditional university-based research project, the method of data collection and types of questions asked evolved with the needs of the funding agency and with the strengthening of the PCCD-university partnership. In 2001 representatives from all agencies/schools that had received funding since the start of the PCCD initiative in 1998 (N=50) were invited to attend an annual meeting to learn more about issues related to program implementation. Attendees (N=32; 64%) were asked to complete self-report questionnaires. Given the relative newness of the initiative, the questionnaire focused on barriers to program implementation and funded agencies/schools were not required to participate.
As the interventions began to move into the post-funding phase and the monitoring needs of PCCD increased, the survey was adapted to meet these increased needs (e.g., asking about sustainability planning) and existing measures were refined. By 2005, intervention representatives from all agencies/schools that had received funding since 1998 (N=115) were encouraged to complete a web-based questionnaire regarding their experiences implementing and sustaining their chosen interventions. Representatives of 84 agencies/schools (73%) completed the survey. This process was repeated again with a similar survey in 2007. Of 130 possible respondents, 99 (76%) completed the survey.
Sample
Predicted Post-funding Sustainability
The sample utilized in order to examine the correlates of predicted post-funding sustainability included agency/school intervention representatives who completed surveys in 2001 (N=32). The sample utilized in order to examine the relationship between predicted post-funding sustainability and actual sustainability 1–3 years post-funding included interventions (a) whose agency/school representatives completed surveys in 2001, (b) whose agency/school representatives completed surveys in 2005, and (c) that were at least 1 year post-PCCD funding at the time of survey completion in 2005. Thus, the maximum possible sample size was 32.
As a function of evolving data collection procedures and participant attrition, only 15 of the 32 interventions with data for 2001 also had data for 2005. Although 2007 sustainability data also were available, they were not used because they only increased the sample size by two, but substantially increased the range of time post-funding (from 1–3 years to 1–5 years). Of the 15 interventions with data for 2001 and 2005, three were family-based prevention programs (e.g., Families and Schools Together), five were family-based treatment programs (e.g., Multisystemic Therapy and Functional Family Therapy), four were school-based prevention programs (e.g., Promoting Alternative Thinking Strategies and LifeSkills Training), and three were community-based prevention programs (e.g., Big Brothers Big Sisters).
Due to the small sample size, we utilized Mann-Whitney U tests, the nonparametric alternative to t-tests, in order to determine whether interventions with data for 2001 only differed from interventions with data for 2001 and 2005 on predicted post-funding sustainability and the hypothesized correlates of predicted sustainability. We did not find any statistically significant differences.
Post-funding Sustainability
In order to maximize the sample size for the examination of post-funding sustainability and the correlates of post-funding sustainability, data from the 2005 and 2007 surveys were utilized. In the event that an intervention representative completed surveys in both 2005 and 2007, information from the most recent survey was used in the analysis. This sample included interventions (a) whose representatives completed surveys in 2005 and/or 2007 and (b) that were at least 1 year post-PCCD funding at the time of survey completion (N=67; 70% of all post-funding interventions). Of these 67 interventions, 15 were family-based prevention programs, 12 were family-based treatment programs, 23 were school-based prevention programs, and 17 were community-based prevention programs.
Measures Collected in 2001
Barriers to Program Implementation (2001)
As noted, in the early years of the PCCD initiative data were primarily collected on the process of and problems associated with program implementation. Thus, the hypothesized correlates of sustainability examined in this study were measured as barriers to program implementation. Participants were asked to rate the extent to which several factors were barriers to program implementation (0 = not a barrier, 1 = somewhat of a barrier, 2 = significant barrier). These included indicators of community readiness (i.e., the implementing agency; program staff; overall community support; support of community leaders; overall school support; support of school administrators; fit between program and other agency programs and goals), communication with the collaborative system (i.e., community coalitions), and training and technical assistance.
Predicted Post-funding Sustainability (2001)
Predicted sustainability was assessed with the question “How likely do you think it is that your program will continue beyond the PCCD funding period?” (1= highly unlikely, 2 = somewhat unlikely, 3 = somewhat likely, 4 = highly likely).
Measures Collected in 2005 and 2007
Indicators of Community Readiness
Indicators of community readiness included questions concerning the support of key leaders, as well as a four-item scale assessing readiness to implement the chosen intervention during the first year of PCCD funding. In order to assess support of key leaders, respondents were asked “How supportive or resistant was each of the following key parties: School Administration; Community Leaders; Agency Director(s)” (0 = very resistant; 4 = very supportive).
The Community Readiness scale (α=.77) included the items “We needed more time to prepare,” “We spent most of the first year planning instead of implementing the program,” “Because we were unprepared when funding began we fell behind schedule,” and “We were unprepared because schools or other implementing partners were not fully on board.” Responses were coded such that 0 = strongly agree and 4 = strongly disagree.
Collaboration with Community Coalitions
Both financial and instrumental collaborations with community coalitions were assessed. The Financial Collaboration scale included three items (α=.79), such as “To what extent does/did the coalition provide your program with resources?” The Instrumental Collaboration scale included five items (α=.79) such as “To what extent does/did the coalition promote evidence-based prevention programs in your community?” and “To what extent does/did the coalition give your program data/advice about meeting community needs?” Responses were coded such that 0 = not at all and 4 = a great deal.
Quality of Training
Quality of training was assessed with one item: “Rate the quality of initial training” (0 = very poor; 4 = excellent).
Sustainability Planning
Two types of sustainability planning were measured: planning for financial sustainability and planning to align with the goals of the agency/school. The survey stated, “The following questions ask you about certain aspects of sustainability planning. For each question, please indicate the degree to which you planned for sustainability during the period of PCCD funding.” The Financial Planning scale included three items (α=.55) such as “Plan to secure funds by applying for additional grants” and “Developed a fiscal plan outlining the funds needed to sustain the program.” The Alignment Planning scale included six items (α=.79) such as “Plans to discuss with local leaders how the program relates to the community’s overall prevention needs,” and “Plans to turn ownership of the program to the community, schools, or other organizations.” Responses were coded such that 0 = no discussion, 1 = limited discussion with no clear plan, 2 = discussion with tentative plan, 3 = discussion with firm plan, and 4 = executed plan.
Post-funding Sustainability
Post-funding sustainability was assessed with the question “To what extent is your program still operating?” (1 = it is not operating at all, 2 = it is operating at a reduced level, 3 = it is operating at the same level as the final year of PCCD funding, 4 = it is operating at a higher level than the final year of PCCD funding).
Current Funding Status
In order to determine current funding status, participants were asked “How is your program currently funded? Select all that apply” and were provided with three choices: “It is funded through additional PCCD grants; It is funded from other sources of temporary funding (e.g., donations, fundraising, other grants, etc.); It is funded by non-grant sources (school or agency budget line-item).”
Analysis Plan
Due to the small sample size in this study, Spearman’s Rank Order Correlation, the nonparametric equivalent to Pearson’s correlation, was utilized to examine the correlates of predicted and actual post-funding sustainability. Differences in sustainability relative to intervention type were examined with the Kruskal Wallis test, the nonparametric alternative to one-way ANOVA. Descriptive statistics were used to determine post-funding sustainability and current funding status.
Results
Correlates of Predicted Post-funding Sustainability
Descriptive statistics concerning predicted post-funding sustainability in 2001 and the hypothesized correlates of predicted post-funding sustainability are presented in Table 1. In 2001, intervention representatives predicted that it was somewhat to highly likely that their interventions would be sustained beyond the PCCD funding period (M=3.39, SD=0.72). On average the interventions were still being implemented in 2005 (1–3 years post-funding; M=2.60, SD=1.12), although it is important to note that some were being implemented at a reduced level compared to their implementation levels in the final year of PCCD funding.
Table 1.
Descriptive statistics for hypothesized correlates of predicted and actual post-funding sustainability
| N | Mean | SD | Possible range | |
|---|---|---|---|---|
| Predicted post-funding sustainability 2001 | 23 | 3.39 | 0.72 | 1–4 |
| Hypothesized correlates of predicted sustainability | ||||
| Implementing agency | 30 | 1.10 | 0.31 | 1–3 |
| Program staff | 26 | 1.31 | 0.47 | 1–3 |
| Overall community support | 25 | 1.12 | 0.33 | 1–3 |
| Support of community leaders | 32 | 1.28 | 0.46 | 1–3 |
| Overall school support | 24 | 1.46 | 0.59 | 1–3 |
| School administrator support | 28 | 1.39 | 0.57 | 1–3 |
| Alignment with existing goals | 31 | 1.10 | 0.30 | 1–3 |
| Comm. with collaborative system | 32 | 1.25 | 0.51 | 1–3 |
| Training and technical assistance | 27 | 1.41 | 0.57 | 1–3 |
| Post-funding sustainability 2005 | 15 | 2.60 | 1.12 | 1–4 |
| Post-funding sustainability 2005/2007 | 67 | 2.33 | 0.98 | 1–4 |
| Family-based prevention | 15 | 1.93 | 1.03 | 1–4 |
| Family-based treatment | 12 | 2.25 | 1.14 | 1–4 |
| School-based prevention | 23 | 2.39 | 0.78 | 1–4 |
| Community-based prevention | 17 | 2.65 | 1.00 | 1–4 |
| 1 year post-funding | 19 | 2.37 | 0.96 | 1–4 |
| 2 years post-funding | 14 | 2.36 | 1.00 | 1–4 |
| 3 years post-funding | 23 | 2.22 | 0.90 | 1–4 |
| 4 years post-funding | 2 | 3.00 | 1.41 | 1–4 |
| 5 years post-funding | 9 | 2.33 | 1.23 | 1–4 |
| Hypothesized correlates of post-funding sustainability | ||||
| Community leader support | 53 | 2.96 | 1.14 | 0–4 |
| School administrator support | 54 | 3.17 | 0.72 | 0–4 |
| Agency administrator support | 56 | 3.43 | 0.78 | 0–4 |
| Community readiness | 64 | 2.63 | 0.86 | 0–4 |
| Financial collaboration with comm. coalitions | 54 | 2.53 | 1.09 | 0–4 |
| Instrumental collaboration with comm. coalitions | 54 | 2.49 | 0.93 | 0–4 |
| Quality of training | 52 | 3.21 | 0.89 | 0–4 |
| Financial sustainability planning | 57 | 2.28 | 0.89 | 0–4 |
| Alignment planning | 57 | 2.16 | 0.86 | 0–4 |
Significant correlates of predicted post-funding sustainability (see Table 2) included program staff (rs=−0.47, p<.05), overall school support (rs=−0.88, p<.01), and school administrator support (rs=−0.71, p<.01). These findings suggest that perceiving that program staff, overall school support, and school administrator support are not barriers to intervention implementation is associated with expectations of post-funding sustainability. Additionally, the correlation between predicted post-funding sustainability in 2001 and actual post-funding sustainability in 2005 was statistically significant (rs=0.70, p<.05), indicating that the intervention representatives’ sustainability predictions in 2001 generally were accurate. All other hypothesized correlates of predicted post-funding sustainability were in the expected direction.
Table 2.
Spearman’s rank order correlations among predicted sustainability and hypothesized correlates
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1. Implementing Agency | 1 | ||||||||||
| 2. Program Staff | 0.28 | 1 | |||||||||
| 3. Overall Community Support | 0.34 | 0.04 | 1 | ||||||||
| 4. Support of Community Leaders | 0.27 | 0.28 | 0.54** | 1 | |||||||
| 5. Overall School Support | 0.32 | 0.21 | 0.51** | 0.07 | 1 | ||||||
| 6. School Administrator Support | 0.19 | 0.08 | 0.51** | 0.14 | 0.93*** | 1 | |||||
| 7. Alignment with Existing Goals | 1.00*** | 0.28 | 0.34 | 0.31 | 0.32 | 0.20 | 1 | ||||
| 8. Communication with the Collaborative System |
0.32* | −0.04 | 0.18 | 0.16 | 0.54*** | 0.48*** | 0.33* | 1 | |||
| 9. Training and Technical Assistance | 0.37* | 0.57*** | 0.36 | 0.28 | 0.36 | 0.30 | 0.34* | 0.22 | 1 | ||
| 10. Predicted Post-Funding Sustainability 2001 | −0.31 | −0.47** | −0.34 | −0.05 | −0.88*** | −0.71*** | −0.33 | −0.33 | −0.32 | 1 | |
| 11. Post-Funding Sustainability 2005 | 0.70** | 1 |
p<.10.
p<.05.
p<.01
Post-funding Sustainability
Descriptive statistics concerning post-funding sustainability of the 67 interventions whose representatives provided data in 2005 and/or 2007 (1–5 years post-funding) are presented in Table 1. Overall, the interventions were still being implemented in 2005/2007, but in some cases were being implemented at a reduced level compared to the final year of PCCD funding (M=2.33, SD=0.98). Although the mean level of post-funding sustainability was lower for family-based interventions compared to the other three types of interventions, there were not statistically significant differences in level of post-funding sustainability based on intervention type, χ2(3, 67)=5.00, p=.17.
Level of sustainability by number of years post-funding is presented in Table 1 (means) and Table 3 (percentages). Overall, 21% of interventions were no longer being implemented, 40% were being implemented at a reduced level, and 39% were being implemented at the same level or a higher level as in the final year of PCCD funding. Of the 19 interventions that were 1-year post-PCCD funding, 16% were not being implemented, 47% were being implemented at a reduced level, and 37% were being implemented at the same level or a higher level as the final year of post-PCCD funding. Of the 9 interventions that were 5 years post-PCCD funding, 33% were no longer being implemented, 22% were being implemented at a reduced level, and 45% were being implemented at the same level or a higher level as in the final year of PCCD funding.
Table 3.
Sustainability of interventions 1–5 years post-PCCD funding (percentages)
| 1 year post- funding (n=19) |
2 years post- funding (n=14) |
3 years post- funding (n=23) |
4 years post- funding (n=2) |
5 years post- funding (n=9) |
All post- funding (n=67) |
|
|---|---|---|---|---|---|---|
| To what extent is your program still operating? |
||||||
| Not operating at all | 16 | 21 | 22 | 0 | 33 | 21 |
| Operating at a reduced level | 47 | 36 | 44 | 50 | 22 | 40 |
| Operating at the same level or a higher level |
37 | 43 | 34 | 50 | 45 | 39 |
Current Funding Status
In addition to examining the long-term sustainability of interventions after the end of PCCD funding, another aim of this study was to determine the relationship between current funding status and sustainability. Seventy-seven percent of intervention representatives reported that their interventions were currently receiving funding from one or more sources. Nine percent were receiving funding through additional PCCD grants, 41% were receiving funding from temporary sources, and 41% were receiving funding from non-grant sources (i.e., a line-item in the agency/school budget). Not surprisingly, all intervention representatives who reported that their interventions were no longer operating also reported currently not receiving any funding for their interventions. Additionally, all but one representative who reported that their interventions were operating at a reduced level, the same level, or a higher level than the final year of PCCD funding reported currently receiving funding from at least one source.
Correlates of Post-funding Sustainability
Descriptive statistics for the hypothesized correlates of post-funding sustainability are presented in Table 1 and the correlations between post-funding sustainability and hypothesized correlates of post-funding sustainability are presented in Table 4. Of the nine hypothesized correlates of post-funding sustainability, only two were statistically significant: planning for financial sustainability (rs=0.27, p<.05) and planning relevant to aligning the intervention with the goals of the agency/school (rs=0.37, p<.01). Additionally, community leader support was significant at the trend level (rs=0.24, p=.08).
Table 4.
Correlations between post-funding sustainability and hypothesized predictors of post-funding sustainability
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
|---|---|---|---|---|---|---|---|---|---|---|
| 1. Community Leader Support | 1 | |||||||||
| 2. School Administrator Support | 0.64*** | 1 | ||||||||
| 3. Agency Administrator Support | 0.53*** | 0.22 | 1 | |||||||
| 4. Community Readiness | 0.27* | 0.36*** | 0.04 | 1 | ||||||
| 5. Financial Collaboration with Comm. Coalitions | 0.16 | −0.03 | 0.16 | −0.25* | 1 | |||||
| 6. Instrumental Collaboration with Comm. Coalitions | 0.12 | −0.07 | 0.22 | −0.25* | 0.85*** | 1 | ||||
| 7. Quality of Training | 0.13 | 0.21* | 0.06 | 0.01 | 0.13 | 0.12 | 1 | |||
| 8. Financial Sustainability Planning | 0.24* | 0.21 | 0.25* | 0.17 | 0.10 | 0.12 | 0.35** | 1 | ||
| 9. Alignment Planning | 0.31** | 0.32** | 0.25* | 0.43*** | 0.00 | −0.04 | 0.19 | 0.58*** | 1 | |
| 10. Post-Funding Sustainability | 0.24* | 0.12 | 0.07 | −0.03 | 0.04 | 0.12 | 0.10 | 0.27** | 0.37*** | 1 |
p<.10.
p<.05.
p<.01
Discussion
The present study examined the long-term, post-funding sustainability of evidence-based delinquency, substance abuse, and violence interventions implemented by community agencies and schools under real-world conditions (i.e., apart from a traditional research study), as well as the correlates of predicted and actual post-funding sustainability. The interventions included in this study had not received funding from the original funding agency (the Pennsylvania Commission on Crime and Delinquency [PCCD]) for at least 1 year and up to 5 years.
Long-term, Post-funding Sustainability
A key finding in this study was that even 5 years post-PCCD funding the majority of interventions still were being implemented, albeit at reduced levels in some cases. Additionally, 77% of all interventions were able to obtain permanent and/or temporary funding after the 4-year PCCD funding period ended, and only a small portion of interventions received additional funding from PCCD. These findings suggest that it is possible to sustain the funding and implementation of evidence-based interventions over the long-term but also raise many important questions.
Although it is often assumed that insufficient funding leads to a low sustainability, it is equally plausible that an agency/school that is dissatisfied with the characteristics and results of a given intervention would be less likely to pursue funding for that intervention. Thus, additional research is needed to understand (a) the process by which agencies/schools make decisions regarding the priority of implementing evidence-based interventions (e.g., the informal cost-benefit analysis that guides decisions to apply or not apply for additional grants; formal and informal methods of assessing intervention effectiveness) and (b) how prevention scientists can help guide and support these efforts in order to promote the diffusion and long-term sustainability of evidence-based programs that meet community needs.
It is important to note that a large percentage of interventions in this study were operating at a reduced level post-PCCD funding and different methods of reducing implementation (e.g., eliminating components of an intervention; reducing the number of sites implementing an intervention) may have different implications for program effectiveness and public health. For example, eliminating intervention components may lead to no outcomes or worse, iatrogenic effects, whereas reducing the number of sites may result in a positive but limited public health impact. Therefore, additional research is needed in order to better understand the ways in which implementing agencies modify and reduce intervention implementation over time. For example, are decisions about eliminating intervention components made at the level of the agency or the level of the facilitator, and are these decisions made by the individuals who received training in the intervention logic model? All in all, these questions highlight the need for additional research on the factors that motivate the long-term sustainability or discontinuation of evidence-based interventions.
Program Differences in Post-funding Sustainability
As different types of interventions may have different factors that influence the availability of funding and long-term sustainability, in this study we also examined differences in sustainability based on intervention type. Although family-based prevention programs appeared to have a lower level of post-funding sustainability compared to the other three types of interventions (i.e., family-based treatment, community-based prevention, school-based prevention), these differences were not statistically significant. These results must be considered, however, in light of the small sample size and limited power to detect effects. Given the known difficulty of recruiting families into preventive interventions, family-based preventive interventions may in fact have lower sustainability than other types of interventions. Additional large-scale research is needed that examines the factors impeding or facilitating the long-term sustainability of interventions, as well as whether these factors are different for different types of programs.
Correlates of Predicted Post-funding Sustainability
In addition to describing the long-term sustainability of evidence-based interventions, we also examined the correlates of predicted and actual post-funding sustainability. Perceiving that program staff, overall school support, and school administrator support were not barriers to program implementation was associated with predicted post-funding sustainability. As suggested in the community readiness literature (Altman 1995; Elliott and Mihalic 2004; Johnson et al. 2004), these findings also indicate that securing and maintaining support for interventions may increase the likelihood of long-term sustainability. Therefore, future research should focus on developing strategies to effectively engage school and community members and garner support for evidence-based interventions.
Characteristics of the implementing agency, alignment with existing agency goals, overall community support, support of community leaders, communication with the collaborative system, and training and technical assistance were not significantly associated with predicted post-funding sustainability, although all relationships were in the expected direction. Although it is possible that these variables truly are not associated with long-term, post-funding sustainability, it is equally plausible that the small sample size resulted in inadequate power to detect relationships. Additional empirical research is needed to definitively establish common barriers to successful intervention implementation, as this information can guide the development of barrier-specific solutions that may increase both implementation and sustainability.
The fact that predicted post-funding sustainability was strongly associated with actual post-funding sustainability 4 years later suggests that agencies/schools are aware early in the implementation process of the future of their interventions. Thus, taking sustainability predictions seriously and working with implementing agencies/schools to change the factors that influence these predictions may serve to improve the implementation and sustainability of evidence-based interventions.
Correlates of Post-funding Sustainability
Although our measurement of the correlates of predicted and actual post-PCCD funding sustainability differed as a result of the evolution of the PCCD initiative, several of the constructs were similar. School administrator support was not significantly correlated with actual post-funding sustainability, but the relationship between sustainability and community leader support was significant at the trend level. Additionally, planning to secure additional funding and planning relevant to aligning the intervention with the goals of the agency/school were significant correlates of post-funding sustainability. As with predicted post-funding sustainability, the five other hypothesized correlates of actual post-funding sustainability (agency administrator support, community readiness, financial and instrumental collaborations with community coalitions, quality of training) were not statistically significant.
These results suggest that financial and alignment sustainability planning are as important, if not more important, than the factors that are more commonly discussed in the sustainability literature. Therefore, incorporating information concerning sustainability planning (e.g., possible funding sources, ways to understand and align with the goals and needs of the community) into initial and ongoing training and technical assistance may be one avenue for increasing sustainability. Alternatively, it may be necessary for funders to require sustainability training and planning as a condition of receiving financial support.
Limitations
The real-world nature of this study defines its unique contributions to the field of prevention science yet also presents significant limitations that should be taken into consideration when interpreting the results. First, the sample size was notably small from a statistical standpoint. As a result, the absence of statistical significance may have been the result of Type II error. Larger initiatives would overcome this limitation, but also would be financially and logistically challenging.
Second, the implementing agencies/schools were not required to complete surveys in the early years of the initiative, thus limiting generalizability. As a result, there may be significant differences between the agencies/schools that did and did not participate in the data collection. This limitation was addressed to some degree in later years through the use of web-based rather than paper-based surveys.
A third limitation concerns the measurement of key constructs. As noted, the nature of the initiative as a diffusion project rather than a traditional research study led to changes in the measurement of constructs over time. In the early years of the initiative the focus was on implementation, whereas in the later years the focus broadened to also include sustainability. Thus, it was not possible to use the same measures for the examination of the correlates of predicted post-funding sustainability and actual post-funding sustainability. Additionally, since self-report instruments were used, it is possible that social desirability may have led sites to under-report barriers to implementation and over-report predicted and actual sustainability. Future research on sustainability could be improved by including independent assessments of implementation fidelity and sustainability.
Further, self-reports and independent assessments of sustainability could be strengthened by including a more precise measure of sustainability. In the present study, it was not possible to ascertain what aspects of the intervention were being changed when an agency/school representative reported that the intervention was being implemented at a reduced level. For example, limiting the number of schools or families receiving an intervention versus leaving out important components may have different implications for intervention effectiveness and long-term sustainability. As the PCCD initiative is ongoing, recent refinements of the survey items will shed light on some of these issues.
Conclusions
Despite the abovementioned limitations, this study makes several unique contributions to the study of the sustainability of evidence-based interventions under real-world conditions. First, this study suggests that sustainability is possible and funding is often obtainable. Additionally, this study suggests that obtaining and maintaining school support for evidence-based interventions should be a priority, as school support is associated with predicted post-funding sustainability, and predicted sustainability is associated with actual post-funding sustainability. Finally, this study suggests that sustainability planning is a key factor in post-funding sustainability. Future research is needed to replicate these findings and to further build the body of research on the sustainability of evidence-based interventions.
Acknowledgements
This research was funded by grants from the Pennsylvania Commission on Crime and Delinquency and NIH Award Number F31 DA024535. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Drug Abuse or the National Institutes of Health.
Contributor Information
Melissa K. Tibbits, Email: mtibbits@unmc.edu, Department of Health Promotion, Social and Behavioral Health, College of Public Health, University of Nebraska Medical Center, 986075 Nebraska Medical Center, Omaha, NE 68198-6075, USA.
Brian K. Bumbarger, Prevention Research Center, The Pennsylvania State University, University Park, PA, USA
Sandee J. Kyler, Prevention Research Center, The Pennsylvania State University, University Park, PA, USA
Daniel F. Perkins, Prevention Research Center, The Pennsylvania State University, University Park, PA, USA
References
- Adelman HS, Taylor L. Moving prevention from the fringes into the fabric of school improvement. Journal of Educational & Psychological Consultation. 2000;11:7–36. [Google Scholar]
- Altman DG. Sustaining interventions in community systems: On the relationship between researchers and communities. Health Psychology. 1995;14:526–536. doi: 10.1037//0278-6133.14.6.526. [DOI] [PubMed] [Google Scholar]
- Chilenski SM, Greenberg MT, Feinberg ME. Community readiness as a multidimensional construct. Journal of Community Psychology. Special Issue: Exploring the Intersection of Organization Studies and Community Psychology. 2007;35:347–368. doi: 10.1002/jcop.20152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Domitrovich CE, Greenberg MT. The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation. 2000;11:193–222. [Google Scholar]
- Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research. 2003;18:237–256. doi: 10.1093/her/18.2.237. [DOI] [PubMed] [Google Scholar]
- Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prevention Science. 2004;5:47–53. doi: 10.1023/b:prev.0000013981.28071.52. [DOI] [PubMed] [Google Scholar]
- Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science. 2003;4:1–14. doi: 10.1023/a:1021777109369. [DOI] [PubMed] [Google Scholar]
- Feinberg ME, Greenberg MT, Osgood DW. Readiness, functioning, and perceived effectiveness in community prevention coalitions: A study of Communities That Care. American Journal of Community Psychology. 2004;33:163–176. doi: 10.1023/b:ajcp.0000027003.75394.2b. [DOI] [PubMed] [Google Scholar]
- Gager PJ, Elias MJ. Implementing prevention programs in high-risk environments: Application of the resiliency paradigm. American Journal of Orthopsychiatry. 1997;67:363–373. doi: 10.1037/h0080239. [DOI] [PubMed] [Google Scholar]
- Gingiss PM, Roberts-Gray C, Boerm M. Bridge-it: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science. 2006;7:197–207. doi: 10.1007/s11121-006-0038-1. [DOI] [PubMed] [Google Scholar]
- Gomez BJ, Greenberg MT, Feinberg ME. Sustainability of community coalitions: An evaluation of Communities That Care. Prevention Science. 2005;6:199–202. doi: 10.1007/s11121-005-0003-4. [DOI] [PubMed] [Google Scholar]
- Gottfredson DC, Gottfredson GD. Quality of school based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency. 2002;39:3–35. [Google Scholar]
- Greenberg MT. Current and future challenges in school-based prevention: The researcher perspective. Prevention Science. 2004;5:5–13. doi: 10.1023/b:prev.0000013976.84939.55. [DOI] [PubMed] [Google Scholar]
- Hawkins JD, Catalano RF, Arthur MW. Promoting science-based prevention in communities. Addictive Behaviors. 2002;27:951–976. doi: 10.1016/s0306-4603(02)00298-8. [DOI] [PubMed] [Google Scholar]
- Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation and Program Planning. 2004;27:135–149. [Google Scholar]
- Mancini JA, Marek LI. Sustaining community-based programs for families: Conceptualization and measurement. Family Relations: Interdisciplinary Journal of Applied Family Studies. 2004;53:339–347. [Google Scholar]
- Payne AA, Gottfredson DC, Gottfredson GD. School predictors of the intensity of implementation of school-based prevention programs: Results from a national study. Prevention Science. 2006;7:225–237. doi: 10.1007/s11121-006-0029-2. [DOI] [PubMed] [Google Scholar]
- Pentz MA. Institutionalizing community-based prevention through policy change. Journal of Community Psychology. Special Issue: Bridging the Gap Between Research and Practice in Community-based Substance Abuse Prevention. 2000;28:257–270. [Google Scholar]
- Rohrbach L, Grana R, Sussman S, Valente TW. Type II translation: Transporting prevention interventions from research to real-world settings. Evaluation & The Health Professions. 2006;29:302–333. doi: 10.1177/0163278706290408. [DOI] [PubMed] [Google Scholar]
- Rohrbach L, Ringwalt CL, Ennett ST, Vincus AA. Factors associated with adoption of evidence-based substance use prevention curricula in US school districts. Health Education Research. 2005;20:514–526. doi: 10.1093/her/cyh008. [DOI] [PubMed] [Google Scholar]
- Scheirer M. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Americal Journal of Evaluation. 2005;26:320–347. [Google Scholar]
- Spoth R, Greenberg M, Bierman K, Redmond C. PROSPER Community-University Partnership Model for Public Education Systems: Capacity-building for evidence-based, competence-building prevention. Prevention Science. 2004;5:31–39. doi: 10.1023/b:prev.0000013979.52796.8b. [DOI] [PubMed] [Google Scholar]
- Swisher JD. Sustainability of prevention. Addictive Behaviors. 2000;25:965–973. doi: 10.1016/s0306-4603(00)00124-6. [DOI] [PubMed] [Google Scholar]