A qualitative evaluation of the 2005–2011 National Academic Centers of Excellence in Youth Violence Prevention Program

Abstract

The Centers for Disease Control and Prevention’s Division of Violence Prevention (DVP) funded eight National Academic Centers of Excellence (ACEs) in Youth Violence Prevention from 2005 to 2010 and two Urban Partnership Academic Centers of Excellence (UPACEs) in Youth Violence Prevention from 2006 to 2011. The ACEs and UPACEs constitute DVP’s 2005–2011 ACE Program. ACE Program goals include partnering with communities to promote youth violence (YV) prevention and fostering connections between research and community practice. This article describes a qualitative evaluation of the 2005–2011 ACE Program using an innovative approach for collecting and analyzing data from multiple large research centers via a web-based Information System (ACE-IS). The ACE-IS was established as an efficient mechanism to collect and document ACE research and programmatic activities. Performance indicators for the ACE Program were established in an ACE Program logic model. Data on performance indicators were collected through the ACE-IS biannually. Data assessed Centers’ ability to develop, implement, and evaluate YV prevention activities. Performance indicator data demonstrate substantial progress on Centers’ research in YV risk and protective factors, community partnerships, and other accomplishments. Findings provide important lessons learned, illustrate progress made by the Centers, and point to new directions for YV prevention research and programmatic efforts.

1. Introduction

Youth violence is a major public health problem that results in significant negative impact on victims and communities. In 2011, more than 4700 youth between the ages of 10 and 24 died by homicide, making homicide the third leading cause of death for this age group (Centers for Disease Control and Prevention [CDC], 2011). The negative consequences of youth violence are experienced most directly by individuals and families who may experience fear, injuries, and death caused by violence (CDC, 2011; Mercy, Butchart, Farrington, & Cerda, 2002). Yet, communities and society also experience the negative effects of violence, such as increased cost of health care, reduced productivity, diminished property values, and negative impacts on social cohesion and collective efficacy (Mercy et al., 2002; Sampson, Raudenbush, & Earls, 1997). Research in the field of youth violence has advanced our understanding of the context in which it occurs, its risk and protective factors, and its consequences. Additionally, empirical research has guided communities toward implementing promising and effective strategies to prevent violence before it starts (David-Ferdon and Simon, 2014). Partnerships between researchers and communities are essential to ensuring that the best available science informs violence prevention efforts in communities. Community–research partnerships can provide reciprocally beneficial relationships and inform researchers of community needs while simultaneously educating community members about the value of implementing evidence-based approaches to violence prevention (Massetti and Vivolo, 2010).

CDC’s Division of Violence Prevention (DVP) has funded the National Academic Centers of Excellence in Youth Violence Prevention (ACEs) Program since 2000 (now referred to as the National Centers for the Prevention of Youth Violence, YVPCs). The ACEs use a unique, multidisciplinary approach to research youth violence prevention strategies, collect and analyze surveillance data, and foster relationships with community partners to develop, implement, and evaluate prevention programs. Collaborations between universities, health departments, communities, and community-based organizations are developed to empower communities to address the problem of youth violence by building the necessary infrastructure to implement local programming.

Since the inception of the ACE Program, 16 research universities received ACE funding over three rounds of funding. In the first round (2000–2005), CDC funded ten ACEs. CDC funded eight ACEs from 2005 to 2010 and two Urban Partnership Academic Centers of Excellence in Youth Violence Prevention (UPACEs) from 2006 to 2011 in the second round. The ACEs and UPACEs together constituted DVP’s 2005–2011 ACE Program. Currently, six academic institutions are receiving funding from 2010/2011 to 2015/2016 as the CDC’s National Centers for the Prevention of Youth Violence. The objectives of the ACE Program have progressed with each new funding cycle, with the first and second funding cycles (2000–2005 and 2005–2011, respectively) focusing on building the infrastructure required to implement and evaluate youth violence prevention strategies, and the most recent funding cycle (2010–2016) aiming to evaluate the effectiveness of such strategies at reducing rates of youth violence in high-risk communities. Some of the goals of the ACE Program have remained constant, including the goals to serve as models for youth violence prevention; support the translation and application of research findings into communities; and enhance academic and community capacity and partnerships to prevent violence. In each round of funding, the Centers in the ACE Program have been expected to work with key stakeholders, youth, and community organizations, among others, to identify areas of need in their defined communities and projects and programs to implement to address those needs.

To more thoroughly understand how the Centers meet the goals and objectives of the ACE Program, and disseminate lessons learned to the larger youth violence prevention community, CDC tracks the ACE Program’s accomplishments over time using qualitative and quantitative evaluation methods. The current report presents findings from the evaluation of the 2005–2011 ACE Program. The emphasis of the ACE Program in this round of funding was to advance research in youth violence risk and protective factors and further community-research partnerships to build community capacity to prevent violence rather than to demonstrate reductions in youth violence in the target communities. The qualitative evaluation of work completed by the 2005–2011 ACE Program grantees was conducted at the end of their funding cycle and examined the extent to which they fulfilled major performance indicators outlined in the CDC’s 2005 Request for Applications (RFA). The performance indicators were developed to ensure that the inputs and activities of the Centers that were necessary to achieve expected outputs and outcomes would be implemented. The fulfillment of these performance indicators would suggest the successful development of Center infrastructures that can enable communities’ success in using evidence-based approaches and reducing rates of youth violence.

Using data regarding major performance indicators, the primary goal of this qualitative evaluation was to examine whether the 2005–2011 ACEs and UPACEs met objectives prescribed in the RFA. These objectives included tasks such as tracking the distribution of youth violence in a defined community; building the scientific infrastructure necessary to support the development and widespread application of effective youth violence interventions; promoting interdisciplinary research strategies to address youth violence in a defined community; fostering collaboration between researchers and communities by bringing together individuals with diverse perspectives; and mobilizing and empowering communities to address youth violence.

A secondary goal of this evaluation process was to assess the novel data collection and review process used to conduct this qualitative evaluation. The ACE-Information System (ACE-IS; described below) was used to collect and aggregate a large amount of qualitative and quantitative data submitted by the Centers. This process and the utility of an online information system for data collection are described and critiqued below.

1.1. Conceptual framework of the ACE Program

The conceptual framework for the 2005–2011 ACE Program was developed to describe the orientation of the ACE Program, its activities, and the outcomes it was expected to achieve (Vivolo, Matjasko, & Massetti, 2011). The elements of the ACE logic model (see Fig. 1) and its linkages are consistent with the Congressional language authorizing the ACE Program and CDC’s National Center for Injury Prevention and Control (NCIPC) research priorities. This model served as a planning mechanism and guided ACE Program activities during the 5-year grant period and identifies the inputs, activities, outputs, and outcomes common to Centers funded from 2005 to 2011. Finally, the model illustrates the expected relationships among these components.

Fig. 1.

ACE Program logic model.

The logic model (which was published in the ACE RFA) included a set of 14 performance indicators to track and measure the progress of the ACE Program and each Center in meeting its goals. The performance indicators provided a set of key determinants for the evaluation of the ACE Program’s impact on improved practice and policy, as well as the Centers’ impact on reduction of risk factors or promotion of protective factors over time. Together, the ACE Program logic model and performance indicators served as the basis for conducting the ACE Program evaluation.

The purpose of this manuscript is to describe aggregate data compiled across the eight ACEs and two UPACEs demonstrating their progress on ACE performance indicators perceived as critical for the development of a Center that could successfully accomplish the objectives prescribed in the RFA. Data related to the indicators highlight the successes, outcomes, and accomplishments for each Center since the beginning of the 2005 funding cycle and demonstrate progress in building community capacity to address the impact on youth violence rates, reduce risk factors, and promote protective factors relative to youth violence prevention efforts in communities.

2. Method

The ACE Program evaluation was conducted by CDC staff at the end of the 2005–2011 ACE Program funding cycle. Data were compiled on the extent to which the logic model inputs were used to shape the ACE Program, the activities that were undertaken by the Centers in the ACE Program, the quality of the activities, and the specific outcomes of the ACE Program.

The ACE-Information System (ACE-IS) is a CDC-hosted online tool for data collection and was used by the Centers in the ACE Program over the course of their funding for reporting purposes (see Fig. 2 for a screenshot of the ACE-IS report format; data entry pages appear similar but include check boxes, radio buttons, and dropdown menus for item selection). Using the key performance indicators as a guide, the Centers entered data about their projects and activities, workplans, and components of the ACE logic model into the ACE-IS two times per year from 2007 to 2010. Prior to administration of the ACE-IS in 2006, CDC staff received Office of Management and Budget approval for data collection (OMB # 0920-0767). The ACE-IS included questions to assess progress related to each of the 14 original performance indicators and served as the primary source of data collection for the ACE Program evaluation.

Fig. 2.

Screenshot of the University of Hawaii ACE’s ACE-IS community committee information page.

2.1. Procedure

ACE Principal Investigators (PIs) and staff members received an ACE-IS Instruction Manual as well as in-person training on its use. The manual provided detailed instructions on a standardized method of ACE-IS data entry and the type of data each ACE-IS item intended to collect. CDC staff provided technical assistance to the Centers on data entry as needed throughout the 2005–2011 funding cycle. As part of the data quality review procedures, prior to data abstraction from the ACE-IS CDC staff reviewed data entries for each Center and documented sections with missing data. PIs were notified of the areas within the ACE-IS that required attention and were provided a set of guidelines for inputting missing data and highlighting existing data relevant to the performance indicators. To ensure consistent data abstraction, a protocol for abstracting ACE Program evaluation data from the ACE-IS was developed. A comprehensive coding guide to accompany the protocol for data abstraction was also developed (available upon request). Using the coding guide, data were abstracted by three CDC staff members who were familiar with the ACE-IS and the ACE Program. Each coding sheet was double coded to ensure accuracy and quality control. Because data were not easily aggregated across sites within the ACE-IS, abstracted data were then entered into an Excel spreadsheet to facilitate aggregation and interpretation.

Using the data abstraction sheet, counts of data regarding ACE community committees, Center and project partners, community mobilization plans, types of ACE projects, surveillance sources, and dissemination of research and results were entered into an Excel spreadsheet. Responses to dichotomous items (e.g., whether a training plan is uploaded in the ACE-IS) were coded (Yes, 1; No, 0). An aggregate spreadsheet of data was compiled by summing the responses of all eight ACEs and two UPACEs.

2.2. Measures

The list of 14 performance indicators referenced in the RFA was revised during the ACE Program evaluation process to address overlap between indicators, difficulties in obtaining meaningful data from the Centers, and to better reflect the data that were collected through the ACE-IS (Table 1). Ultimately, nine of the original 14 performance indicators had sufficient data in the ACE-IS for use in the ACE Program evaluation. Only two of the original indicators were eliminated due to difficulty in obtaining accurate or complete data; other indicators were combined or eliminated because of similarity between indicators. The ACE-IS measures that captured information for each of the program evaluation indicators are identified below. In some cases, multiple items were used to assess performance for a particular indicator.

Table 1.

ACE Program performance indicators.

Original indicator in logic model	Status of indicator for program evaluation
Evidence of ACE community committee participation in the determination of ACE violence prevention priorities	Maintained as is
Level of ACE community committee members’ satisfaction with participation	Eliminated due to difficulty in obtaining accurate operationalization and assessment data
Evidence of establishment of partnerships and maintenance of local/project partnerships	Maintained as is
Evidence of establishment of a community mobilization plan	Maintained as is
Establishment of a research agenda	Combined with indicator #11 and expanded to indicate establishment of projects and outcome/impact of effectiveness in the community
Evidence of community improvements in the ability to monitor and describe youth violence	Maintained as is
Extent to which the research portfolio is contributing to new methods of study, understandings of, or ways to prevent youth violence	Eliminated due to overlap with indicator #5
Evidence that the ACE Center is mobilizing the community to implement evidence-based strategies or promising programs	Maintained as is
Evidence of a communication and dissemination plan, developed with input from key partners	Modified to: evidence of a communication and dissemination plan
Evidence of producing and disseminating research findings through peer-reviewed publications, and educational or technical materials	Combined with indicator #12
Evidence of a plan for training researchers, practitioners and community members	Combined with indicator #5
Extent to which center activities and evidence-based strategies have been translated into the outcomes listed	Maintained as is
Extent to which researchers, practitioners, and community members have been trained, mentored, or provided technical assistance in youth violence prevention	Combined with indicators #5 and #11
Evidence of new grants, contracts, or other resources awarded to the ACE Center or its partners	Eliminated due to difficulty in obtaining accurate data

1. Evidence of ACE community committee participation in the determination of Center’s violence prevention priorities

Centers provided information about each of the community committees affiliated with their program. Data included the number of members; the frequency with which they met; the sectors represented in the committee (e.g., community-based organizations, criminal justice, etc.); how the committee was structured, governed, and whether a documented decision-making process existed; and the Center activities in which the committee was engaged (e.g., administration and infrastructure, outreach and education, establishing project goals and objectives, establishing research priorities, etc.).

Data for these items were first recorded for each Center separately and then aggregated across all Centers. For instance, one Center may have reported that it was affiliated with one community committee, which had 150 members, met once a month, and was composed of representatives from community based organizations, the criminal justice system, and the local educational system. These data were then aggregated across all Centers to demonstrate the variety and types of community committees in which Centers took part.

2. Evidence of establishment of partnerships and maintenance of local/project partnerships

Centers indicated the number of new partnerships established each year, the sectors represented by the partnering organizations (e.g., community-based organizations, criminal justice, etc.), and how partners were engaged in the Centers’ work (e.g., assists in conceptualizing the project, selecting the project, etc.). Maintenance of partnerships was assessed through updates to the ACE-IS regarding the continued engagement of partners. While partners often were engaged in community committee activities, they were not required to participate as members of the community committee. Partnering organizations often held similar roles as the community commit-tee in the Centers’ work, but were typically engaged in the work in a more specific, as opposed to a broad, way.

3. Evidence of establishment of a community mobilization plan

Coders assessed whether Centers had uploaded a community mobilization plan in the ACE-IS that described the activities in which the community committee was involved (e.g., identifying priorities, developing overall plan/agenda, etc.).

4. Evidence that the Center is mobilizing the community to implement evidence-based strategies or promising programs

Centers were expected to include information regarding the activities they conducted enabling their targeted community to implement evidence-based strategies or promising programs. For instance, Centers detailed their training, technical assistance, mentoring, dissemination, and mobilization efforts. Further, for each specific Center project, Centers were expected to include whether their projects increased the community capacity to develop, implement, and evaluate evidence-based youth violence prevention programs.

5 and 6. Evidence of complete ACE Center Plans and are search agenda

Centers were expected to develop comprehensive plans including a logic model, evaluation plan, training plan, and research agenda. Because Centers implemented more than one project, coders assessed the number of surveillance, training, outreach, and research projects implemented by each Center, and the focus of each project (i.e., intervention research, implementation research, mentoring, communication and dissemination, community mobilization, etc.).

7. Establishment of projects in a high-risk community and outcome/impact of effectiveness in the community

Centers were expected to describe their defined community(ies) by listing the name of the community, size of community, racial/ethnic breakdown, sex breakdown, setting and density (i.e., city, rural, etc.), and other characteristics such as the percentage of the defined community that is below poverty, the percentage of the defined community that speaks a language other than English as the primary language, the percentage of the defined community that has immigrated in the last 10 years, the percentage of the defined community that is unemployed, and the percentage of students in the defined community that receive free or reduced cost school lunch.

For each of their projects, Centers were required to provide a brief description of the study purpose and goals, the methods, a planned project period, and any findings (i.e., impact/outcome in the community or new methodology for advancement of youth violence prevention, monitor trends in the burden of youth violence). For surveillance projects, they identified the data sources included in surveillance systems (e.g., 911 calls, police data, etc.), the intended uses of the data/data system (e.g., describe the trajectory of violence, etc.), policy and practice improvements or enhancements, and the outcome of the project.

For research projects, Centers were expected to identify the intended outcome of the project (i.e., risk and protective factors at the individual, family, peer/school, and neighborhood/community level) and whether their project resulted in the reduction of risk factors and/or the increase in protective factors. However, the ACE Program RFA indicated that examining the impact of youth violence prevention strategies was an optional goal for grantees and was not required.

For outreach and training projects, Centers described the intended audience, the instructional methods/format used, and the changes that occurred as a result of training/technical assistance/mentoring activities in attitudes/beliefs, knowledge, skills, and/or practices/behaviors.

8. Evidence of the community’s ability to monitor and describe youth violence

For this indicator, Centers identified the types of data they used to monitor youth violence in their defined community (e.g., ambulance records, coroner data, etc.) and the rates of homicides, sexual assaults, ER visits, etc. within the defined community. They also described how the Center attempted to improve their community’s ability to collect, understand, and use available youth violence data.

9. Evidence of a communication and dissemination plan

This indicator assessed whether a communication and dissemination plan was uploaded in the ACE IS, the goals and objectives outlined in the communication and dissemination plan, the primary audience and venue and the communication strategies for each, and the products developed as part of the Center (e.g., book chapters, evaluation reports, etc.).

3. Results

The ACE Program evaluation highlights the key accomplishments of the eight ACEs and two UPACEs and their efforts to address the performance indicators for the 2005–2011 funding cycle. The following data represent results aggregated across all Centers and are presented in sections representing each performance indicator.

1. Evidence of ACE Community Committee participation in the determination of violence prevention priorities

The ACE Program was associated with 30 community committees, composed of 1059 members. Sectors represented by community committee members varied widely and included community-based organizations, criminal justice, education, health, business, foundations, government agencies, and non-profit organizations. Centers reported that committee members were involved in many aspects of the ACE Program, including collaborations and partnerships, establishing research project goals and objectives, evaluation, infrastructure, surveillance, core research, small studies, seed projects, and outreach and education (e.g., communication and dissemination, community mobilization, training, technical assistance, and mentoring).

Centers had the option to start a new community committee or to join an existing one. One example of a Center joining an existing community committee is the Columbia Center for Youth Violence Prevention (CCYVP). They joined the UNIDOS Coalition, an established community coalition that focused on individual, family, block, organizational, neighborhood, and built environment-level activities. Forty members and three CCYVP project partners comprised this committee, with representation spanning community-based and non-profit organizations and educational and health sectors. CCYVP researchers held regular meetings with their partners and routinely involved them in many aspects of the CCYVP projects, from conceptualizing the projects to conducting or implementing activities to communicating and disseminating project results.

2. Establishment of partnerships and maintenance of local/project partnerships

Data about ACE Program partnerships provide insight regarding perspectives represented by various partners and the activities in which partners were involved. Center-level partners were partnerships the Centers engaged in as a whole, while project-level partnerships comprised collaborations specific to project goals and objectives. Across the eight ACEs and two UPACEs, 93 Center-level partners and 129 project-level partners were reported and were maintained over time. Only local partners were counted in this tally, as much of the focus of ACE Program research was on community-based participatory research and national partners tended not to participate substantially in ACE Program research and other activities. For example, a state health department may serve as a Center-level partner engaged in multiple aspects of the Center, including identification of research priorities, conceptualization of research projects, and administrative and infrastructure activities, such as providing surveillance data for evaluation purposes. Conversely, project-level partnerships are more focused in nature and may serve to develop, plan and implement specific projects, such as school-based violence prevention strategies. Project-level partners are typically very engaged in research activities, as evidenced by their help to recruit research participants, provide space for program implementation, and collect data, among other activities.

3. Evidence of establishment of a community mobilization plan

Each ACE and UPACE established a community mobilization plan which described the manner in which Center and project partners and community members participated in Center activities. In response to the items that assessed fulfillment of this performance indicator, the Centers reported the types of activities in which their Center and project partners were involved. Partners were involved in a wide range of activities and were not limited to participating in only one activity per partner. The majority of partners (70%) were involved in outreach and education support activities. Partners were also involved in providing administrative and infrastructure support (58%), conducting or implementing project activities (44%), communicating or disseminating project information (39%), and conceptualizing projects (36%). Other activities in which partners participated included developing and planning projects (53%), establishing project goals and objectives (51%), evaluating projects (39%), selecting projects (35%), and providing surveillance and research support (35%).

One example of a community mobilization plan developed by the Asian/Pacific Islander Youth Violence Prevention Center (APIYVPC) at the University of Hawai‘i at Mānoa involved five main goals: (1) to assist in the facilitation of a strategic planning process for a collaborative one-stop youth and family center within Waimānalo, their defined community; (2) to continue to build community capacity to address youth violence and related concerns through the Waimānalo Togetherness Group; (3) to facilitate the strengthening and formalizing of school–community partnerships for youth violence prevention; (4) to develop youth leadership for violence prevention; and (5) to disseminate a community empowerment approach to youth violence mobilization efforts of the APIYVPC.

4. Evidence that the Centers mobilized the community to implement evidence-based strategies or promising programs

Centers mobilized their communities using many different strategies. Eight Centers employed training and mentoring sessions for community members and partners. They also engaged their communities by involving them in project activities, such as those listed above (i.e., conceptualizing the project, selecting the project, etc.). The activities of the Academic Center of Excellence in Youth Violence Prevention at the University of California Riverside (ACE-UCR) are examples of how Centers’ mobilized their communities. The ACE-UCR worked with local leadership, including the Mayor of Riverside, to convene representatives from their community to help guide prevention efforts by identifying community issues and high risk youth. ACE-UCR staff mobilized the community with respect to the Center’s program planning and implementation phases by encouraging regular participation, data sharing, conducting focus groups to identify issues and prevention strategies, and providing technical assistance and training on evidence-based practices.

5. and 6. Evidence of complete ACE Center plans and a research agenda

Centers’ plans were communicated by developing research agendas, training plans, communication/dissemination plans, logic models, and community mobilization plans. Each Center developed an agenda with projects, goals, and objectives specifically designed to address the needs of its defined community. Across all 10 Centers, a total of 99 projects were developed and implemented. Projects were specified as surveillance, research, training, or outreach (Table 2). A total of 16 surveillance projects, 49 research projects, 17 training activities, and 17 outreach activities were conducted across the eight ACEs and two UPACEs.

Table 2.

ACE Program projects by type.

Type of ACE project	Number
Intervention research	27
Implementation research	4
Dissemination research	4
Other	14
Research total	49
Training	15
Technical assistance	0
Mentoring	1
Other	1
Training total	17
Communication and dissemination	7
Community mobilization	8
Other	2
Outreach total	17
Surveillance activities	17
Surveillance total	17

7. Establishment of projects in a high-risk community and outcome/impact of effectiveness in the community

This indicator focused on measuring Centers’ progress in establishing projects within defined, high-risk communities (Table 3) and in identifying outcomes or impact of programming. This included adoption of interventions, improved practice and policies for youth violence prevention, and the reduction of risk factors and increase in protective factors as a result of program implementation. All of the Centers successfully established surveillance, research, training, and outreach projects (Table 2). However, assessing the impact of youth violence prevention strategies was listed as an optional goal in the RFA for the 2005–2011 funding cycle; only three Centers chose to conduct effectiveness research.

Table 3.

Defined communities associated with the ACE Program, 2005–2011.

ACE	Defined community	Population	Ethnicity	Race	Setting	% below poverty
University of Illinois at Chicago ACE	Chicago West Side	442,817	36% Hispanic/Latino	56% African American/Black	City, urban	21–30
			64% not Hispanic/Latino	1% Asian
				18% White
Children’s Hospital of Philadelphia UPACE	Haddington	27,151	2% Hispanic/Latino	96% African American	City urban	21–30
			98% not Hispanic/Latino	1% White
	Cobbs County	40,198	1% Hispanic/Latino	96% African American	City, urban	21–30
			99% not Hispanic/Latino	1% White
	Kingsessing	34,075	1% Hispanic/Latino	95% African American	Neighborhood, urban	21–30
			99% not Hispanic/Latino	1% Asian
				2% White
Columbia University ACE	Washington Heights/Inwood	260,525	69% Hispanic/Latino	14% African American/Black	City, urban	31–40
			31% not Hispanic/Latino	1% Am. Indian/Alaskan Native
				2% Asian
				1% Native Hawaiian/Pacific Islander
				14% White
Harvard University ACE	Boston, MA	589,141	14% Hispanic/Latino	24% African American/Black	City, urban	11–20
			86% not Hispanic/Latino	8% Asian
				49% White
University of Hawaii ACE	Waimanalo	3664		26.8% Asian	Rural, neighborhood	5.9
				22.8% Native Hawaiian/Pacific Islander
				10.8% White
				2.9% Other Pacific Islander
				37% Unknown
	Waimanalo Beach	4271		5.3% Asian	Rural, neighborhood	5.5
				45.6% Native Hawaiian/Pacific Islander
				13% White
				1.8% Other Pacific Islander
				33.3% Unknown
	Waimanalo Homestead Land	3028		4% Asian	Rural, neighborhood	3.7
				52.5% Native Hawaiian/Pacific Islander
				6.6% White
				1.9% Other Pacific Islander
				33.9% Unknown
	Kailua	36,513		21.1% Asian	Rural, neighborhood	3.6
				7% Native Hawaiian/Pacific Islander
				43.8% White
				1% Other Pacific Islander
				25% Unknown
Johns Hopkins University ACE	Baltimore		3% Hispanic/Latino	64% African American/Black	City, urban	21–30
			97% not Hispanic/Latino	2% Asian
				33% White
Meharry Medical College (Nashville) UPACE	Nashville/Davidson County	635,710	9% Hispanic/Latino	28% African American	County, rural, suburban, urban	11–20
			91% not Hispanic/Latino	1% American Indian/Alaskan Native
				3% Asian
				70% White
University of California Berkeley ACE	Oakland Asian/Pacific Islander and Latino Immigrant Youth	44,853	58% Hispanic/Latino	2% African American/Black	City, urban	21–30
			42% not Hispanic/Latino	1% Am. Indian/Alaskan Native
				41% Asian
				1% Native Hawaiian/Pacific Islander
				21% White
University of California Riverside ACE	Santa Ana, CA	61,363	92% Hispanic/Latino	1% African American/Black	Neighborhood, urban
			8% not Hispanic/Latino	2% Asian
				5% White
	Riverside, CA (92503)	71,714	44% Hispanic/Latino	5% African American/Black	City, urban	11–20
			70% not Hispanic/Latino	4% Asian
				61% White
	Perris, CA	36,189	56% Hispanic/Latino	16% African American/Black	City, suburban	21–30
			42% not Hispanic/Latino	3% Asian
				23% White
Virginia Commonwealth University ACE	Richmond, VA	42,176	4% Hispanic/Latino	63% African American/Black	City, urban	21–30
			96% not Hispanic/Latino	2% Asian
				35% White

We assessed establishment of projects and outcome/impact of effectiveness in the community by abstracting project descriptions and data regarding the intended outcomes of the project reported as a reduction in risk and increase in protective factors, actual outcomes reported about risk and protective factors, as well as policy and practice outcomes. Three projects in particular, the Youth Violence Surveillance Project (Columbia Center for Youth Violence Prevention); the Safe Streets implementation and evaluation (Johns Hopkins Center for the Prevention of Youth Violence); and the Schools and Families Educating Children (SAFE) effectiveness evaluation (Chicago Center for Youth Violence Prevention) reported results regarding outcomes or impacts of effectiveness in their communities.

The Columbia Center for Youth Violence Prevention (CCYVP) collaborated with the New York City Department of Health and Mental Hygiene (NYC DOHMH) to collect and interpret a variety of data on violence indicators from multiple sources in their targeted community of Washington Heights/Inwood. The work accomplished through this collaboration was considered cutting-edge and served as a model for other surveillance systems nationwide. Contextual data regarding violent events and relationships were also collected and were unique features of this surveillance system. Analyses of these datasets enabled NYC DOHMH and the CCYVP to monitor changes in violent injury and death citywide, by borough, and by neighborhood. The DOHMH’s surveillance data informed mobilization activities and policy development, and shaped research and surveillance priorities at the health department and citywide.

The Johns Hopkins Center for the Prevention of Youth Violence conducted a community-based implementation and evaluation of the Safe Streets program, which is a public health program designed to reduce severe forms of youth violence through outreach to high-risk youth, conflict mediation, service delivery, and efforts to change social norms surrounding violence (Webster, Mendel Whitehill, Vernick, & Curriero, 2012). The Baltimore Safe Streets Program is a replication of the Chicago CeaseFire Model (Skogan, Hartnett, Bump, & Dubois, 2009), which illustrated some success in reducing shootings through the use of community-based violence mediators. The Safe Streets evaluation was a neighborhood-level longitudinal study to examine trends in youth violence in intervention communities and non-intervention communities before and after the program was implemented. The program was implemented and evaluated in four Baltimore communities. The evaluation of Safe Streets demonstrated that while there was some variation in the community sites, there was evidence that implementation of the program was associated with reductions in gun violence in three of the four intervention neighborhoods. There was also some evidence to suggest that there were spillover effects from the program, as there were reductions in gun violence in the neighborhoods adjacent to the intervention sites. However, one of the intervention neighborhoods experienced an increase in homicides. In the two program sites with the largest reductions in homicides, outreach workers mediated three times as many conflicts per month as the other two program sites, one of which experienced an increase in homicide rates. This finding suggests that conflict mediations were key to the program’s success (Webster et al., 2012).

Finally, the major research project at the Chicago Center for Youth Violence Prevention was the SAFE effectiveness evaluation (SAFE-E; Gorman-Smith, Tolan, Henry, & Schoeny, under review). The SAFE-E study involved the implementation of an evidence-based intervention at entry to school, which is a critical time in affecting the development of aggression. The study was intended to establish the utility of an efficacious program for preventing violence in inner-city communities. This study had three major goals: (1) to test the effectiveness of a family-based preventive-intervention targeting key risk markers for later aggression and related problem behaviors among families in inner-city Chicago; (2) to demonstrate the intervention can be implemented with fidelity in conditions that are common for service to inner-city families; and (3) to determine how intervention effects are influenced by variations in characteristics among the population and neighborhood conditions.

Gorman-Smith et al. (under review) found that among those who attended at least half of the sessions, there were significant improvements in parent-rated social skills and increases in parental monitoring practices at post-intervention as compared to those in the control condition. While no significant effects were found for youth aggression, some of these results may be due to lack of fidelity to program implementation. Motivated community organizations administered the program; however, some organizations lacked the capacity to do outreach with parents, which resulted in lower dosage of the intervention among the targeted sample and likely influenced the outcomes of the intervention.

8. Evidence of community ability to monitor and describe youth violence

Centers collected youth violence surveillance data from their defined communities using a variety of sources. The most commonly used data sources were de-identified hospital/emergency department records and police data (i.e., incidents, arrests, etc.). Data from these sources were used to track and monitor youth violence trends in defined communities.

One example that demonstrates how surveillance data were used to support and inform communities’ work includes surveillance activities from the Virginia Commonwealth University ACE. Center investigators obtained data from the state health department and vital records and health statistics, the VA Department of Education, the Richmond Department of Justice Services, VCU Health System’s Emergency Department, Richmond Police, and Richmond Ambulance Authority. Investigators collected, analyzed, and summarized data on violence among and against youth, locations where violent events occurred, times of heightened risk for violence, and the activities associated with violence. The Center disseminated quarterly fact sheets and reports to community organizations and researchers. The fact sheets included maps of violent incidents and summary information about violence in the community. Data collected allowed Center and community partners to compare violence rates in their defined community to other communities, monitor changes over time, and identify areas and populations in greatest need of intervention. Masho, Bishop, Edmonds, and Farrell (2014) conducted a study using data collected through these efforts to compare ambulance pick-ups among youths involved in violence in areas where alcoholic beverage sales were restricted with areas where beverage sales were not restricted. Their findings indicating that ambulance pick-ups were significantly lower in the intervention community than in the comparison community demonstrate how surveillance data can be used to guide community action.

9. Evidence of a communication and dissemination plan

Data from the communication and dissemination plans indicate that the ACE Program generated 857 different communication strategies ranging from audio recorded sessions and newsletters to television broadcasts and websites. The most widely used communication and dissemination strategies were presentations (n = 261), journal articles (directly related to Centers’ work n = 38; indirectly related to Centers’ work n = 184), conference papers (n = 46), book chapters (n = 35), and conference posters (n = 37). These communication efforts provided information to a wide range of audiences, including academics, public health practitioners, community-based organizations, Centers’ partners, community members, policy makers, and youth.

One way in which the Centers demonstrated evidence of communication and dissemination was through the development and publication of a special issue in the American Journal of Preventive Medicine. Edited by Mark and other ACE Program staff (2008), the special issue describes the Centers’ community mobilization efforts and the importance of engaging communities in the research that aims to positively impact them.

4. Discussion

Program evaluation is critical for effective program management, and a key component of program evaluation involves using data to improve and account for public health research practices. We used a novel data collection system, the ACE-IS, to examine the extent to which Centers met the performance indicators prescribed in the 2005 RFA in an effort to evaluate the ACE Program. This approach enabled us to identify attributes that we believe are key to successful community-based violence prevention research initiatives. First, many of the performance indicators outlined in the RFA were specifically designed to ensure that an infrastructure conducive to community-based participatory research was established early in the research process. We found that all of the Centers actively participated in a community committee and engaged their partners in a range of research activities. While it perhaps seems most natural for community members to be involved in activities aimed at identifying the problems in their area, the Center researchers continued to engage community members far beyond that first step – inviting them to help conceptualize research projects, collect data, co-author peer-reviewed publications, and even present findings at national conferences (e.g., Leff et al., 2010). Centers also maintained close connections with their partners through regular meetings and attendance at community events to ensure they were abreast of community needs, but also to confirm that their community mobilization plans were implemented as planned and to determine how such efforts impacted their defined communities.

The overarching goal of the 2005–2011 ACE Program was to establish the infrastructure necessary and to build communities’ capacity to implement youth violence prevention strategies. This lofty goal required much strategic planning, which was evident through the Centers’ extensive Center plans and research agendas. The RFA further indicated that Centers may strive to achieve the optional objective of establishing evidence of impact of the prevention strategies they implemented. Only through the thorough research and evaluation planning process, which several Centers underwent, was achieving this objective made possible.

4.1. Lessons learned

This evaluation of the 2005–2011 ACE Program, along with the strategic planning process that involves research-community collaborations, has informed future evaluation of large-scale prevention efforts, such as that of the 2010–2016 Youth Violence Prevention Program. In particular, this article describes the use of an approach for conducting program evaluation of a large-scale research initiative that involved establishing a research program logic model, performance indicators to monitor the application of the logic model, and data gathering approach from multiple large research centers. The ACE-IS served as a valuable mechanism for collecting data about the activities and accomplishments of the Centers that captured key information about activities and progress on key goals and objectives. However, given the complex nature of the Centers’ structure and their many activities and projects, it proved difficult to collect standardized information regarding the range of ACE Program communities, partners, and prevention strategies. In an attempt to resolve this problem, the ACE-IS provided space for Centers to enter open-ended responses for more detailed performance indicators in order to provide clarity regarding complex data. Still, the lack of standardized data limited our ability to use the ACE-IS to evaluate the impact of similar prevention strategies and to compare performance indicator outcomes across sites. However, this finding informed future CDC-funded youth violence prevention work and resulted in efforts to identify common indicators and measures across currently funded Centers in order to facilitate cross-site comparisons. Additionally, the currently funded Centers have been tasked with conducting full scale evaluations of the prevention strategies they selected for implementation, which will serve to strengthen the evidence base regarding what works to prevent youth violence.

Further, the approach we implemented for the ACE Program evaluation resulted in extensive data abstraction and review processes. While Centers were encouraged to continually update and enter new information in the ACE-IS, requiring them to enter data biannually at a minimum helped to ensure that information was updated regularly, and that Center products and accomplishments were appropriately represented in the evaluation. The evaluation protocol required confirmation of all information included in the evaluation reports through both a secondary source (i.e., the CDC staff assigned to each Center) and a tertiary source (i.e., the PI of each Center). This system of checks and balances ensured that all data reported were accurate. Ongoing, regular collection of such a large amount of data without the aid of the ACE-IS would have taken substantial time and resources. Using the ACE-IS as an evaluation tool through which data from large, multi-center programs were collected demonstrates how reporting, abstraction, and interpretation of a breadth of work conducted by each center, and the program as a whole, can be accomplished successfully. Further, CDC, Centers, and other stakeholders can use evaluation findings for many purposes, including modifying program activities or enhancing and strengthening relationships with community partners. Additionally, evaluation findings can be shared with external stakeholders, can help document the Program’s value, and may provide justification for continuing the Program.

5. Conclusion

The ACE Program continues to be funded under a new title, the CDC’s National Centers of Excellence in Youth Violence Prevention (YVPCs). Six YVPCs are currently funded to evaluate the effectiveness of evidence-based youth violence prevention strategies and to assess their impact on reducing rates of youth violence. While the ACE-IS was discontinued in 2012 due to funding restrictions, the method by which data were collected and evaluated in the program evaluation described here provides important insight for the planning and development of the multi-site evaluation of the 2010–2015 YVPC program.

The occurrence of youth violence is influenced by a number of factors, and in order for communities to achieve sustained reductions in violence, public health efforts are needed that bring the best science to bear on the needs of communities. Partnerships between research institutions and community organizations are a critical foundation to maximizing the effectiveness and reach of prevention efforts in communities. The CDC-funded ACE Program has established models for advancing community–research partnerships for youth violence prevention since 2000. The ACE Program has evolved over time to reflect the public health needs of communities and the research gaps. Results of the evaluation presented here highlight the progress and accomplishments of the ACE Program. In particular, the data document the results of tremendous efforts to build and sustain community partnerships between researchers and communities. The number and range of participants in the community partnerships demonstrated the reach and representativeness of the research–community relationships established by the Centers. The Centers’ partners played critical roles in efforts to reduce youth violence by setting priorities, developing community prevention plans, and supporting research and surveillance activities. Further, the Centers’ community mobilization efforts contribute to prevention efforts by building community capacity to address and prevent youth violence.

The progress of the ACE Program has evolved over time as a reflection of the state of the science and prevention opportunities in the field. The program has had a gradual evolution from emphasis on defining the problem of youth violence and conducting research on risk and protective factors in 2000–2005 to greater emphasis on community–research partnerships to develop, implement, and evaluate promising prevention practices in 2005–2011. The current round of centers (2010–2016) has placed emphasis on implementation of comprehensive, evidence-based strategies for youth violence prevention. This evolution has reflected the opportunities presented by program evaluation efforts. Program evaluation findings shown here were critical in establishing the successes and key accomplishments for the 2005–2011 Centers and identifying the “next steps” for the future of the program. Evaluation data showed that research had emphasized risk and protective factors, and had begun to move into establishing efficacy and effectiveness of programs. These findings highlighted the need to emphasize demonstrating impact on community-wide rates of violence in the 2010–2016 round of funding. The successes and accomplishments of the Centers documented through the program evaluation were also reflected in the 2010–2016 round of funding. The strength of community-research partnerships that were clearly demonstrated in the program evaluation served as the foundation for the concept of the 2010–2016 YVPC Program. Additionally, the importance of integrating community partners into data collection and strategic planning for prevention and evaluation was an emphasis in the current funding cycle and a direct extension of the successes of the 2005–2011 ACE Program.

Biographies

Kristin M. Holland, PhD, MPH is a behavioral scientist in the CDC’s Division of Violence Prevention. There, she leads the School-Associated Violent Death Study, which aims to collect qualitative and quantitative data on all violent school-related fatalities in the U.S. Her research interests include suicide and youth violence etiology, surveillance, and prevention. She is also interested in incorporating mixed methods in her research, using both qualitative and quantitative data to better understand risk factors to and potential precipitators of suicide and youth violence.

Alana M. Vivolo-Kantor, MPH is a behavioral scientist in the CDC’s Division of Violence Prevention. There, she focuses her research on understanding the underlying causes and consequences of violence, specifically youth violence, bullying, and teen dating violence. She has published extensively in these areas and currently serves as the Division’s subject matter expert for bullying by serving on federal organizations to prevent bullying and by regularly responding to media and public inquiries regarding bullying research. She currently leads the CDC’s National Centers for the Prevention of Youth Violence and is also pursuing a Ph.D. in Public Health at Georgia State University.

Jason dela Cruz, MPH was an American Society of Public Health research fellow in the CDC’s Division of Violence Prevention from 2008 to 2010. There, he worked closely with the Academic Centers of Excellence in Youth Violence Prevention Program staff and grantees. He assisted with the development of the ACE-Information System, and also significantly contributed to the ACE Program evaluation plans.

Greta Massetti, PhD is currently the Associate Director for Science in the CDC’s Division of Cancer Prevention and Control. Formerly, she was the Branch Chief of the Research and Evaluation Branch in the CDC’s Division of Violence Prevention. While in the Division of Violence Prevention, she led the Academic Centers of Excellence in Youth Violence Prevention Program with the goal of identifying and evaluating evidence-based youth violence prevention programs.

Reshma R. Mahendra, MPH is a Public Health Advisor in the CDC’s Division of Violence Prevention. There, she focuses on the implementation of youth violence, child maltreatment, intimate partner violence, and sexual violence prevention strategies and programs. Reshma led the development of the ACE-Information System for the Academic Centers of Excellence in Youth Violence Prevention Program.