Implementation Measurement for Evidence-Based Violence Prevention Programs in Communities

Abstract

Increasing attention to the evaluation, dissemination, and implementation of evidence-based programs (EBPs) has led to significant advancements in the science of community-based violence prevention. One of the prevailing challenges in moving from science to community involves implementing EBPs and strategies with quality. The CDC-funded National Centers of Excellence in Youth Violence Prevention (YVPCs) partner with communities to implement a comprehensive community-based strategy to prevent violence and to evaluate that strategy for impact on community-wide rates of violence. As part of their implementation approach, YVPCs document implementation of and fidelity to the components of the comprehensive youth violence prevention strategy. We describe the strategies and methods used by the six YVPCs to assess implementation and to use implementation data to inform program improvement efforts. The information presented describes the approach and measurement strategies employed by each center and for each program implemented in the partner communities. YVPCs employ both established and innovative strategies for measurement and tracking of implementation across a broad range of programs, practices, and strategies. The work of the YVPCs highlights the need to use data to understand the relationship between implementation of EBPs and youth violence outcomes.

Implementation and Fidelity Monitoring for Evidence-Based Violence Prevention Programs in Communities

The last two decades have seen a dramatic emphasis on evaluating preventive interventions and establishing evidence-based programs [1]. This emphasis has led to a wealth of research on “what works” and a growing list of programs that have been evaluated and have had demonstrated effects on a range of health, education, and social outcomes. The proliferation of these programs and policy efforts to promote the use of them are grounded in the proposition that implementation of programs demonstrated to be effective is the most efficient and effective way to ensure similar outcomes for future participants and communities. As the availability of evidence-based programs has grown, so has the demand for them in schools and communities. The term “evidence-based programs” (EBP) here refers to a wide variety of interventions, prevention programs, or strategies that have been rigorously evaluated and demonstrated effectiveness in preventing or reducing one or more negative outcomes or promoting one or more positive outcomes. The spread of EBPs has presented the prevention science and practice community with a new set of challenges and opportunities related to their high-quality implementation and delivery. As Durlak and DuPre [2] effectively put it, “implementation matters”.

The relationship between implementation and outcomes has consistently been documented in the literature. In a comprehensive meta-analysis of over 500 studies, Durlak and DuPre [2] reported definitive evidence for a relationship between implementation quality and outcomes for prevention programs. They defined implementation as “what a program consists of when it is delivered in a particular setting” (p. 239) [2]. This definition has served as a standard in the literature on dissemination and implementation.

Monitoring of implementation and fidelity is useful for accountability, but also important for continuous quality improvement efforts through the use of data and monitoring tools to assess and improve implementation fidelity. For example, implementation data can be used to identify areas where training or technical assistance is needed. Throughout the implementation planning stages, benchmarks and metrics for implementation quality can be established to assess implementation progress. Implementation data can also guide meaningful efforts to provide ongoing consultation and feedback to implementers.

A variety of strategies are often used to measure implementation. These strategies vary widely in costs and other resources necessary to collect and monitor the data they produce. Typical strategies for tracking implementation include semi-structured interviews, case studies, review of secondary documents and/or administrative databases, direct observation, focus groups, and self-report surveys [2–4]. Beyond methods for measuring implementation, the fundamental principle of measuring implementation is to assess the internal validity for an intervention study. Implementation data provides qualitative and quantitative metrics of the extent to which the intervention is implemented as the developer intended. Without data on what was implemented, how it was implemented, and how well it was implemented, it is not possible to determine whether the intended intervention contributed to any changes in outcomes for participants. So while implementation, at its core, is about program delivery, it is also a fundamental component of evaluation.

Overview of Implementation Concepts as They Relate to YVPCs

The primary objective of the National Centers of Excellence in Youth Violence Prevention (YVPCs) is to partner with communities to implement a comprehensive community-based strategy to prevent violence, and to evaluate that strategy for impact on community-wide rates of violence. The Centers for Disease Control and Prevention (CDC) funded six YVPCs in 2010–2015. These include the Chicago Center for Youth Violence Prevention (CCYVP) at the University of Chicago, the University of Michigan Youth Violence Prevention Center (MI-YVPC), the Virginia Commonwealth University Clark-Hill Institute for Positive Youth Development (Clark-Hill Institute at VCU), the North Carolina Rural Academic Center for Excellence in Youth Violence Prevention (NC-ACE) at the University of North Carolina at Chapel Hill, the Johns Hopkins Center for the Prevention of Youth Violence (JHCPYV) at Johns Hopkins University, and the “Steps to Success” Denver Collaborative to Reduce Youth Violence at the University of Colorado at Boulder (Steps to Success). Each center has established partnerships with communities and the organizations within them to support efforts to implement EBPs that meet the unique needs of each community, and that fit within its context and infrastructure. The strategies and programs employed by the center-community partnerships involve comprehensive approaches that address individual, relationship, and community factors [5, 6]. The prevention strategies combine both universal approaches delivered to all youth or residents regardless of risk as well as selected strategies directed at youth or their families who are at elevated risk for violence perpetration.

YVPCs were required to include a strategy to document implementation of and, if relevant to specific components, fidelity to the components of the comprehensive youth violence prevention strategy. The purpose of this article is to describe the strategies and methods used by the six YVPCs to assess implementation and to use implementation data to inform program improvement efforts. The information presented describes the approach and measurement strategies employed by each center and for each program implemented in the partner communities.

The many programs and strategies implemented across the YVPCs vary widely in the scope, content, and approach. As such, there is also variability in the opportunities to collect valid implementation data efficiently. Because each community implemented different packages of intervention components and strategies, investigators at each site designed implementation measurement approaches that matched the structure of the prevention packages unique to their communities and that took into account the unique community context and infrastructure for implementation. The structure, content, and context of programs lend themselves differently to tracking and fidelity monitoring. For our overview of implementation measurement across YVPCs, we relied on Durlak and DuPre’s [2] definition of implementation. Table 1 includes a summary of the core programs implemented as part of the YVPCs’ comprehensive EBP packages, the communities, and implementation sites, and Table 2 includes a brief description of each of the programs.

Table 1.

Youth violence prevention strategies implemented across the Youth Violence Prevention Centers

Center	Intervention community	Program name	Program type	Setting
Chicago Center for Youth Violence Prevention	Humboldt Park, Chicago, Illinois	Chicago CeaseFire [7]	Community-based and school-based components	Neighborhood High school-based CeaseFire
		SAFE Children [12]	Family/parenting and school-based components	All middle schools in intervention community
University of Michigan Youth Violence Prevention Center	Durant-Tuuri-Mott Neighborhood, Flint, Michigan	Community mobilization efforts	Community-based program	Neighborhood
		Clean and Green/Adopt-A-Lot	Community-based program	Neighborhood
		Youth Empowerment Solutions [15]	School-based and community-based components	Neighborhood and high school
		Fathers and Sons [16]	Family/parenting program	Community-based organization
		Project SYNC [17]	Healthcare-based program	Hospital emergency department
Virginia Commonwealth University Clark-Hill Institute for Positive Youth Development	3 neighborhoods in Richmond, Virginia	Youth Empowerment Solutions [15]	School-based program	All middle schools in intervention communities
		Olweus Bullying Prevention Program [18, 19]	School-based program	All middle schools in intervention communities
		Staying Connected with Your Teen [20]	Family/parenting program	All middle schools in intervention communities
		Parenting Wisely [21]	Family/parenting program	All middle schools in intervention communities
North Carolina Rural Academic Center for Excellence in Youth Violence Prevention	Robeson County, North Carolina	Teen Court [22]	Community-based program	County government
		Positive Action [23, 24]	School-based program	All middle schools in intervention communities
		Parenting Wisely [21]	Parenting/family program	Community-based organizations and schools
Johns Hopkins Center for the Prevention of Youth Violence	Neighborhood in Baltimore, Maryland	Safe Streets [7]	Community-based program	Neighborhood
		Problem Alcohol Outlet Monitoring Program	Community-based program	Neighborhood
		Positive Behavioral Interventions and Supports [25]	School-based program	All middle schools in intervention communities
		Olweus Bullying Prevention Program [18, 19]	School-based program	All middle schools in intervention communities
		Coping Power [26]	School-based and family/parenting components	All middle schools in intervention communities
University of Colorado Boulder—The “Steps to Success” Denver Collaborative to Reduce Youth Violence	Montbello neighborhood, Denver, Colorado	Communities That Care [27]	Community-based program	Neighborhood, community-based organizations
		Promoting Alternative Thinking Strategies [28]	School-based program	All elementary schools in intervention communities
		Positive Family Support [29]	Family/parenting program	All middle schools in intervention communities
		Strengthening Families 10–14 [30]	Family/parenting program	All middle schools in intervention communities

Table 2.

Descriptions of Youth Violence Prevention Centers and youth violence program implementation activities

Youth Violence Prevention Center	Program implementation activities
Chicago Center for Youth Violence Prevention	Community-based and school-based
	Chicago CeaseFire, a community-level violence prevention strategy, is implemented to influence neighborhood norms regarding aggression and violence and the occurrence of retaliatory violence. In addition to targeting neighborhood level norms and the behavior of adolescent and young adult high-risk youth, universal and targeted strategies are implemented within the six elementary schools in the community. Chicago CeaseFire is implemented in the community and in the one high school within the Humboldt Park neighborhood.
	Family/parenting and school-based
	The six middle schools in the intervention community implement the SAFE-Children universal program delivered to families and a classroom curriculum. SAFE-Children is a family-based, comprehensive preventive intervention that helps families gain parenting and family management skills that facilitate successful child academic and social adjustment. A comprehensive version of SAFE-Children focused on parenting and family functioning related to aggression and violence is also implemented with families of kindergarten and first grade children who have shown early signs of aggression or problem behavior in school.
University of Michigan YVPC	Community-based
	The community mobilization program provides support to the Flint Police Department and neighborhood organizations and has three main areas of focus: distributing real-time crime data analyses and conducting crime prevention workshops; convening community discussions and focus groups; and providing technical support to neighborhood organizations and law enforcement. Clean and Green/Adopt-A-Lot and Youth Empowerment Solutions (YES) offer two universal community-based programs that serve all community youth by reducing urban blight, improving the physical spaces in which youth live, and empowering them to be engaged in pro-social community life. Within Flint, neighborhood residents and community organizations are engaged in the maintenance of over 3600 vacant or bank-owned property parcels in the area. Clean and Green/Adopt-a-Lot aims to beautify these areas and create green space and involves lot maintenance (e.g., mowing) and developing at least one gardening project.
	Family/parenting
	Fathers and Sons works with individual youth and their fathers to address indications of risky behaviors and support father-son relationships and parenting skills.
	Healthcare-based
	Project SYNC is a brief motivational interviewing intervention for youth who come to the emergency department.
Virginia Commonwealth University Clark-Hill Institute for Positive Youth Development	School-based
	The center is implementing two comprehensive school-based programs focused on middle-school students, the Olweus Bullying Prevention Program, and Youth Empowerment Solutions (YES). Olweus is a bullying prevention program focused on creating a positive school climate, improving peer relations, and increasing awareness of and reducing the opportunities for bullying behavior. YES is a curriculum-based program that promotes positive development and prevents risk behaviors among youth by empowering them to change their physical and social environments.
	Family/parenting
	Two evidence-based programs for high-risk youth and their families, Parenting Wisely and Staying Connected with Your Teen are implemented in both group and individual formats. Parenting Wisely involves training for parents through interactive, computer-based modules and aims to increase parental communication and discipline skills. Staying Connected with Your Teen is a seven-session universal prevention program.
North Carolina Rural Academic Center for Excellence in Youth Violence Prevention	School-based
	Positive Action is implemented in all middle schools in Robeson County, North Carolina. Positive Action is a comprehensive classroom curriculum for grades K through eight designed to improve academic achievement, school attendance, and problem behaviors such as substance use, violence, suspensions, disruptive behaviors, dropping out, and sexual behavior.
	Community-based
	High-risk youth in the county participate in Teen Court. Teen Courts are problem-solving courts that provide an alternative disposition for juveniles who have committed a delinquent act, have committed a minor offense, or have been charged with a misdemeanor, and are otherwise eligible for diversion.
	Family/parenting
	Families are referred to the Parenting Wisely program to strengthen parent-youth relationships and address parenting skills. Parenting Wisely is described in the Clark Hill Institute section above.
Johns Hopkins Center for the Prevention of Youth Violence	Community-based
	The two community-based interventions being implemented include the Baltimore City Health Department’s Safe Streets Program and the Problem Alcohol Outlet Monitoring Program. Safe Streets is the name of the Baltimore implementation of Chicago’s Cure Violence (also known as CeaseFire), a violence prevention program that uses trained street violence interrupters and outreach workers, public education campaigns, and community mobilization to reduce shootings and killings. Problem Alcohol Outlet Monitoring involves a collaboration between the Baltimore City Health Department, the Baltimore City Planning Department, the Baltimore City Council and the Mayor’s Office to aid in the crafting and dissemination of the Baltimore City Zoning Plan which identifies strategies to reduce youth access to alcohol.
	School-based
	Three school-based youth violence prevention programs are being implemented, including Positive Behavioral Interventions and Supports, the Olweus Bullying Prevention Program and the Coping Power Child and Parent Program. PBIS is a universal, school-wide prevention strategy aimed at reducing behavior problems that lead to office discipline referrals and suspensions, and change perceptions of school safety. Coping Power is a cognitive-based intervention for aggressive children and their parents during the transition to middle school that aims to increase competence, study skills, social skills, and self-control in aggressive children. Olweus is described in the Clark Hill Institute description above.
University of Colorado Boulder—The Denver Collaborative to Reduce Youth Violence	Community-based
	Steps to Success is implementing an adapted version of the Communities That Care (CTC) system. CTC is a strategic planning mechanism that builds upon a concept of first identifying and then targeting various risk factors that lead to violent behavior in a given community. The community drives the process to identify priority risk factors targeted to address violence and other problem behaviors. A Community Board that is responsible for planning and implementation conducted school surveys and was provided with a “menu” of evidence-based programs that impact community risk factors related to violence.
	School-based
	Promoting Alternative Thinking Strategies (PATHS) is a classroom-based prevention program that promotes emotional and social competencies and reduces aggression and behavior problems in elementary school-aged children.
	Family/parenting
	Positive Family Support and Strengthening Families 10–14 (SF 10–14). Positive Family Support is a family centered intervention targeting middle school youth at risk and their families. SF 10–14 is a 7-session group parenting and youth skills program that includes separate weekly parent effectiveness training and child skills-building, followed by a family session to promote good parenting skills and positive family relationships.

Implementation and Fidelity Monitoring Activities at the YVPCs

Table 3 includes information on the measurement strategies used for each YVPC program, described as measures of fidelity, quality of delivery, dosage, reach, or outputs. For the purpose of the table, fidelity was defined as the assessment of whether the program was delivered as intended. Quality of delivery was defined as the assessment of the extent to which the program was delivered in the manner in which it was intended (the how of delivery, rather than the what, which refers to fidelity). Reach was defined as the proportion of the target population that was reached by the program. Dosage was defined as the “amount” of program that was delivered. Outputs was a catch-all category that included process measures and tracking of materials generated in the course of program delivery.

Table 3.

Implementation measurement for youth violence prevention programs and strategies implemented across the Youth Violence Prevention Centers

Youth Violence Prevention Center	Program	Implementation Measurement
Chicago Center for Youth Violence Prevention	Chicago CeaseFire	Outputs
		Number of conflict mediations; proportion of mediations that address gang conflicts
		Dosage
		Hours spent mediating
		Reach
		Tracking students involved in mediations
	SAFE Children and GREAT Families	Fidelity, Quality of Delivery
		Parent Group Fidelity Measure; Parent Group Process Measure; Family Group Leader Fidelity Measure; Family Group Leader Process Measure
University of Michigan YVPC	Community Mobilization efforts	Reach
		Number of residents participating in mobilization activities
		Outputs
		Mobilization session discussions; technical assistance provided on community improvements
	Clean and Green/Adopt-A-Lot	Outputs, Fidelity, Dosage, Reach
		Property parcel ratings
	Youth Empowerment Solutions	Fidelity
		Session fidelity observations
		Dosage
		Tracking session delivery
		Reach
		Youth participation in sessions
	Fathers and Sons	Fidelity, Quality of Delivery
		Session observations and monitoring forms; participant surveys
	Project SYNC	Reach
		Percent of youth participating in the intervention
		Fidelity, Quality of Delivery
		Motivational Interviewing Treatment Integrity Code; session content evaluations
Virginia Commonwealth University Clark-Hill Institute for Positive Youth Development	Olweus Bullying Prevention Program and Youth Empowerment Solutions	Fidelity, Quality of Delivery
		Classroom observations; classroom climate assessments; staff surveys
		Dosage
		Tracking session delivery
	Staying Connected with Your Teen and Parenting Wisely	Fidelity, Dosage
		Tracking of family outreach activities; participant assessments; participant evaluations; Workshop Leader’s Rating Sheets
North Carolina Rural Academic Center for Excellence in Youth Violence Prevention	Teen Court	Fidelity, Quality of Delivery, Outputs
		Teen Court compliance monitoring observations
		Fidelity
		Tracking compliance with dispositions
	Positive Action	Fidelity, Quality of Delivery
		Classroom observations
		Fidelity, Dosage
	Parenting Wisely	Weekly teacher implementation logs (Computer-administered version)
		Fidelity, Dosage
		Computer tracking of sessions and assessments completed (Group-administered version)
		Fidelity, Dosage
		Tracking of sessions completed; session logs
	Safe Streets	Reach
		Numbers of participants in outreach activities
		Outputs
		Tracking monthly referrals by outreach workers, number of mediations
Johns Hopkins Center for the Prevention of Youth Violence	Problem Alcohol Outlet Monitoring
	Positive Behavioral Interventions and Supports, Coping Power, Olweus Bullying Prevention Program	Quality of Delivery
		Interviews with school staff, administrators, and students
		Fidelity
		School-wide Evaluation Tool (SET) and Individual Student Systems Evaluation Tool (I-SSET) assessments
		Outputs
		Program material reviews
	Communities That Care	Fidelity, Quality of Delivery
		Communities That Care Readiness Checklist; CTC delivery monitoring
University of Colorado Boulder—The Denver Collaborative to Reduce Youth Violence	Promoting Alternative Thinking Strategies	Fidelity, Quality of Delivery
		Classroom observations
		Fidelity
		Interviews with teachers and administrators
	Positive Family Support	Fidelity, Quality of Delivery
		Fidelity check-up observations; staff interviews; participant interviews
	Strengthening Families	Dosage
		Family attendance tracking
		Fidelity, Quality of Delivery
		Session observations; observer checklists; facilitator interviews

Community-Based Youth Violence Prevention Programs

Chicago CeaseFire and Baltimore Safe Streets, Chicago and Baltimore

Implementation measurement for both CeaseFire and Safe Streets involves tracking of outputs for the interventions. Previous research has reported that the number of mediations as a measure of implementation is associated with intervention effectiveness [7]. In order to track implementation of the conflict mediation component of the community-based Chicago CeaseFire intervention, CCYVP investigators assess the number of conflict mediations conducted by outreach workers and violence interrupters. Additionally, they assess the proportion of these mediations that involve gang conflict. For the school-based CeaseFire intervention, CCYVP investigators are tracking the number of conflicts mediated in the intervention high school, as well as the number of students involved in those mediation activities, and the hours spent mediating. The Safe Streets research team tracks the number of participants in the outreach activities, the number of monthly referrals made by outreach workers, and the number of monthly conflicts mediated by outreach workers and violence interrupters.

MI-YVPC Community Mobilization, Flint

Measurement of the Community Mobilization Component for MI-YVPC involves documenting neighborhood group participation and the content of discussions in the mobilization sessions. Staff maintain records of technical assistance provided for community improvement and crime prevention activities and track activities and outputs related to technical assistance.

Clean and Green/Adopt-A-Lot, Flint

MI-YVPC investigators monitor fidelity for the Clean and Green intervention through systematic property ratings, which document mowing, litter, landscaping, etc. Property ratings are assessed using the Parcel Maintenance Observation Tool, a validated instrument that captures physical disorder of property parcels and provides a record of the neighborhood physical environment [8].

Teen Court, Robeson County, North Carolina

The Robeson County Teen Court program is highly structured and follows a scripted format laid out in a program manual. Teen Court Coordinators observe every session and ensure compliance with the manual. If a proceeding strays from the format required by the manual, coordinators step in to provide in vivo feedback and get the proceedings back on track. Investigators also assess the extent to which offenders carry out their restitution plans as prescribed in the Teen Court dispositions. If defendants do not complete their restitution plans, notices of non-completion are sent to the referral source (typically, the school or police).

Problem Alcohol Outlet Monitoring Program, Baltimore

To track progress of the Problem Alcohol Outlet Monitoring Program, the Center investigators identified current alcohol outlet zoning regulations and track proposed zoning modifications and whether they are adopted by Baltimore City at voting times. Researchers also regularly monitor changes in the number and locations of local alcohol outlets.

Communities That Care (CTC), Denver

Center investigators at the University of Colorado YVPC are evaluating the extent to which the intervention community is implementing CTC with fidelity. Quantitative and qualitative data are collected to assess whether project goals are continually met and occur in a timely manner based on the prescribed CTC timeline, consistent with previous research on CTC implementation [9]. The CTC Readiness Checklist is used to track CTC through the five phases of implementation: Getting Started; Organizing, Introducing and Involving; Developing a Community Profile; Creating a Community Action Plan; Implementing and Evaluating the Community Action Plan. A scoring team completes a survey with questions specific to each phase of CTC (e.g., Did the board approve a community action plan? Does a new member orientation exist?). Implementation data are discussed at research staff meetings and with the Community and Key Leader Boards to address implementation issues identified in the process evaluation. This fidelity monitoring process also identifies in real-time areas where implementation may be failing to meet goals in order to ensure that technical assistance is immediately provided to improve implementation quality.

School-Based Youth Violence Prevention Programs

Strategies to measure fidelity of school-based prevention strategies include classroom observations, interviews with individuals implementing sessions, and self-report logs or surveys to measure adherence to the implementation protocol. Such measures are described here in detail for the programs implemented by the YVPCs.

Youth Empowerment Solutions (YES), Flint

MI-YVPC investigators conduct fidelity observations for YES twice each month while the program is in session and meet with the YES staff on a regular basis to monitor program implementation and provide coaching and support. Implementation measurement focuses on delivery of the sessions and youth participation in the program.

Youth Empowerment Solutions and Olweus Bullying Prevention Program, Richmond

Clark-Hill Institute investigators implementing YES and Olweus collect fidelity data through observations of approximately 20 % of Olweus classroom meetings to assess adherence to the protocol, competence of delivery, student engagement, and classroom climate. Anonymous surveys are administered to a random sample of school staff to obtain more general feedback about the program. Key findings from each school are presented to and discussed with school staff to reinforce and improve implementation.

Positive Action, Robeson County, North Carolina

NC-ACE investigators monitor implementation of Positive Action classroom-based activities in several ways. Center staff observe teachers and complete rating forms to document implementation and delivery of the curriculum lessons. Throughout implementation of Positive Action, teachers complete self-report weekly implementation logs annotating which lessons are implemented. Center investigators collect weekly logs and track progress in implementation, using the data to generate graphs and other tools to assess progress. Center investigators also establish benchmarks for implementation that are used to monitor progress and provide support to teachers in understanding a manageable pace for implementation of lessons. Implementation data are also used to identify teachers who needed additional support in implementing lessons.

Positive Behavioral Interventions and Supports, Coping Power, Olweus Bullying Prevention Program, Baltimore

School visits are used by JHCPYV investigators to assess the implementation quality of the school-based interventions, including Positive Behavioral Interventions and Supports (PBIS), Coping Power, and the Olweus Bullying Prevention Program [10]. Assessment of the programs and implementation of program components is conducted at schools in both intervention and comparison communities. This enables Center investigators to monitor the extent to which components of these different models are present in the comparison schools absent of formal training or center support.

Two core measures are administered in a single school visit. The School-wide Evaluation Tool (SET) and the Individual Student Systems Evaluation Tool (I-SSET) [10, 11]. The SET is a validated measure of school-wide PBIS implementation fidelity and is completed by a trained external observer who assesses the following seven key features of school-wide PBIS: Expectations Defined, Behavioral Expectations Taught, System for Rewarding Behavioral Expectations, System for Responding to Behavioral Violations, Monitoring and Evaluation, Management, and District-Level Support. The I-SSET was developed to document the fidelity with which school-based programs are implemented. School visits include brief interviews with an administrator and the student support team leader regarding the types of programs and supports provided to students. Assessors briefly interview teachers, support staff members, and students from each grade level to gather information about the schools’ procedures, policies, and standards for positive behavior. The assessor also reviews intervention planning materials to determine the quality of systems foundations, targeted interventions, and intensive individualized interventions occurring within the school. The I-SSET also covers the key features of the Coping Power program and Olweus, including a brief interview with the school-based program facilitators, youth participants, and administrators. Trained assessors who are unaware of the schools’ implementation condition conduct the SET and I-SSET fidelity measures. Center investigators also use fidelity data to inform implementation. A report summarizing the SET fidelity data is made available to the PBIS team leader and coach to promote high quality implementation.

Promoting Alternative Thinking Strategies (PATHS), Denver

Classroom observations conducted by trained observers are used to collect fidelity and implementation data for teachers implementing PATHS. Five observations are conducted with each teacher. Individual interviews with principals, teachers, and classroom observers are used to collect additional process data to inform implementation of the program.

Family and Parent-Focused Interventions

Family and parent-focused interventions typically include fidelity and implementation monitoring strategies such as parent interviews and surveys, group observations, training of program implementers and observers, and tracking of program dosage.

SAFE Children, Chicago

Center investigators at the CCYVP use fidelity and intervention monitoring measures that were originally derived from the GREAT Schools and Families intervention and the SAFE Children program [12]. The measures include a Parent Group Fidelity Measure, which assesses attitudes about the group and the activities completed. The survey is completed by parents during four sessions throughout the program. The Parent Group Process Measure is a parent participant survey completed during three separate sessions and assesses attitudes about the group leader, group content, and family involvement in the intervention sessions. In addition to these parent-report measures, group leaders complete the Family Group Leader Fidelity Measure four times throughout the program and the Family Group Leader Process Measure three times. The Fidelity Measure assesses factors related to the group, the content, and the activities completed. The Process Measure assesses attitudes about the families, group content, and family involvement in the sessions.

Fathers and Sons, Flint

To capture fidelity to the Fathers and Sons program curriculum, program implementers and program observers undergo extensive training. Program observers attend all sessions and complete a monitoring form to assess adherence to the curriculum content and materials. Male youth and fathers participating in the program complete surveys pre- and post-administration, and survey data are used to assess attitudes and behaviors of participants.

Parenting Wisely, Robeson County, North Carolina

The computer-based delivery for Parenting Wisely and the activities parents complete on the computer facilitate data collection throughout delivery. The use of technology allows for real-time data collection of progress in completing lessons and activities. It also ensures that the content is delivered consistently in that all videos and activities are presented through the computer-based format. NCACE investigators track the number of lessons completed for parents who participate in the program on the computer. Parenting Wisely is also administered in groups for parents who select that option. For group administration, investigators track the number of group sessions attended and the number of lessons completed by each participant, providing measures of dosage for participating families.

Staying Connected with Your Teen and Parenting Wisely, Richmond

Fidelity and dosage data for the Staying Connected with Your Teen and Parenting Wisely individual family programs implemented by Clark-Hill Institute investigators are collected from families over the phone and/or in person. Family consultants record the time spent trying to contact families, time spent on the phone or in-person with families, and total number of calls/home visits. Fidelity data on parents’ level of understanding of the material and their satisfaction with the program are gathered during the intervention. Upon completion of the individual Parenting Wisely or Staying Connected with Your Teen program, parents complete an evaluation form rating their family consultant and their satisfaction with and usefulness of the program. For Staying Connected with Your Teen and Parenting Wisely groups, each parent completes an evaluation form at each session that asks them to rate each topic in the session and provide narrative feedback about the group meeting. Each of the two co-facilitators also completes a Workshop Leader’s Rating Sheet after each session that rates group member participation, indicates the amount of material covered, and provides narrative feedback about the group session.

Positive Family Support, Denver

University of Colorado YVPC investigators conduct staff interviews to collect process data on implementation of Positive Family Support. Additionally, fidelity check-ups are collected twice per year by the technical assistance provider to monitor implementation and fidelity data. Finally, data from parent and youth interviews are collected on satisfaction and engagement with the program.

Strengthening Families 10–14 (SF 10–14), Denver

Bilingual facilitators keep track of youth and family attendance. Observations of SF 10–14 youth group, parent group, and the combined family group sessions are conducted by trained observers who collect information on whether the program facilitators follow SF procedures and the activities of the SF program. Each observer completes a checklist to assess the degree to which the program is implemented as intended. Facilitators are also interviewed at the end of each 7-week program about their experiences conducting program sessions.

Healthcare-Based Interventions

Project SYNC, Flint

Project SYNC’s fidelity plan is threefold. The first component of the fidelity evaluation determines the reach of the study by evaluating the percent of youth in the intervention neighborhood that come to the Emergency Department (ED) and receive the intervention. Detailed logs of all patients seeking care at the ED (even during hours not staffed by Project Sync recruiters) are updated daily by research staff. The second component uses the MITI3, or Motivational Interviewing Treatment Integrity Code [13]. The MITI3 is used to evaluate the therapists’ adherence to Motivational Interviewing concepts and values. All sessions are taped, and 20 % of them are coded using the MITI3. The third component of the fidelity evaluation measures content adherence of the intervention. Twenty percent of sessions are coded based on content delivered as intended.

Discussion

Accountability for implementation is critical. It is an integral part of implementation of evidence-based interventions and must be planned for—from budget planning to staffing to timelines. Implementation and fidelity measurement is highly dependent on the unique features of each program. Measurement of implementation is also directly related to quality of implementation. Special attention is given to aspects of the programs that are being measured, which provides opportunity to improve implementation quality. The approaches and methods employed by YVPCs to track implementation demonstrate rigor and creativity in efforts to open the “black box” of implementation. The variety of programs and strategies implemented by the YVPCs required that they devise efficient and systematic efforts to measure and monitor implementation.

In establishing plans for measuring and tracking implementation, YVPCs faced unique challenges related to the broad and comprehensive scope of their prevention strategies. Because each center implements multiple programs across different contexts and with multiple partners, efforts to measure implementation across those programs, contexts and partners reflect that complexity. For EBPs that use more common or traditional implementation and delivery approaches, such as family-based or school-based programs, the program materials sometimes included tools or previously used materials and methods for measuring implementation. Center investigators also relied on the published literature to identify measurement strategies. Although these programs have an established literature with respect to measurement of implementation, challenges remain in measuring implementation in a valid and efficient way. For example, trials for school-based or family/parenting programs typically evaluate these programs as implemented on their own. Therefore, the literature on measurement of implementation of these programs can sometimes provide a thorough and deep analysis of each EBP. When these programs are implemented in the context of a package of interventions within each YVPC community, that level of depth is often not feasible. Therefore, center investigators established plans to capture the “key elements” of each program as implemented in order to provide snapshots of implementation and fidelity that inform efforts to improve implementation quality.

Center investigators also face unique challenges in measuring implementation for programs that are not traditional EBPs. These typically include the community-based strategies and policy approaches that rely less on curriculum delivered to youth or families. YVPCs have demonstrated creativity and innovation in identifying ways to capture information about how these strategies are implemented. For example, the property assessments used by the center at the University of Michigan constitute a systematic approach to tracking the physical changes that are expected to take place as a reflection of the place-based strategies employed within the community [14]. This has allowed center investigators to track block-by-block data about the changes in physical characteristics of the intervention community.

Another example of innovative implementation measurement strategies can be seen in the efforts to assess implementation for the CeaseFire/Safe Streets program used at both the University of Chicago and the Johns Hopkins University centers. Previous research on the Chicago CeaseFire program has indicated that poor implementation can result in poor outcomes [7]. Therefore, careful measurement of implementation factors is critical for ensuring the intended effects on violence in the CeaseFire intervention communities. Both of the centers are collecting detailed data on the interpersonal components of the intervention, including the number, types, and nature of mediations. This information will be critical in making the connections between community-level effects and the work of the mediators and outreach workers for CeaseFire.

The information about implementation measurement across the programs, strategies, and centers yields some interesting themes in the process of measuring implementation. Most of the measurement strategies employed by the YVPCs involve measures of fidelity and quality of implementation. This may reflect the fact that there is a stronger literature on the process of measuring these elements of implementation than for others, such as reach or dosage. Additionally, research on the relationship between implementation and intervention effectiveness typically relies on measures of fidelity and quality [2]. The YVPCs’ reliance on these elements is likely to yield important information on the potential effectiveness of the programs implemented. Relatively fewer YVPCs are measuring reach or dosage; this is likely a reflection of the fact that these elements of implementation are more challenging to define and track.

Another theme from the implementation measures involves the fact that many YVPCs rely on resource-intensive strategies for measurement, such as in-person observations. These strategies have the potential to yield accurate and valid data that can inform quality improvement and implementation efforts for the programs. However, this form of measurement also raises questions about sustainability as the programs transition from start-up to maintenance.

YVPCs employ measures previously used in the literature for some programs, while for others they develop new measures or tools, likely reflecting the availability of existing measures in the field. Opportunities to publish data on the newly developed implementation measures have the potential for making important contributions to the implementation literature.

There was also significant variability in implementation measures across programs and centers. This was the case even across similar strategies, such as school-based programs, where some YVPCs are using observations, others are relying on implementation logs, and some use both. There is a relatively larger literature on measurement of implementation in school-based programs, yet research is needed to identify the most valid and useful methods for measuring implementation of these programs. For parenting programs, the variability in measurement reflects differences in delivery modes, such as computer-based, self-guided delivery versus group-based. Nonetheless, some variability in measurement of implementation is to be expected, as a valid measurement strategy needs to be consistent with the unique program features it is designed to assess. The measurement strategy also needs to yield useful data to inform quality improvement efforts and reflect a valid assessment of implementation quality. This is best done by retaining some level of flexibility in measurement to match the program characteristics and delivery methods.

There are additional gaps to be filled in the literature on implementation fidelity that have important implications for the implementation of EBPs in communities. First, it is not clear how to best conduct quality improvement efforts and use implementation data effectively to improve delivery and fidelity. The information from the YVPCs lacked specifics on how implementation data were used to inform quality improvement efforts, yet this information is critical in informing future efforts to implement evidence-based programs in real-world settings. There was also a lack of information on how measurement and tracking was embedded in the delivery systems for programs. Several of the measurement strategies appeared to rely on research staff to collect implementation data, and it was unclear where the implementation data were housed, whether within the YVPCs’ research units or with the partners implementing the interventions. It is important to understand the process for carrying out implementation measurement to know to what extent these processes were embedded within the implementation delivery system or whether they were kept separate from implementation delivery. This is a critical issue to address future efforts for sustaining implementation and continuing to improve the quality of implementation efforts.

As is often the case with implementation science in the context of prevention, the work of the YVPCs highlight the need to use data to understand the relationship between implementation of EBPs and youth violence outcomes. The data provide valuable information not only for the researchers in the academic-community partnerships at the core of the YVPC work. They also establish a foundation to build the capacity of communities to implement EBPs with quality. For several of the communities, the partnership with YVPCs constitutes their first attempts at implementing youth violence prevention programs based on scientific evidence. As with any innovation, there is a learning curve to good-quality implementation of EBPs, whether they are school-based curricula, family programs, or community and place-based strategies. The implementation data collected by the YVPCs provides community partners and implementers ongoing data to inform their efforts. These data are used to establish benchmarks and goals to improve implementation quality and continue to build on the foundation established by the YVPC-community partnerships. The ultimate goal of the YVPCs is to establish models for building community capacity to implement EBPs and prevent violence within high-risk communities. The implementation work taking place at the centers demonstrates the critical value in understanding not only the outcome in the prevention equation, but also the process of implementation and the opportunities to build the infrastructure necessary to use data to guide implementation.