Establishing and Evaluating the Key Functions of an Interactive Systems Framework Using an Assets-Getting to Outcomes Intervention

Abstract

Community practitioners can face difficulty in achieving outcomes demonstrated by prevention science. Building a community practitioner’s prevention capacity—the knowledge and skills needed to conduct critical prevention practices—could improve the quality of prevention and its outcomes. The purpose of this article is to: (1) describe how an intervention called Assets-Getting To Outcomes (AGTO) was used to establish the key functions of the ISF and present early lessons learned from that intervention’s first 6 months and (2) examine whether there is an empirical relationship between practitioner capacity at the individual level and the performance of prevention at the program level—a relationship predicted by the ISF but untested. The article describes an operationalization of the ISF in the context of a five-year randomized controlled efficacy trial that combines two complementary models designed to build capacity: Getting To Outcomes (GTO) and Developmental Assets. The trial compares programs and individual practitioners from six community-based coalitions using AGTO with programs and practitionersfrom six similar coalitions that are not. In this article, we primarily focus on what the ISF calls innovation specific capacity and discuss how the combined AGTO innovation structures and uses feedback about its capacity-building activities, which can serve as a model for implementing the ISF. Focus group discussions used to gather lessons learned from the first 6 months of the AGTO intervention suggest that while the ISF may have been conceptualized as three distinct systems, in practice they are less distinct. Findings from the baseline wave of data collection of individual capacity and program performance suggest that practitioner capacity predicts, in part, performance of prevention programs. Empirically linking practitioner capacity and performance of prevention provides empirical support for both the ISF and AGTO.

Introduction

Youth are exposed to a variety of risk factors that influence outcomes about whether they become healthy adults (Hawkins et al. 1992). Prevention programming can improve these outcomes and recoup costs (Miller and Hendrie 2009), but needs to be comprehensive and implemented with quality (e.g., with fidelity, tracking outcomes, using data to continually improve) to reap these benefits (Backer 2001). However, community practitioners can face difficulty in implementing high-quality prevention and achieving outcomes demonstrated by prevention science for many reasons. High quality prevention can be complicated and requires many steps to reach outcomes. Many evidence based programs have only been tested under ideal research conditions and lack dissemination supports for communities. However, a significant “gap” between science and practice (e.g., Wandersman and Florin 2003; Green 2001) can result when practitioners themselves lack the capacity (knowledge and skills) needed to conduct critical prevention practices that lead to the successful adaptation and implementation of “off the shelf” programs. Common approaches to bridging this gap, such as information dissemination, fail to change practice or outcomes at the local level in part because they do not sufficiently address capacity. Also, many programs focus only on improving youth deficits, despite evidence showing the need for complementary efforts to promote positive youth development (Benson 2002; Scales 1999). Therefore, building a community organization’s prevention capacity, with a focus on positive youth development (not just reducing risk), could improve the quality of prevention and outcomes.

The Interactive Systems Framework for Dissemination and Implementation (ISF) (Wandersman et al. 2008) offers a framework to build capacity in order to narrow the gap between science and practice; however, policy makers, researchers and communities need practical guidance on how to make use of its systems to support the movement of knowledge into practice. The ISF has three systems: The Prevention Delivery System, which includes local practitioners responsible for implementing and evaluating community-based interventions; the Support System, which includes providers of training and technical assistance (TA) who support the Delivery System in implementation and evaluation activities; and the Synthesis and Translation System, which includes researchers, program developers, and others that distill information about innovations and prepare them for wider dissemination and implementation.

The purpose of this article is to: 1) describe how an intervention called Assets-Getting To Outcomes (AGTO) was used to establish the key functions of the ISF (e.g., to improve practitioner capacity) and present the early lessons learned from such efforts and 2) examine whether there is an empirical relationship between practitioner capacity at the individual level and the performance of prevention at the program level—a relationship predicted by the ISF but is untested. Findings in this article come from the first 6 months of a randomized controlled trial of AGTO (Fisher et al. 2006) currently in 12 communities in Maine. AGTO is a combination of two innovations specifically designed to enhance prevention capacity: Getting To Outcomes® (GTO®) and Developmental Assets™. The two are complementary: the GTO intervention enhances local capacity and empowers practitioners to complete critical prevention tasks (e.g., planning, implementation, evaluation); the Assets approach supports community mobilization and collaboration to promote positive youth development. In this article, we primarily focus on what the ISF calls innovation specific capacity, or the specific motivation and skills (individual level) and human, technical, and fiscal conditions (organizational level) necessary to successfully implement a particular innovation (Wandersman et al. 2008)—in this case AGTO. We discuss how the combined AGTO innovation structures and uses feedback about its capacity-building activities, which can serve as a model for implementing the ISF. We present lessons distilled from focus group discussions that were held among staff implementing the AGTO intervention in its first 6 months. In addition, we present findings from the baseline wave of data collection to examine the degree to which practitioner capacity predicts performance of prevention programs. Demonstrating such a relationship would provide empirical support for the ISF and AGTO.

Getting To Outcomes®¹ (GTO) with Developmental Assets™²

Below we describe in more detail the two components of the combined innovation: (1) Getting To Outcomes (GTO); and (2) Developmental Assets.

Getting To Outcomes

GTO® is an implementation model—specifying the ten steps (or sets of activities) prevention practitioners should take when carrying out high-quality programming. GTO presents each step as a question which can be used to guide practitioners’ to complete those steps (Chinman et al. 2008, see Table 1). There are six steps for planning activities (steps 1–6), two steps for process and outcome evaluation (7–8), and two steps on the use of data to improve and sustain programs (9–10).

Table 1.

The 10 GTO steps and information presented in the Assets-Getting To Outcomes Manual

What are the needs to address? GTO Step #1 provides information about conducting an assets-based needs assessment What are the goals & objectives? GTO Step #2 has worksheets for creating measurable goals & objectives that focus on building assets among youth Which evidence-based programs can be useful in reaching the goals? GTO Step #3 overviews the importance of using evidence-based programming, that successfully build assets What actions need to be taken so the selected program fits the community context? GTO Step #4 prompts readers to reduce duplication and facilitate collaboration with other asset-promoting efforts What capacity is needed for the program? GTO Step #5 prompts readers to ensure there is sufficient organizational capacity to conduct the program What is the plan for this program? GTO Step #6 presents information and tools to plan activities How will implementation be assessed? GTO Step #7 provides information/tools to do process evaluation How well did the program work? GTO Step #8 presents information/tools to do outcome evaluation How will CQI strategies be incorporated? GTO Step #9 prompts practitioners to reassess Accountability Questions 1–8 after completing the program to improve the program If the program is successful, how will it be sustained? GTO Step #10 presents several ideas to consider when attempting to sustain an effective program

Getting To Outcomes is also an intervention that helps community practitioners apply the GTO steps, and build the knowledge and skills they need to answer those ten questions with quality—i.e., to perform each set of activities as close to optimal as possible. The intervention has three types of assistance: (1) the GTO manual of text and tools originally published by the RAND Corporation (Chinman et al. 2004) and adapted and integrated with the Developmental Assets approach by Search Institute (Fisher et al. 2006) (synthesis and translation in ISF terms); (2) face-to-face training, and (3) onsite technical assistance (TA; Training and TA are aspects of the Prevention Support System in ISF terms). There are other models that aim to build capacity—e.g., in the area of alcohol and drug prevention (e.g., Communities That Care: Hawkins et al. 2009 and PROmoting School-community-university Partnerships to Enhance Resilience: Spoth et al. 2007). Unlike these interventions, which provide support and resources to help community practitioners select and begin implementing evidence-based prevention programs, GTO engages existing programs (from evidence-based to homegrown) and the leadership of organizations which house these programs to improve their quality through support with no additional resources for implementation. As such, the goal of the GTO intervention is to help the leadership of community organizations to integrate the practices GTO targets into routine operations, closing the gap between research and practice. In a small quasi-experimental trial (Chinman et al. 2008) and a larger trial that had quasiexperimental and randomized arms (Chinman et al. 2009), the GTO intervention was found to help individual program staff improve their capacity (knowledge, skills) and their performance of prevention (how well programs were rated performing various prevention tasks) more than the comparison programs did.

Combining GTO with Developmental Assets™

Search Institute has identified 40 key developmental supports and experiences (called “Developmental Assets”, which include but are broader than the notion of protective factors) that young people need to thrive. There are 20 external assets (e.g., health-promoting features of the environment) grouped into four categories: (a) support, (b) empowerment, (c) boundaries and expectations, and (d) constructive use of time and 20 internal assets (e.g., individual commitments, values, and competencies, and self-perceptions) grouped into four categories: (a) commitment to learning, (b) positive values, (c) social competencies, and (d) positive identity. From a caring school climate to participation in high-quality after-school programs, these factors have been shown to predict health and well-being outcomes consistently across sex, race-ethnicity, and family income (Benson et al. 1999; Leffert et al. 1998; Scales et al. 2000). For example, in their study of nearly 100,000 6th-12th graders, Benson et al. (in press) found that the higher the number of Assets that students reported experiencing, the less likely they were to report engaging in a variety of high-risk behaviors. To help youth receive these Assets, Search Institute uses a community mobilization and planning process to engage a wide range of individuals, organizations, and systems (Benson et al. 2003). This community mobilization and planning process was integrated with the 10 GTO steps, so that each step was enhanced to include a specific focus on building Assets. For example, during Step 1 (Choose which problem[s] to focus on) individuals utilize both asset and thriving indicators, as well as risk and deficit information to help select priorities for attention (Table 1).

Theory of AGTO

Similar to the generic GTO model, AGTO is an operationalization of Empowerment Evaluation theory (Fetterman and Wandersman 2005), which posits a greater probability of achieving positive results when evaluators collaborate with practitioners implementing positive youth development programs and provide them with tools and opportunities to plan, implement with quality, evaluate outcomes, and use a continuous quality improvement system themselves. As such, AGTO also has roots in traditional evaluation and results-based accountability (Wandersman et al. 2000). Collectively, we have defined the knowledge and skills related to the activities targeted by AGTO’s 10 steps as prevention “capacity” at the individual level (Chinman et al. 2008), which is posited to be related to how well prevention is carried out at the program level (Chinman et al. 2005). Consistent with social cognitive theories of behavioral change (Fishbein and Ajzen 1974, 1975; Ajzen and Fishbein 1977; Bandura 2004), we hypothesize that among community practitioners, exposure to AGTO training and TA leads to more knowledge about performing AGTO-related activities, which in turn leads to improved attitudes towards these activities, which in turn leads to the performance of more AGTO-related behaviors. The hypothesized relationship between knowledge, attitudes, and performance is also an untested premise of the ISF.

The Delivery System: 12 Coalitions and Their Programs

The research participants in this study are practitioners from 12 community-based prevention coalitions in Maine. Community-based coalitions are popular public health promotion mechanisms, simultaneously intervening across multiple levels (individual, organizational, policy) and sectors (parents, youth, criminal justice, and education) in order to improve community health (Butterfoss et al. 1993). Their broad focus also makes them a popular mechanism for promoting positive youth development, which necessarily involves a range of sectors focused on physical, social, psychological, and intellectual needs of youth (Kegler and Wyatt 2003). The 12 coalitions are similar in that they all have a core group of paid staff supporting a volunteer base. They operate in similar geographic and demographic settings³ and have comparable rates of alcohol and other drug use among youth.⁴ Each receives a small amount of funds from state and federal agencies to diagnose and prioritize community needs and assets related to positive youth development, implement programs to address those needs using community assets, and then evaluate progress. The coalitions have similar annual budgets and a similar number of distinct programs, although the programs themselves differ. Their programming can be characterized broadly as positive youth development programming with middle and high school youth, using some evidence-based and mostly locally developed programs. For example, across all 12 coalitions, less than 10 % of the programming mix in each coalition is evidence-based programs such as Reconnecting Youth (Eggert et al. 1994), Lifeskills Training (Botvin et al. 1995), and the RealCare Parenting Program (Somers and Fahlman 2001). The locally developed programs include mentoring, social norm campaigns, juvenile justice diversion programs, and leadership training (see Table 2). The wide variety of programs is a hallmark of AGTO: coalition and organization leaders can choose which program(s) best fits their general organizational capacity and community needs. As a result, most programs included in the study are different from one another. Community practitioners from the 12 coalitions had no prior exposure to AGTO.

Table 2.

Descriptions and categorizations of participating programs

Program purpose	Program name (n = 50)
Diverting youth from  the juvenile justice  system	Diversion to Assets (2 sites) Youth Diversion Program Asset Building Committee Resource  Project Boomerang Program
Empowering youth	Opportunity for Teens and K-Club Youth Advocacy Program (4 sites) Youth Empowerment Program Youth Summit Community Conversations Youth Empowerment Through  Employment Natural Helpers (2 sites) Girls Circle
Enhancing youth  leadership	Core of Leaders Youth on Boards Youth Philanthropy Project YMCA Learning Leadership  Program Leadership & Resiliency Program Voices Committed to Change
Providing recreational  opportunities	ArtsNKids The Game Loft Ice Rink Committee Challenging Choices Summer Recreational Program Camp Kiev
Building academic and  vocational skills	21st Century/Afterschool Program Entrepreneurial Program Shaw House Day Program Afterschool Aspirations Programs Project Succes
Building positive  relationships with  adults	Boys and Girls Club Mentoring  Program YMCA Boys Mentoring Big Brothers/Big Sisters Mentoring  Program Positive Ticketing Colby Cares About Kids
Preventing risky  behaviors (substance  abuse, violence, teen  pregnancy)	Communities Mobilizing for Change  Against Alcohol (2 sites) Life Skills Project ALERT New Chance Reconnecting Youth My Attitude Saves Kids The Tool Shed Project RealCare Parenting Program
Improving parenting  skills	Good Samaritan Parent Program Parent Education

An Example of an Operational ISF: The AGTO Intervention

To support the positive youth development programming consistent with the Developmental Assets model, weoperationalized the three components of the ISF below (summarized in Table 3). In addition, we also included a “Community Research Workgroup”, which in our project has been serving as a forum for all three components to interact at one time. The description below is based on what we established during the first 6 months of the AGTO intervention.

Table 3.

Functions, measures, and feedback mechanisms utilized by the AGTO Project within each ISF system

ISF systems	Functions and feedback mechanisms
Prevention delivery system	Plan, implement, and evaluate Assets-based programs using GTO 10 steps
Provides feedback to support system a	Postcard survey of prevention practitioners satisfaction Post-training evaluation survey
Prevention support system	Deliver annual AGTO training for prevention practitioners Provide bi-weekly technical assistance to prevention practitioners Conduct quarterly in-person meetings with practitioners to review program progress and plan activities  for the next quarter Provide ongoing support to TA providers across AGTO model Provide monthly professional development training to TA providers (logic modeling, process and  outcome evaluation, continuous quality improvement) Conduct quarterly in-person meetings with TA providers to review program progress and plan activities  for the next quarter
Provides feedback to synthesis and   translation system	TA monitoring form housed on a shared website Weekly phone conference with TA providers Weekly phone conference with project leadership team
Prevention synthesis and translation  system	Create a repository of process and outcome evaluation measures that could be used by programs Create detailed discussion guides that contained key questions and tips for TA providers and for  practitioners to utilize as they implemented GTO steps 7 (Process Evaluation), 8 (Outcome  Evaluation), and 9 (Continuous Quality Improvement)
Provides feedback to support and   delivery systems	Weekly phone conference with project leadership team to discuss project measures, including   evaluation design and analysis for individual programs Bi-monthly meetings of the Community Research Workgroup

^aItalicized text refers to measures and processes by which feedback has been provided to other systems within the ISF

Prevention Delivery System

The community practitioners and the programs from the 12 coalitions comprise the Prevention Delivery System. These practitioners are responsible for planning, implementing, and evaluating Assets-based programs using the AGTO 10 steps. The Prevention Delivery System has been receiving feedback in several ways. First and foremost, the TA providers have been giving practitioners feedback about their programs in relation to the AGTO 10 steps on an ongoing basis. Second, three annual assessments of capacity (Coalition Survey of all coalition members) and performance (an interview with the directors of all the programs) are conducted.⁵ To date, reports from the baseline administrations of both of these measures were presented, discussed, and used in planning at meetings of coalition representatives (Community Research Workgroup, see below). These measures are discussed in more detail in the method section of this paper.

Prevention Support System

Training and TA comprises the Prevention Support System. TA providers delivered a full day of training at baseline to each coalition and have been making bi-weekly visits, providing consultation and feedback to practitioners on conducting tasks in accordance with the AGTO 10 steps. The consultation method could be considered “facilitation,” in which changes are stimulated through encouragement, clarification of the tasks that need to be completed, and promotion of actions needed to make improvements (Kitson et al. 1998; Rycroft-Malone et al. 2002; Stetler et al. 2006). Practitioners have been interfacing with the TA providers by making TA requests, sharing results of prevention activities, and informing TA providers about local circumstances.

Based on the TA literature (Chinman et al. 2005; Mitchell et al. 2004; O’Donnell et al. 2000; Stevenson et al. 2002), the AGTO TA process involves three structured steps, but allows the practitioners to guide the direction. First, TA providers collected background information from coalitions on each of their programs’ (e.g., targetpopulation, purpose, type, funding, prior TA). Second, TA providers and practitioners jointly developed a program logic model to identify where the program stood in following each of the AGTO 10 steps. Then, third, on a quarterly basis, TA providers and practitioners have been revisiting the logic model and jointly developing a plan for making improvements to their programs for the upcoming quarter. Between visits, practitioners and TA providers have been interacting as needed and have access to all project documents stored on a website.

Using a tiered Prevention Support System, TA providers have been receiving supervision weekly by phone and quarterly via in-person meetings with experts in the GTO and Assets models. In weekly meetings, the TA supervisors have been providing enhanced support for TA providers by fielding coalitions’ requests for assistance beyond the expertise of the TA providers and troubleshooting day to day issues. In the quarterly meetings, each program is being discussed in detail and a summary of the progress, next steps, and ultimate goals for each program are being transcribed into a plan for the upcoming quarter. The supervisors have also been providing monthly trainings to TA providers to build their capacity to serve communities. These trainings have covered topics such as advanced logic modeling, how to design a process evaluation, how to interpret evaluation analyses, and how to utilize process and outcome evaluation for continuous quality improvement.

The larger project leadership team has also been providing guidance to the TA providers and supervisors during weekly team meetings. A separate weekly meeting was created as an opportunity for TA supervisors and the larger project leadership team to help TA providers address the coalitions’ questions about process and outcome evaluation measures and how to use evaluation findings for program improvement according to the AGTO model. The project leadership team includes broad representation from the all study partners including (1) the RAND Corporation, which co-leads the project and provides the survey and data analysis capabilities, (2) Communities for Children and Youth, which co-leads the project, provides leadership to the Community Research Workgroup, and recruited the 12 coalitions from among their communities, (3) Search Institute, which provides expertise in the Developmental Assets model, (4) Vision Training Associates, the training provider for the Search Institute which provide the TA supervisor with expertise in Developmental Assets; (5) University of Southern Maine, which provides leadership to the Community Research Workgroup; and (6) the University of South Carolina, which provides expertise in statistical analysis.

Feedback about the Prevention Support System has been generated in several ways by practitioners from theDelivery System, TA supervisors, and the Project Leadership Team:

• Practitioners have been completing a postcard survey quarterly about their satisfaction with TA (1 = Very Dissatisfied to 5 = Very Satisfied); use of AGTO (1 = Never to 5 = Daily); and improvement in knowledge (1 = Not a lot to 5 = A lot).

• Training evaluations were completed by practitioners to assess their satisfaction with the content, format, and pace of the training.

• TA supervisors have been providing feedback about the quality of the Prevention Support System’s TA through the through weekly meetings between TA providers and supervisors.

• TA supervisors and providers have been using a TA Monitoring Form—a shared database that tracks TA providers’ activities, questions, and TA requests by each of the 10 AGTO steps. TA providers enter notes from their bi-weekly meetings, time spent providing TA to programs by AGTO step, questions, and TA requests (e.g., need to develop an outcome evaluation instrument, need to synthesize the literature on best practices to improve school climate).

• TA supervisors have also been tracking the programs’ TA Utilization Index, the proportion of TA hours that the coalition is receiving (tracked through the TA Monitoring Form) over the total number of TA hours available. Each program’s Utilization Index score has been shared with both TA providers and practitioners working on that program as a way to encourage practitioners to utilize TA.

Prevention Synthesis and Translation System

The tools contained in the Search Institute-published manual, Getting To Outcomes with Developmental Assets: Ten steps to measuring success in youth programs and communities (Fisher et al. 2006) are the crux of AGTO Synthesis and Translation System. The manual contains both examples from past project work (e.g., logic models) and newly developed tools organized around each of the 10 steps. The manual has been enhanced by the addition of several new tools developed or adapted by TA supervisors to support the TA providers’ work (Support System), in particular in assisting with program evaluation. For example, TA supervisors have created a bank of generic process and outcome questions communities could use. Also, the TA supervisors have created detailed discussion guides of key questions and tips that both TA providers (Support System) and community practitioners (Delivery System) are utilizing as they implement AGTO steps 7 (Process Evaluation), 8 (Outcome Evaluation), and 9 (Continuous Quality Improvement). The evaluation discussion guide asks questions about the purpose of evaluation surveys, preferred length, plans for data collection and how practitioners will use the data, and with whom practitioners plan to share data. A Continuous Quality Improvement (CQI) discussion guide has also been created for TA providers and practitioners and presents questions that help practitioners use process and outcome evaluation to make decisions about program improvement strategies (e.g., What has significantly improved? Did any of the scales that you expected to improve NOT improve? If so, why do you think this happened [e.g., program content was not delivered as intended, attendance for the program was low?). The CQI discussion guide is accompanied by worksheets that help practitioners summarize lessons learned and plan improvement.

Community Research Workgroup: An Interface of All Three ISF Components

To date, the three components in the ISF—the practitioners (Prevention Delivery System), TA providers, TA supervisors and the project leadership team (Prevention Support System), and the AGTO manual and tools (Prevention Synthesis and Translation System)–all have been interacting during bi-monthly in-person meetings of the Community Research Workgroup (CRWG) in the following ways: (1) support personnel have provided further, targeted training of practitioners (Delivery) to enhance specific capacities (e.g., youth engagement) and their use of AGTO tools (Synthesis and Translation); (2) practitioners (Delivery) have offered feedback to the larger project leadership team (Support) about ways to improve the AGTO intervention (Support) and tools (Synthesis and Translation); (3) the project leadership team (Support) has shared data about the performance of the participating practitioners (Delivery), discussed collaboratively with practitioners what the data mean, and have offered recommendations to practitioners about ways to improve their practice; and (4) practitioners (Delivery) have used the group setting for peer-to-peer learning (i.e., establish a “community of practice” among participating practitioners).

In summary, the AGTO intervention is a unique operationalization of the ISF and offers rich opportunities to research the processes and outcomes achieved by using the ISF model to assist community organizations in the use of quality prevention programming. Specifically, we hypothesize that use of capacity supports from AGTO will improve program and subsequently youth outcomes by strengthening capacity of individual practitioners. 6 months into a 2 year intervention, we present lessons distilled from focus groups that were held among staff; and findings from the baseline wave of data collection to examine the degree to which practitioner capacity predicts performanceof prevention programs. The link between individual capacity and program performance at baseline (dashed oval in Fig. 1) is a key relationship in the ISF that to date has not been assessed.

Fig. 1.

Logic model for the AGTO study and focus of the analyses

Methods

Study Design

The AGTO study uses a five-year cluster randomized controlled trial design (Donner and Klar 2004) to assess the implementation and impact of the 2-year AGTO intervention. Community practitioners from six alcohol and drug prevention coalitions across the state of Maine who receive AGTO are compared to another six in Maine who do not. Randomization was done using matched pairs of coalitions. The coalitions were matched based on the total population and demographic characteristics from the 2000 US Census for the community associated with each coalition and the TA staff’s initial rating of coalition functioning using a measure developed by Office of National Drug Control Policy’s in the evaluation of the Drug-Free Communities program (Battelle 2008). Leaders from each of the 12 coalitions nominated five programs to participate in the study (30 AGTO, 30 control). Each coalition receives $3,000 a year to defray the cost of the research participation. All participants gave written consent. The study was approved by the RAND Corporation’s IRB. The primary measures are at two levels: program performance (program level) and prevention capacity of community practitioners (individual level, see below for a description of these measures). Each of those measures are assessed at baseline, midpoint (after 1 year of AGTO implementation), and posttest (after 2 years of AGTO intervention). Youth outcomes are also being measured through a Maine-sponsored middleschool survey. As shown in the logic model for the AGTO study (Fig. 1), we hypothesize that the individual capacity of community practitioners influences the quality of program performance, given certain characteristics (e.g., funding and organizational climate), and then ultimately youth outcomes. This article focuses on results from the baseline capacity and performance assessments and qualitative findings from focus group discussions 6 months into the intervention (dashed oval).

Measures and Data Collection

Program Performance (Prevention Delivery System: Program Level)

Drawing on previous GTO research (Chinman et al. 2008, 2009), we are using a structured interview to assess performance of tasks associated with high-quality prevention. Although programs consist of individual people with varying levels of abilities, performance ratings are made at the program level, because programs operate as a unit. The Performance Interview protocol is used to gather data needed to make two sets of ratings assessing performance of activities targeted by the 10 steps of the AGTO model: (1) 14 Likert items (or “components”) that broadly correspond to the activities of the AGTO steps, averaging to a Components Total Score. Each component has seven response choices, described with specific, observable behaviors, that range from “highly faithful” to the ideal prevention practice to “highly divergent” from ideal practice. In a previous evaluation of GTO (Chinman et al. 2008), the Components Total Score was sensitive to change, as the amount of change was highly correlated with the amount of TA hours (r = .59). The average inter-rater reliability for the Components Total Score was .74. Inter-rater reliability for each component ranged from .65-.96. (2) A checklist of 76 Yes/ No items, developed for this study assessed the more micro activities that comprise the AGTO steps.

Assets-Getting To Outcomes research staff conducted the structured interviews by telephone with the program directors of 51 programs⁶ and used those interviews to make the ratings across each of the Performance components and checklist items (100 % of all operating programs). All interviews were digitally recorded. A second AGTO staff person double-rated 10 % of the programs on both measures by listening to audio recordings. Inter-rater reliability across all the items in the checklist, measured by Kappa and Percent Agreement, was .81 and 91 % respectively. Inter-rater reliability across the 14 components, which requires more rater judgment, was .59 and 79 % respectively. In cases where there were discrepancies, the two raters discussed the scoring until a consensus was achieved.

Funding Stability (Prevention Delivery System: Program Level)

A third program-level measure, developed for this study, was a categorical rating of each program’s funding stability, defined as prospects for continued funding through June 2010, mid-point for the AGTO intervention (which continued until June 2011). Program funding prospects were rated by TA providers as “high” (strong likelihood of continued funding), “medium” (possibly defunded), or “low” (strong likelihood of being defunded) based on information collected at the start of the AGTO intervention as part of the TA needs assessment. This measure was developed because it is widely believed that any analysis ofcapacity and program performance ought to take funding resources into account. However, given that programs’ funding was so fluid, a simple rating of funding stability was believed to be more useful in the analyses.

Prevention Capacity (Prevention Delivery System: Individual Level)

We used the AGTO Coalition Survey to assess baseline individual practitioner prevention knowledge and skills, defined here as prevention capacity (1 and 2 year follow-up assessments are forthcoming). The Knowledge Score is the mean of seven items assessing the degree to which a respondent knew enough to carry out various prevention activities targeted by AGTO in the last 12 months (needs assessment, setting goals and objectives, using evidence-based practices, determining fit, conducting process and outcome evaluations, engaging in continuous quality improvement activities, and program sustainability) with a three-point response scale (1 = “would need a great deal of help to carry out this task”, 2 = “could carry out this task, but would need some help”, 3 = “could carry out this task without any help”), alpha = 0.84. The Skills Score is the mean of six items with a seven-point response scale (1 = “never” to 7 = “very often”) assessing respondents’ frequency of these same prevention activities during the 12 months before the survey, alpha = 0.92. Since AGTO is an innovation, according to the ISF, these capacity items represent innovation-specific capacity.

Perceptions of the Coalition (Prevention Delivery System: Individual Level)

Additional measures from the Coalition Survey used in the analysis include ratings of coalition leadership and receptivity to change, both of which have proven to be important in the functioning of coalitions and in the incorporation of new practices. The Leadership Score (Weiss et al. 2002) is the mean of ten items rating the overall effectiveness of coalition leadership from five response options (1 = “poor” to 5 = “excellent”), alpha = 0.95. The Receptivity to Change Score was a measure adapted from the Staff Survey of Organizational Readiness for Change (Lehman et al. 2002) and is the mean of five items with five response options (1 = “strongly disagree” to 5 = “strongly agree”), alpha = 0.73. Also, respondents were asked to select a program in which they were involved from a list of the participating programs, if applicable (many practitioners participate in general coalition activities not tied to any of the specific programs we were working with on the AGTO project). Of 376 respondents, 196 (52 %) selected a specific program. Scale scores were aggregated across respondents involved in each program. According to the ISF, these measures represent general organizational capacity.

The baseline Coalition Survey was a self-administered pen-and-paper survey conducted with the intervention and control coalitions just prior to the start of the AGTO intervention. The survey was conducted in-person by a trained survey facilitator (typically one of the TA providers) in various coalition meetings whenever possible or by mail for the rest. Across all sites, the response rate was 82 %; for intervention coalitions it was 91 %; and among the control coalitions it was 74 %. We have stronger relationships and more contact with the intervention coalition members which could have accounted for their higher response rate. The within-coalition response rate ranged from 71 to 93 %.

Focus Groups (Prevention Support System/Prevention Synthesis and Translation System: Individual Level)

To identify lessons learned during the implementation of AGTO in the first 6 months, we hosted two semistructured focus group discussions with TA providers, TAsupervisors, and the Project Leadership Team. The first centered on interpreting findings from the statistical analysis (described below) to generate some reflections and explanations for the baseline survey and interview findings. The second focused on the strategies for Prevention Support and Prevention Synthesis and Translation that have been working the most effectively among coalitions and programs. Each of these discussions lasted approximately 45 min and was facilitated by the Project Director. Notes were taken by two separate recorders during the meeting to ensure all details of the conversation were captured, and then combined to form a single transcript of the discussions.

Data Analysis

Comparing Intervention and Control Groups at Baseline

We first assessed any baseline differences between practitioners in the intervention and control coalitions on their Knowledge and Skills scores through a multi-level regression with intervention as a predictor variable, adjusting for paired randomization and clustering within coalitions.

Model of Capacity Predicting Performance

Although this is a baseline analysis and does not actually test the effectiveness of the Support System, testing the hypothesis that prevention capacity (knowledge and skills) predicts prevention performance is important to the ISF because it underscores the importance of the Support System, whose purpose is to build capacity. To do that, we used data pooled across the 12 coalitions to fit two separate linear regression models—one predicting each of the two program performance measures (Components and Checklist Total Scores)—from individual capacity measures (Knowledge and Skills Scores), adjusting for select covariates (Fig. 1’s dashed oval). The covariates included the Leadership Score and Receptivity to Change Score, and the categorical rating of a program’s Funding Stability. Of 51 programs involved in the AGTO study, one did not have funding information available, and two programs were not designated by any survey respondents, leaving 48 programs in the analytic dataset. Continuous variables (all except funding) were standardized to have a mean of zero and variance of one. Indicators of medium and low Funding Stability were entered into the models, with high stability being the reference group. A random intercept for coalition was added to the regressions to adjust for correlation of the performance of programs within a given coalition. Regressions were fit using PROC MIXED using SAS 9.22.

Focus Groups

A transcript combined from both discussions was analyzed using constant comparative analysis to identify common themes or “lessons learned” from the first 6 months of implementation (Glaser 1965). Lessons learned were extracted and then shared back with focus group participants for confirmation.

Results

Comparing Intervention and Control Groups at Baseline

On Knowledge, the intervention (M = 2.26, SD = 0.44) and control groups (M = 2.18, SD = 0.45) were similar at baseline [t(5) = 1.57, p = .18]. Skills scores between the intervention (M = 4.49, SD = 1.45) and control groups (M = 3.92, SD = 1.57) differed significantly [t(5) = 3.857, p = .01].

Model of Capacity Predicting Performance

Unadjusted means and frequencies of all study variables are presented in Table 4. Program level means and frequencies are from respondents from all 12 coalitions, regardless of intervention arm. The Individual level variables have been aggregated to fit to the program level. As shown in the table, the measures of performance and capacity at baseline are moderate, falling at about the midpoint of the possible range.

Table 4.

Unadjusted means and frequencies of study variables

	Mean or %	SD	Range
Program-level variables (possible range)
Checklist total scorea	37.7	11.7	14.0–71.5
Components total score (0–70)	36.0	8.9	23.0–58.0
High funding stability	56 %	–	–
Medium funding stability	24 %	–	–
Low funding stability	20 %	–	–
Individual level variables aggregated to program level (possible   range)
Knowledge score (1–3)	2.3	0.2	1.7–2.9
Skills score (1–7)	4.6	0.8	2.3–6.0
Receptivity to change score (1–5)	3.9	0.3	3.2–4.5
Leadership score (1–5)	4.0	0.5	2.8–4.9

^aThere is no range possible because some items have no upper bound

The Knowledge Score was significantly and positively related to both program performance measures (Components and Checklist Total Scores). An increase of one standard deviation on the Knowledge Score was associated with an average increase of 0.30 standard deviations on the Checklist Total Score and 0.29 standard deviations on the Components Total Score. The Skills Score was not related to either the Components or the Checklist Total Scores of program performance.

Programs with medium Funding Stability averaged 0.78 standard deviations below programs with high Funding Stability on the Checklist Total Scores but was not significantly related to the Components Total Score, p < 0.05. The remaining two variables in the models (Leadership Score, Receptivity to Change Score) were not significantly related to either the Components or the Checklist Total Scores of program performance. In summary, our analyses suggest that greater knowledge about prevention activities AGTO targets (innovation specific capacities) and funding (general organizational capacity) were related to betterprogram performance, although greater skills was not (Table 5).

Table 5.

Regression coefficients from models predicting coalition performance measures from capacity scales and select covariates

Effect	Model 1: Checklist total			Model 2: Components total
	Estimate	SE	Pr < |t|	Estimate	SE	Pr < |t|
Knowledge score	0.30	0.14	0.0457	0.29	0.14	0.0442
Skills score	−0.15	0.18	0.3951	−0.17	0.17	0.3365
Receptivity to change score	−0.24	0.19	0.2097	−0.23	0.18	0.2246
Leadership score	0.18	0.18	0.3259	0.00	0.18	0.9827
High funding stability (ref.)	0.00	–	–	0.00	–	–
Medium funding stability	−0.78	0.35	0.0336	−0.60	0.34	0.0905
Low funding stability	−0.07	0.36	0.8518	−0.35	0.35	0.3173

Focus Groups

Below are lessons learned about how, within the first 6 months, the AGTO intervention has been used to operationalize the ISF. We present these lessons according to the three ISF components.

Prevention Delivery System

The first lesson is that, to date, the programs have varied widely in their ability to use the training and TA offered by the Prevention Support System. Some programs have been eagerly engaging with the TA providers—attending training and TA sessions regularly and performing work in between sessions—and others have not. TA staff observed that the largest barrier for practitioners using TA was funding for staff (general organizational capacity). For example, one participating program has less than $1,000 of direct funding, serves 12 youth, with one practitioner working for 2 h a week. Across all programs assigned to AGTO, less than 5 % has funding that is expected to be stable throughout the entire two-years of the intervention and less than 3 % has one or more full-time equivalent staff dedicated to the program. The second lesson follows from the first: given that practitioners in the Prevention Delivery System have been limited in their ability to utilize the assistance from the Support System, the Support System needs to conduct a thorough assessment of the needs, capacities, and resources of those in the Delivery System. While this was done as described above, the nature of the support being delivered through a research study (as opposed to a government or private TA provider) meant that all programs that agreed to participate have been engaged by TA staff in AGTO work, regardless of their initial capacity and resources.

Prevention Support System

The first 6 months of the AGTO intervention identified several key lessons about implementing a comprehensive Prevention Support System. First, the work of the TA providers has been relationship-driven and any gains in capacity have been mediated by the formation of a strong relationship. Second, TA providers observed that practitioners who participated in the initial training have had more enthusiasm for working with AGTO than those who did not. TA providers indicated that this early momentum has been an important foundation that allowed those practitioners to get started more quickly working on program logic models and activities related to the 10 AGTO steps.

A third lesson is that because TA is relationship-based, tensions have been emerging between TA (the Support System) and the other systems. For example, tensions have been emerging as TA providers (Support System) pushed practitioners (Delivery System) to engage in the core steps of the AGTO process. At times, practitioners have not had time, resources, capacity or interest to engage in the self-reflective activities outlined in the 10 AGTO steps or to make the sometimes difficult changes to improve their practice. There also have been tensions between the TA providers and the AGTO researchers (within the Support System) who at times desired that community practitioners use more of the AGTO resources offered by the Prevention Support System than was feasible given practitioners’ limited time, resources, and capacity. To navigate these barriers, TA providers have been providing feedback to the TA supervisors and the Project Leadership Team (enhanced Support System) about the on-the-ground circumstances that has been hindering practitioners’ progress and together they have been brainstorming new ways to help the Delivery System.

Fourth, focus group members identified a number of characteristics exhibited by the TA providers during the first 6 months of the AGTO intervention that greatly facilitated the technical assistance so far:

• Flexibility: TA providers found that they have needed to be available for impromptu or last-minute meetings and be willing to schedule meetings at off hours in order to reach all practitioners.

• Persistence: Repeatedly sending emails and calling practitioners to schedule meetings and following-up on agreed upon tasks has been helping TA providers maintain momentum and ensure consistent interaction with practitioners.

• Adaptive: When TA providers’ advice has been met with resistance because a task seemed too complex or time-consuming, they have narrowed the scope of work to make the next steps more concrete, tangible, and feasible for practitioners. Instead of pushing practitioners to work on multiple steps of the AGTO model at a time, TA providers have been focusing on a single step (e.g., Step 8: Outcome Evaluation) or a single activity within that step (e.g., developing an outcome evaluation instrument).

• Assertive: Providing TA requires an ability to persuade practitioners to make improvements. Finding the right balance between politeness and assertiveness can be difficult but has been critical to ensuring that TA remains focused and builds momentum needed for practitioners to improve their practice.

• Farsighted: TA providers commented that, early on, developing a strong sense of the goals and objectives for TA and which AGTO steps and tools are needed has optimized their efforts. TA providers have been working with several practitioners who have tended to focus on meeting immediate demands, but at the expense of longer-term planning. While addressing these immediate demands has been supporting relationship development, the TA providers’ long-term vision has been helping bring the focus back to the activities needed for the practitioners to meet their long-term plan for their programs.

• Non-judgmental: Practitioners reported being pleasantly surprised that the TA has been offered without judgment or requirements usually associated with a supervisor. Much of the TA that practitioners had received in the past had been delivered to provide support but also to monitor practitioners’ compliance with grant or contract requirements. Although TA providers have been making suggestions to encourage quality improvement, the TA still does not have any ‘strings attached’, which practitioners have found to be stressful and unhelpful in the past.

Synthesis and Translation System

During the first 6 months, several focus group members commented that in order to be most useful, the Delivery and Support Systems have needed to closely collaborate to shape synthesis and translation activities so that they are tailored to specific programs. For example, practitioners have been asking for more synthesis and translation than is available in existing literature summaries or online registries of evidence-based practices (e.g., National Registry of Evidence-Base Practices: http://nrepp.samhsa.gov/, Blue-prints for Violence Prevention: http://www.colorado.edu/cspv/blueprints/). In these cases, practitioners and TA providers have been working together to shape specific requests for synthesis and translation, namely a summary of literature that fits their own specific situation. From there, other members of the Support System have been developing literature summaries tailored to the specific situations and questions posed by the Delivery System (e.g., what are the best juvenile diversion programs that encourage youth leadership).

Discussion

In this article, we set out to describe the details, and with focus groups the early lessons learned, of an AGTO operationalization of the ISF. Using baseline survey and interview ratings data, we have also assessed whether the capacity of individual practitioners is related to the performance of programs in which those practitioners work. We discuss the baseline analyses and lessons learned below.

Model of Capacity Predicting Performance

Focusing on innovation-specific capacity of the Delivery System, our results found a partial relationship between greater capacity (i.e., knowledge) and better performance, while controlling for leadership, ability to change, andfunding stability (general capacities according to the ISF). These results suggest that using the ISF, as operationalized through AGTO, to build the knowledge of practitioners about how to conduct high-quality prevention (a component of innovation-specific capacity) can be useful in improving prevention. This result is consistent with social cognitive theories of behavioral change (Ajzen and Fishbein 1977; Bandura 2004; Fishbein and Ajzen 1974, 1975), in which knowledge (of community practitioners in this case) is a precondition for those practitioners to make behavior changes in their performance of prevention. It is also consistent with previous evaluations of the earlier version of AGTO (the generic GTO) in which improved individual capacity and program performance have both been noted (Chinman et al. 2008, 2009).

At baseline the individual level GTO Skills Score, which corresponds to skills needed to fulfill the 10 steps of GTO, did not predict program performance. This may be, in part, because this is the baseline analysis occurring before we implemented the AGTO intervention and because the Skills Score is a frequency measure (e.g., more evaluation is viewed as better) and the Components and the Checklist Total Scores emphasize quality over frequency. Lastly, less funding stability (general capacity) was related to weaker program performance. Practitioners in the Delivery System who do not believe they will continue to be funded in the near future may be less likely to engage in high-quality prevention work, suggesting that the typical situation of year-to-year funding can undermine the quality of prevention performance. This finding also begins to illuminate relationships between general and innovation-specific capacities within the Delivery System, suggesting that without the ISF’s general capacity (like financial resources), even the most promising of innovations will not be used.

Early Lessons Learned

Below we discuss results on implementation lessons reported above by each of the ISF systems and the Community Research Workgroup, which is an intersection of all three systems.

Prevention Delivery System

Regarding the lesson that programs vary in utilization of assistance, we believe there are three reasons for this. First, in the interval between agreeing to participate in the study and the study’s start date, some practitioners’ emphasis on positive youth development receded. For example, because of local funding priorities, some practitioners began to place more emphasis on environmental underage drinking prevention strategies, which emphasize law enforcement, rather than positive youth development. As a result, those practitioners have participated less in a positive youth development project like AGTO than they could have. Second, widespread funding cuts have forced many programs to cut staff, making them less available to participate in the AGTO work. Although funding and focus of work among programs and coalitions are contextual factors in the ISF that are “not the main focus of this Framework (Wandersman et al. 2008, p. 179)”, they clearly are important. Their role and impact on the functioning of the ISF and AGTO needs to be better understood in future research.

It may seem counterintuitive that programs that have had their resources cut have been less inclined to take advantage of the resources AGTO offers, but this situation is consistent with observations from other TA providers: it takes some initial general capacity to use capacity-building assistance (Mitchell et al. 2004; Chinman et al. 2005). Third, the participating coalitions are much more diffuse and have less formalized accountability structures than coalitions that have used GTO in past evaluations. For example, many of the practitioners implementing participating programs do not report at all (or very informally) to the coalition leaders with whom the original agreements to participate in AGTO were made. Given that a core tenet of AGTO is to support accountability with data, so far the utility of AGTO has been undercut in some programs where the practitioners implementing those programs are not held accountable for their performance.

Although AGTO took all comers regardless of initial capacity, in an ISF outside of a research study more care may be needed (in maximizing Support System resources) to determine if programs and the community organizations that house them are a good match to engage in capacity building. Alternatively, a Support System may want to better match the exact nature of the planned capacity building to the existing capacities and resources of the program/ organization in question. A phased approach could be used whereby a Prevention Support System initially enters into an agreement with programs/organizations that matches existing resources and capacity to a narrowly defined scope of improvement work (similar to a proactive TA plan), only expanding the scope after capacity improves. Also, the agreement would not only help the Support System maximize their training and technical assistance resources, but it could also educate the Delivery System about what general and innovation-specific capacity is needed to be successful. For example, a program/ organization may have resources to conduct a basic needs assessment and develop goals and objectives that they would like to achieve (and the Support System might work with them on those tasks—AGTO Steps 1 and 2), but might not have enough general capacity (staff, expertise, funding) to implement the programming needed to achieve those goals. Thus, the Prevention Support System work couldend there and restart once the program/organization has the necessary general capacity and resources. Examples of graduated capacity-building already exist—SAMHSA’s Service to Science Academy, for example, screens prevention programs wanting assistance through an application process, only accepting those that demonstrate a strong initial capacity.

Prevention Support System

Related to the lesson about the importance of relationships to implementation, others have noted that building trust and strong relationships between practitioners (Delivery System) and the TA providers (Support System) helped practitioners take risks associated with engaging in the difficult work needed to improve their prevention practice (Chinman et al. 2008; Hunter et al. 2009a, b). A strategy to establish trust between TA providers and practitioners that has been helpful to us was to hire “local talent”—i.e., TA providers who have existing relationships with local practitioners and train them in AGTO.

A final comment is that given the wide range of programs involved in the AGTO project, it has not been possible for the TA staff to have substantive expertise in all the program types and domains. Therefore, the Support System was enhanced with additional personnel: TA supervisors, GTO experts, and Developmental Assets experts who played an important role in supporting the needs of TA staff during the first 6 months. This has implications for establishing an ISF on a large scale, because do so may require a variety of personnel with different expertise.

Synthesis and Translation System

Tailoring the guidance provided by the Synthesis and Translation System more specifically to each program has been helpful to optimize the use of the assistance offered and is a good representation of the two-way interactive nature of the ISF. It is also consistent with many implementation theories. For example, Rogers (1995) and Green et al. (1980) and others (Rubenstein et al. 2000; Rosenheck 2001; Bartholomew et al. 2001) all state that translation efforts will be maximized when they are based on assessments of the needs, barriers, and incentives of targeted end users and involve local representatives in the planning process.

Community Research Workgroup (CRWG): Interaction Between the ISF Systems

The CRWG has been providing an opportunity for all three levels of the ISF to interact on a regular basis, improving ISF functioning and adding an important professional development opportunity for community practitioners. In addition, the CRWG has been fostering peer learning similar to a Community of Practice (Wenger et al. 2002) by providing practitioners the opportunity to share stories and get advice from each other. Already the group has been expressing interest in continuing meeting after the formal intervention ends and the Support System is currently planning to train the CRWG to operate on its own.

Conclusions

In this article, we have described how AGTO operationalizes the ISF, presented early lessons learned, and have shown that, in part, individual prevention capacity is related to the performance of high-quality prevention. Although the study is at an early stage, the progress to date is encouraging and we look forward to sharing more data in future years, in particular connecting the preliminary findings linking individual capacity to changes in program performance and eventually to youth outcomes. This final link will allow us to completely test the AGTO project logic model (Fig. 1), demonstrate an important causal link between practitioner capacity, program performance, and youth outcomes, and further validate that quality Prevention Support Systems result in more effective Prevention Delivery Systems (a key premise of the ISF). Looking across the lessons learned from the first 6 months, it appears that while the ISF may have been conceptualized as three distinct systems, in practice they are less distinct. In fact, during this early period that the AGTO based ISF has been operating, the overlap in these systems as described above has been beneficial for improved prevention delivery. However, a significant question going forward is the extent to which a tightly integrated ISF as demonstrated in this AGTO intervention could be replicated on a larger scale. Currently, a loosely organized ISF exists at the national level—scattered prevention practitioners, TA providers, and researchers carry out their work without significant connections. Weaving these systems together more tightly may take more resources than are available at the federal level. There may be opportunities, however, to develop the more-resourced and integrated ISF that AGTO envisions. For example, the recently released National Prevention Strategy specifically acknowledges the need for building capacity in the areas in which the ISF and AGTO operate: “Making places healthier requires capacity for planning, delivering, and evaluating prevention efforts (National Prevention Council 2011, p. 14).” Studies on AGTO and other examples of the ISF could play a large role in informing policy makers about the necessary level of federal, state, and local supports for prevention and health promotion among adolescents.