Technical Assistance as a Prevention Capacity-Building Tool: A Demonstration Using the Getting To Outcomes^® Framework

Abstract

Demands on community-based prevention programs for performance accountability and positive outcomes are ever increasing in the face of constrained resources. Relatively little is known about how technical assistance (TA) should be structured to benefit community-based organizations and to lead to better outcomes. In this study, data from multiple sources were used to describe an effective TA model designed to improve the capacity of community-based organizations to plan, implement, and evaluate prevention programming. This article is the first of its kind to provide detailed analyses of the TA delivered to community-based organizations to build substance abuse prevention capacity. The results of this study describe the range of TA services provided and the importance of two-way communication between the TA provider and recipient. TA recipients reported high satisfaction and an improved understanding of targeted TA activities. However, achieving these benefits requires significant program staff time, and not all skills were successfully transferred. Results from this study suggest how TA may be structured to be effective in supporting quality prevention programming in community settings.

Technical assistance (TA) has become a popular vehicle in the prevention field to improve community program capacity and enhance outcomes. TA has been used to describe different types of activities, including community-friendly manuals, on-site consultation, regional workshops, train-the-trainers models, and interactive Web-based systems (Stevenson, Florin, Mills, & Andrade, 2002). Although TA has intuitive appeal and broad support, the research that has examined its impact tends to be theoretically and methodologically inadequate (Chinman et al., 2005). Many questions about the provision of TA remain, given the variation in the kinds of TA that are available and the range of capacity-building areas to be addressed. To help develop theory and to construct an evidence base for TA, this article presents a detailed examination of one effective TA model that has been shown effective in building community program capacity for substance abuse prevention (Chinman et al., 2008).

In this study, TA was delivered to community-based prevention staff to assist them with using the Getting To Outcomes^® (GTO) Framework. GTO is a 10-step process designed to enhance prevention capacity by supporting the use of research-based practices to better plan, implement, and evaluate programs (Chinman, Imm, & Wandersman, 2004; see Figure 1). GTO is an example of empowerment evaluation, a method whereby community practitioners lead the work with an outside consultant or technical assistant often serving as a coach or facilitator (Fetterman, Kaftarian, & Wandersman, 1996). The term coach encompasses many ways of offering support and facilitating action (Motes & Hess, 2006). Coaching in the collaborative evaluation field refers to an individualized tutoring approach that emphasizes understanding (Rodriguez-Campos, 2005). As a result, communication among the technical assistant and recipient is critical for improvement. Facilitation is a consultation method that emphasizes change through encouragement and action promotion (Kitson, Harvey, & McCormack, 1998). In health care settings, facilitators have been described as assisting staff in changing their work practices successfully (Rycroft-Malone et al., 2002; Stetler et al., 2006). In our study, TA was based on these coaching and facilitation approaches. As described in more detail in later sections, TA providers regularly met with individual community practitioners and set agendas that focused on progress related to the GTO 10-step framework using a flexible and motivating communication style.

Figure 1.

The 10 steps of the Getting To Outcomes (GTO) Framework.

Source: Reprinted from Wandersman, Chinman, Imm, & Kaftarian (2000), with permission from Elsevier.

Research describing effective TA models for prevention is lacking. TA studies typically have focused on describing the needs of communities (e.g., data analyses and evaluation assistance) rather than describing effective TA models (Jolly, Gibbs, Napp, Westover, & Uhl, 2003; Kegeles, Rechook, & Tebbetts, 2005). However, Kelly et al. (2000) reported that ongoing consultation led to significant improvements in evidence-based program implementation as compared to TA delivered by manuals only and/or 1-day training workshops, which suggests the value of a consultation model of TA. Despite this, consultation models of TA to build community prevention capacity have not been adequately described. For example, findings from a review of the literature suggest that the effect size of a coach or facilitator is difficult to discern given the diverse ways in which it has been conceptualized and applied (Harvey et al., 2002; Kilburg, 1996). As a result, the concept and impact of such TA approaches are in need of more systematic study.

In this article, we address the existing research gap on TA by applying a mixed methods approach to describe an effective model of TA delivered to six programs located in two substance abuse prevention coalitions. Both quantitative and qualitative data were collected from multiple sources, and these data were analyzed using both quantitative (e.g., frequency counts) and qualitative methods (e.g., narrative analyses) to address a variety of research questions, including how much TA was delivered, what GTO steps were addressed, what the TA consisted of and what was done by whom (the TA provider and recipient), and how it was perceived by TA providers and recipients.

To provide a description of an effective TA model, this article contains the following information: We first describe the amount of TA delivered around the GTO steps by quantitatively analyzing notes created by the TA providers; next, we describe the TA that was delivered (e.g., who engaged in what sorts of activities) from a qualitative analysis of the narrative sections of the TA notes; third, we expand on what was learned from the TA notes by reporting the perspectives of TA recipients as captured through semistructured interviews; and fourth, a descriptive analysis of self-administered questionnaire data from the TA recipients is presented that builds on the recipient interviews by providing a more systematic and detailed account of the recipient experience. As suggested by a mixed methods approach, we phased the data collection and analyses process (Creswell, 2003); the results from the TA notes that were gathered during the course of the demonstration helped inform the data collection and analyses of the semistructured interviews with TA recipients that were conducted at the end of the demonstration period. In sum, the findings from this study may be helpful in developing a better theoretical understanding of TA and may lead to a more efficient TA approach in community settings.

METHOD

Participants

Data were collected from TA providers and prevention practitioners (i.e., program staff) from two community coalitions.

TA Providers.

Three TA providers participated during the 2-year demonstration period. The first TA provider assigned to Coalition 1 served for a year and was replaced by a second provider, who served until the end of the study period. The initial TA provider was a male, Caucasian, master’s-level researcher. The second TA provider was a female, Caucasian, community psychologist (PhD) who had previously served for 10 yeas as an evaluator for the coalition. In Coalition 2, the same TA provider served for the entire demonstration period. This TA provider was a female, Caucasian, social psychologist (PhD) who had no previous experience with Coalition 2 but 3 years of experience conducting community participatory research.

TA Recipients.

Fifteen staff persons from two community coalitions representing six programs participated in this study. One of the coalitions, located in a suburban/rural county in the southeastern portion of the United States (labeled Coalition 1), operated five prevention programs, two of which received TA. The communities targeted by this coalition were on average 65% Caucasian, 30% African American, 2% Latino/a, and 3% other race/ethnicity. The 2000 Census data indicated that about 6% of families and 9% of the population in this county lived below the poverty line (U.S. Census Bureau, Population Division, 2006). Since 1998, Coalition 1 received funding provided by a Center for Substance Abuse Prevention State Incentive Grant program; prior to the start of this project, the coalition had participated in some testing and development of the GTO framework. The two programs receiving TA were (a) a universal prevention program targeting middle school students and (b) a multimedia drug prevention program for parents of children in Grades 4 through 8. The participants who were interviewed for this study were the two program directors (PDs) and the PDs’ supervisors (n = 4). The demographic makeup of the four participants was 100% female, 75% Caucasian, and 25% African American.

The other coalition (labeled Coalition 2) was located on the West Coast and contained urban/suburban/rural areas. The communities targeted by this coalition were on average 54% Caucasian, 34% Latino/a, 5% Asian American, 2% African American, and 5% other race/ethnicity. The 2000 Census data indicated that about 9% of families and 14% of the population in this county were below the poverty line (U.S. Census Bureau, Population Division, 2006). Coalition 2 began in 1991 with support from a large national foundation. Since that time, Coalition 2 has expanded support with a variety of local, state, and federal funding to implement 16 adult and adolescent programs that span the prevention to treatment continuum. Coalition 2 had no previous experience with the GTO framework. For this study, two programs started receiving TA in the first year: (a) a student assistance program in middle and high schools that delivered universal and selective prevention programs and (b) a teen court program for first-time misdemeanor offenders that included alcohol and drug education classes. In the second year of the study, TA was expanded to two additional programs: (c) a mentoring program for high-risk elementary and middle school youth and (d) an adolescent treatment program. The directors for the four programs participated in TA, along with other line, supervisory, administrative, and executive-level staff, for a total of 11 participants. Of the 11 interviewed, 64% were female, 73% Caucasian, and 27% Latino/a.

TA Description

TA was delivered as part of a GTO demonstration study (Chinman et al., 2008). Before TA began, a work group comprising staff and community stakeholders provided input on demonstration planning, including program selection. Selection of programs that were to receive TA were based on which programs targeted the largest populations, required evaluation for funding requirements, and were most likely to be sustained during the demonstration period. Once programs were selected, a daylong training in GTO was delivered to participating program staff. The purpose of the training was to describe the GTO framework and to explain why it was important to the practitioners’ work. This training was provided annually throughout the demonstration.

Following the initial GTO training, TA was initiated. The TA provider was a half-time consultant position provided to each coalition. As part of this position, the TA providers offered to meet in person with staff of each of the GTO programs weekly or biweekly for about 2 hours and provided up to 2 hours of additional support per program a week. The TA sessions were designed to build on the training by providing additional information about the GTO framework and relating it specifically to the programs, following adult teaching methods that emphasize experiential learning (Conner, 2004). The TA providers were instructed to provide structure to the TA meetings but also to be flexible to program staff needs. For example, prior to each TA session, the TA provider e-mailed a meeting agenda organized by the 10 GTO steps to program staff. At the beginning of the sessions, the TA provider encouraged TA recipients to communicate needs in relation to the 10 GTO steps and modified the meeting agenda accordingly. Next, the TA providers and staff discussed ways to address relevant GTO steps based on current program status. At the end of each session, the TA provider supplied a summary of progress that included future plans for both parties. These summaries were documented in the TA notes and e-mailed to program staff.

Measures

TA Notes.

The TA provider logged TA activity by recipient (i.e., program staff), mode (e.g., in-person, phone, or e-mail), time spent by TA provider, GTO step that assistance addressed, and a qualitative description about what had occurred. This log was designed for use in this project.

Semistructured Interviews.

To assess the experience by program staff, interviews were conducted with 15 staff persons at the end of the demonstration. To minimize bias, an independent interviewer who had not been part of the demonstration conducted the interviews. The interviews used open-ended (“grand-tour”) questions followed by focused, standard probes, such as verification and compare-and-contrast questions (O’Brien, 1993). Questions included input about the TA (e.g., “What did you think of the TA?” followed up with “Was it helpful or not helpful?” and “How could the TA be improved?”). Inquiries about broader domains also yielded information about TA, such as facilitators and barriers to using GTO (e.g., “What made it easy or difficult to use GTO?”), followed by probes including reference to the TA.

Self-Administered Questionnaires.

At the conclusion of the interviews, program staff were asked to complete a brief questionnaire that asked about their impression of the TA experience. The response options ranged from 1 to 5 and are included in Table 1. The first 15 items were adapted from scales used in the substance abuse treatment field to assess client satisfaction with treatment services (Dennis, 1998; Hser, Evans, Huang, & Anglin, 2004; Simpson & Joe, 1993). The remaining items were developed for this study and asked about the respondents’ experience with giving feedback on the GTO framework (Section 2), the TA provider’s accessibility (Section 3), the perceived difficulty of performing GTO-related activities (Section 4), and the perceived difficulty in future evaluation tasks (Section 5).

Table 1.

Mean Ratings and Standard Deviations From the Self-Administered Questionnaire Items by Coalit

	Coalition 1		Coalition 2
	(n = 4)		(n = 8)
	M	SD	M	SD
Ratings: 1 = strongly agree, 5 = strongly disagree
1. TA Experience (alpha =.81)
TA was conveniently scheduled.	1.25	0.50	2.13	0.83
TA respected your/your program staff’s opinions.	1.00	0.00	1.75	0.71
TA was too demanding.	4.50	0.58	4.38	0.52
You needed more TA sessions to understand/implement GTO.	4.25	0.50	2.38	1.19
You trusted the TA.	1.25	0.50	1.38	0.52
The kind of TA provided was not very helpful to you.	4.50	0.58	4.38	0.52
You have improved your program because of TA.	1.75	0.50	1.75	0.71
Other staff receiving TA were helpful to you.	1.33	0.58	3.00	0.93
TA helps some staff/programs.	1.50	0.58	1.75	0.71
TA required that you learn about program accountability.	1.50	0.58	1.63	0.74
TA helped you develop an understanding about GTO.	1.25	0.50	1.88	0.99
TA really helped you.	1.50	0.58	1.38	0.52
You were satisfied with the TA.	1.25	0.50	1.50	0.53
Your supervisor wanted you to receive TA.	2.50	1.73	2.00	0.93
TA interfered with your other job responsibilities.	4.50	0.58	3.38	0.92
2. Feedback on GTO (alpha = .94)
I had the opportunity to provide feedback on GTO-related activities.	1.25	0.50	1.50	0.53
When I provided feedback on GTO, it was considered with due respect.	1.25	0.50	1.63	0.74
3. TA Accessibility (alpha = .96)
The TA was accessible to me and my coworkers for us to express our suggestions.	1.00	0.00	1.38	0.52
The TA was accessible to me and my coworkers for us to express our concerns.	1.00	0.00	1.38	0.52
I felt comfortable approaching the TA to ask questions.	1.00	0.00	1.50	0.76
Ratings: 1 = not at all difficult, 5 = very difficult
4. Perceived Difficulty: How difficult was it to … (alpha =.85)
… participate in the GTO TA meetings?	1.50	1.00	2.00	0.93
… prepare for the GTO TA meetings?	1.50	1.00	2.06	0.68
… attend the GTO training?	1.50	1.00	1.86	0.90
… complete the GTO worksheets in the manual?	2.00	1.15	2.63	1.41
… participate in the GTO work group meetings?	1.75	0.96	2.00	0.76
… prepare for the GTO work group meetings?	1.50	1.00	1.71	0.76
5. Perceived Difficulty: How difficult will it be in year ahead to … (alpha =.94)
… decide what questions to ask, given program goals and objectives?	1.75	0.96	2.00	1.07
… design a questionnaire for your program clients	2.50	1.29	2.50	1.41
… decide which clients to select to include in the survey?	1.75	0.96	1.50	0.93
… print and assemble questionnaires and other materials?	1.25	0.50	1.88	0.83
… train staff to administer the survey?	1.75	0.96	2.38	1.51
… enter the data?	1.50	0.58	2.13	1.36
… analyze the data?	1.75	0.50	3.43	1.51
… produce a report on the results?	1.50	1.00	2.86	1.77
… present the results to stakeholders?	1.33	0.58	2.71	1.89
… apply the results to improving program services?	1.75	0.96	2.63	1.77

Note: TA = technical assistance; GTO = Getting To Outcomes.

Analyses

A mixed methods analytic approach was adopted for use in this study. Data were collected from two sources, both the TA providers and recipients, to broaden the findings.

For example, the TA notes and recipient interviews were used to obtain information about different aspects of TA; the TA notes provided information about the amount and type of TA delivered, whereas the recipient interviews offered information about the TA’s helpfulness. To help validate the results, data collected from interviews and self-administered questionnaires from the TA recipients about similar aspects of the TA (satisfaction, accessibility of GTO and the TA) were analyzed. Later on, we discuss the analysis of each data source.

TA Notes.

To better understand the characteristics of the TA, we used a classic content analysis approach to analyze the TA notes (Neuendorf, 2002), using both quantitative and qualitative techniques. First, we examined the amount of TA (in hours) delivered to each program by GTO step (i.e., 1 to 10). Twenty-five percent of the TA note sections were reviewed by the principal investigator (PI) for reliability of the GTO step coding by the TA providers. We then calculated, for each GTO program, the total number of hours of TA delivered and the amount and percentage of TA hours spent on each GTO step (see Table 2).

Table 2.

TA Hours by Coalition, Program, and Year(s) Spent on the Getting To Outcomes (GTO) Steps

	Coalition 1								Coalition 2
	Program A				Program B				Program C				Program D				Program E		Program F		Means Across Programs and Years
Total TA Hours	Year 1 84.8		Year 2 197.75		Year 1 44.58		Year 2 81.25		Year 1 115.25		Year 2 140.25		Year 1 148.00		Year 2 178.25		Year 2 85.25		Year 2 79.50
Time Spent	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	Hours	% of Total	118.90
Logic worksheet	0.50	0.59	1.50	0.76	0.00	0.00	0.00	0.00	4.50	3.90	1.00	0.71	4.75	3.21	0.00	0.00	5.75	6.74	2.00	2.52	2.00
1 = Needs	5.00	5.90	0.00	0.00	2.00	4.49	0.00	0.00	1.75	1.52	0.25	0.18	4.25	2.87	0.00	0.00	0.50	0.59	3.75	4.72	1.72
2 = Goals	3.50	4.13	0.00	0.00	4.00	8.97	0.00	0.00	6.00	5.21	0.00	0.00	5.50	3.72	0.50	0.28	4.00	4.69	2.50	3.14	2.77
3 = Best practices	2.50	2.95	4.00	2.02	0.33	0.74	0.00	0.00	7.50	6.51	0.50	0.36	4.50	3.04	0.25	0.14	4.00	4.69	2.00	2.52	2.42
4 = Fit	3.00	3.54	0.00	0.00	1.00	2.24	0.00	0.00	4.00	3.47	0.00	0.00	0.50	0.34	0.25	0.14	0.00	0.00	0.50	0.63	0.87
5 = Capacities	3.75	4.42	6.00	3.03	2.00	4.49	8.00	9.85	17.25	14.97	26.25	18.72	7.75	5.24	11.00	6.17	5.25	6.16	17.25	21.70	10.29
6 = Plan	16.30	19.22	20.50	10.37	11.00	24.67	10.25	12.62	18.50	16.05	67.00	47.77	7.75	5.24	10.25	5.75	23.50	27.57	35.50	44.65	22.01
7 = Process evaluation	44.00	51.89	33.00	16.69	16.00	35.89	21.00	25.85	33.00	28.63	33.25	23.71	29.75	20.10	74.00	41.51	10.50	12.32	9.50	11.95	27.82
8 = Outcomes evaluation	4.25	5.01	84.25	42.60	8.00	17.95	34.00	41.85	17.25	14.97	3.25	2.32	75.00	50.68	44.50	24.96	29.75	34.90	4.00	5.03	28.43
9 = CQI	1.00	1.18	25.50	12.90	0.25	0.56	5.00	6.15	4.25	3.69	8.00	5.70	0.75	0.51	29.00	16.27	1.75	2.05	1.00	1.26	6.62
10 = Sustain	1.00	1.18	23.00	11.63	0.00	0.00	3.00	3.69	1.25	1.08	0.75	0.53	7.50	5.07	8.50	4.77	0.25	0.29	1.50	1.89	3.99

Note: TA = technical assistance; CQI = continuous quality improvement.

Second, given our goal to describe the TA delivered in these settings, we sorted the narrative text sections of the TA notes by themes that identified what activities occurred in each entry, who was involved, and who initiated the activity. Similar to an approach used by Lincoln and Guba (1985), the PI and one of the TA providers first read sections from all of the programs to identify themes. After more than 50 sections of text from each of the programs had been reviewed by the PI and TA provider, a coding system was developed that included a description of each theme, inclusion and exclusion criteria for sorting sections, and typical and atypical exemplars (Ryan & Bernard, 2003). A Microsoft Excel spreadsheet was used to sort the narrative sections of the TA notes by the identified themes as the original set of notes had been kept into an Excel spreadsheet, and the task involved copying and pasting sections of text into columns associated with the themes. Next, the PI and TA provider trained a bachelor’s-level research assistant (RA) to sort the rest of the narrative data. After approximately 10% of the data had been sorted by the RA, the PI and TA provider reviewed the sorting for accuracy. Less than 5% of the sections were incorrectly sorted by the RA, and inconsistencies were discussed with the RA, PI, and TA provider to determine a final consensus coding decision. After all sorting was completed by the RA, sections were reviewed by the PI and TA provider to check for coding accuracy and to determine whether any additional themes were present. As noted earlier, one of the TA providers was involved in the analyses of the notes. This process was found to be beneficial in that the provider could clarify any ambiguities in the content of the notes for the PI and RA. To prevent bias, the TA provider’s input into the coding was done in collaboration with at least one additional member of the research team, the PI and/or RA.

Semistructured Interviews.

Interviews were conducted in person and tape recorded. Next, following a long tradition in anthropology (Bernard, 2006; Sanjek, 1990), the interviewer developed field notes, which are detailed summaries of the interview, rather than transcriptions. This approach better met the purposes of our study—that is, to document themes based on the interview questions (e.g., how the TA helped/did not help)—and is less time consuming and expensive than transcription analyses. Next, a graduate student familiar with GTO but not otherwise connected to the project created a single narrative from all the interviews, following the interview protocol as a guide while allowing for new themes to emerge. Both the interviewer and graduate student were instructed to include verbatim quotes from interviewees that summarized consensus and minority responses from the key questions (e.g., “What about GTO was helpful?”). The PI, TA providers, and site liaisons then reviewed the summaries and prepared a final narrative that described all the interviews based on consensus discussion. As described in greater detail in the “Results” section, information about the TA process was extracted from these interviews and organized by the themes generated from the TA notes.

RESULTS

Time Spent on TA and the GTO Steps

We quantified the time spent in relation to the 10 GTO steps by program (see Table 2). Each program received approximately 1 to 3 hours of TA per week during the course of the study period. The TA hours varied across the programs for a variety of reasons. First, all programs were invited to participate in regularly scheduled meetings with the TA provider. However, some programs requested more meetings and assistance than others. Second, the initial TA provider for Coalition 1 did not establish a routine schedule with program staff, limiting hours delivered during the first year. Third, in Year 2, the TA provider for Coalition 2 was providing assistance across four programs rather than two while using the same amount of effort. The work group’s preference was for the TA provider to continue to work with the two initial programs over the two newer programs; therefore, an unequal amount of assistance was delivered.

As shown in Table 2, these data from the TA notes indicate that the early TA process consisted of attention to the initial GTO steps (e.g., developing logic models, needs, goals, and best practices). However, time spent on these steps was brief relative to the time spent on planning (GTO Step 6) and evaluation activities (GTO Steps 7 and 8) across all programs. These three steps contributed to the majority of TA time (i.e., greater than 78%). In general, the next most time-intensive step was Capacities (GTO Step 5). Work on continuous quality improvement (CQI; GTO Step 9) and sustainability (GTO Step 10) were more often noted as part of second-year activities. As reported elsewhere (Chinman et al., 2008), the TA time spent on a GTO step was positively related to improvements in prevention capacity, with the greatest amount of change found in program planning and evaluation capacity.

TA Activities

The narrative sections of the TA notes that described the TA by the provider were sorted into 4,717 coded sections of text. Sections ranged from a brief sentence to several sentences. The sections were categorized into six major themes around the “Who did what?” analyses: “Communication,” “Planning,” “Performance,” “Capacities,” “TA Provider Challenges,” and “TA Provider Analyses.” Illustrative examples of the sections of text that were coded into these themes appear in Table 3. Sections of text associated with three major themes—“Communication,” “Planning,” and “Action”—were sorted into subthemes (described later). In addition, the percentages of text sections that were categorized by each theme and subtheme are reported alongside the theme labels in Table 3

Table 3.

Example TA Note Sections Categorized by Themes and Percentage of Total Sections by Theme

Communication (31%)		Planning (24%)		Performance (21%)		Capacities (2%)	Challenges (2%)	Analyses (20%)
TA Provider— Initiated for Coalition’s Education (7%)	“I explained the worksheet: where I got the information from and how I had included program goals that included school data.” “I shared Life Skills fidelity materials with PD.” “I showed staff the cost estimate worksheet from the GTO manual.”	Group (8%)	“We discussed what doing with our TA time and I asked staff what they would like help with in the next meeting.” “We prepared to present issues at work group meeting.” “We discussed a plan for training.”	Group (7%)	“We next turned to the goals tool We went through the different goals that had been identified and made revisions.” “Two clients’ data been put on hold as they are experiencing staff turnover.” “We clarified the data collection procedures on paper.”	“Staff don’t have enough time to get a group up and running before the holidays.” “Training has been put on hold as they are experiencing staff turnover.” “Staff were unable to use the CD we provided because their computers do not have CD drives.”	“Do you think that it is feasible for staff to pursue getting access to school performance for current students?” “Site liaison was asked to provide info about survey follow-up methods.” “If anyone can provide suggestions for sustainment activities, please let me Know.”	“There is some confusion over who should attend the training.” “PD cancelled meeting because they are busy preparing a grant application.” “PD is optimistic that they will get good response rates by using this Web-based survey.”
Coalition— Initiated for TA Provider’s Education (12%)	“PD provided me with grant application that outlined services delivered, population served, and outcomes.” “PD informed me that they are instituting a new activity for all clients on …” “PD showed me the survey that they had implemented last year.”	TA (11%)	“I proposed to write up our work in the logic worksheet and circulate it before our next meeting.” “In prep for our meeting tomorrow, I planned to review CSAP core measures.” “I reviewed Web site and planned to contact developer of the assessment tool.”	TA (11 %)	“I attended the staff meeting and trained staff on new database.” “I ordered evaluation materials from program developer.” “I developed consent materials to be used for the evaluation.”
TA Provider— Request for Info from Coalition (6%)	“I asked PD to report on how long it took to complete the monthly meeting form.” “I asked staff for the numbers from last year so that we could compare.” “I asked staff for a copy of the curriculum that they were using.”	Coalition (5%)	“PD intends to present his idea of having a monthly manager’s meeting with CEO on Monday.” “For our next meeting, staff will prioritize the list of measures.” “PD is going to look into the costs associated with getting the raw data back.”	Coalition (3%)	“PD went through the CSAP core measures and marked the ones that were relevant.” “Staff had a meeting with the truancy officers last week.” “PD sent me raw data file for analyses.”
Coalition— Request for Info from TA Provider (6%)	“Staff would like a shorter instrument, preference is for a self-administered version.” “Staff would like some guidance on implementing new survey materials and how to analyze data.” “I received email: When you come again, please bring the manual.”

Note: TA = technical assistance; PD = program director; GTO = Getting To Outcomes; CSAP = Center for Substance Abuse Prevention.

Communication.

As shown in Table 3, “Communication” represented the highest proportion of coded text (31% of the total sections). The subthemes show that the majority of TA provision was the sharing of information between the TA provider and program staff, either for the purposes of educating (61% of the “Communication” entries) or for requesting information (39% of the “Communication” entries). The notes indicated that the TA providers typically provided information about evaluation, including literature or information about outcome measures and benchmarks, data collection procedures and tools (e.g., databases or surveys), analytic interpretation, and proposed program or evaluation changes as a result of evaluation findings. In contrast, most communication from the program staff to the TA provider involved descriptions of program activities, past assessment tools, target population, funding/ stakeholder requirements, and changes to programming, staffing, training, and/or target population, as well as use of TA-generated tools and program data. Some programs discussed competing programs in community and sustainability activities.

Planning.

The second largest proportion of text was categorized as “Planning” (24% to the total sections). Subthemes indicate whether the planning was done by the TA provider, the TA recipient, or collaboratively by both (i.e., group). A review of the notes shows that the type of planning varied based on the maturity of the program. For example, established programs concentrated more on planning of existing programming, evaluation, CQI, and sustainability, whereas newer programs focused on planning program goals, activities, and staffing. To meet evaluation needs, the TA provider conducted planning independent of program staff (46% of the “Planning” entries), such as literature reviews, database, codebook, and consent form development, staff training in evaluation, statistical analyses, and report writing. Other types of planning by the TA providers included developing meeting agendas, reminders for meetings, and drafts and modifications of the GTO worksheets. With little TA provider input, the notes indicated that all practitioners planned meetings with community partners (i.e., partner agencies, referral sources, funders), staff training, and program recruitment activities (21% of the “Planning” entries). A few practitioners planned evaluation activities such as data collection, data entry, and analyses on their own.

Performance.

The third largest category of text was categorized as “Performance” (21% of the total sections). Like “Planning,” sections were sorted into subthemes, based on whether it was an action performed by the TA provider, recipient, or collaboratively the group. This theme refers to the undertaking of a GTO activity; for example, the notes indicated that the TA provider, independent from staff (11%), completed GTO worksheets, reviewed relevant literature and funding requirements, contacted program developers, disseminated best practice summaries, and conducted staff trainings in data entry, analyses, and report writing (52% of the “Performance” entries). Actions performed by practitioners independent from the TA provider (14% of the “Performance” entries) included disseminating program materials, reviewing reporting requirements, and making presentations about the program to key stakeholders. In each program, practitioners were often in charge of collecting data and data entry.

Capacities.

Notes that were specific to actions or plans that were not able to be carried out because of a lack of staff or technical resources were sorted under the “Capacities” theme (2% of the total sections). Most of the programs reported problems with conducting activities because of staff turnover. About half of the programs mentioned cutting back on activities because of reductions or changes in funding and because of concerns about space to conduct programming. A few programs mentioned difficulties because of lack of staff expertise.

TA Provider Challenges and Analyses.

These categories were created to encompass statements made by the TA provider that were meant to inform the readers of the TA notes about questions or opinions about the demonstration. The site liaison and PI reviewed notes and discussed how to address challenges with the TA provider regularly. The challenges (2% of the total sections) included such issues as strategies to deliver GTO, requests for guidance on completing worksheets/tools, improving staff participation, and agency communication. The “TA Provider Analyses” category (20% of the total sections) consisted of the TA provider’s perception of program progress with GTO and its perceived facilitators (e.g., staff interest) and barriers (e.g., staff turnover). For these sections, the TA provider characterized what was helping or hurting use of the GTO framework, such as practitioners’ positive and negative reactions to it.

TA Recipient Interviews

Interviews with 15 program staff about their experience with the demonstration help characterize the TA in the words of the TA recipients. Using a sequential approach to interpreting our data (Creswell, 2003), we used the major themes generated from the narrative analyses of the TA notes—“Communication,” “Planning,” and “Capacities”—to examine whether the recipients described the TA around these themes and whether program staff mentioned the types of TA provided to them using the GTO steps. Quotes from these interviews were selected that help illuminate these themes and the TA experience from the recipients’ viewpoint.

Communication.

Many recipients mentioned effective communication in response to a question about the helpfulness of the TA provider:

[The TA provider] was helpful in taking things that are complex and making them clear. (PD for 3 years, Coalition 1)

The TA provider’s communication style, being flexible, motivating, and accessible similar to a facilitator or coach, was also expressed:

[TA provider]’s knowledge and patience was very helpful and [TA provider] was able to work with people at the appropriate level—some people had more computer or analytic skills than others and [TA provider] worked well with folks at varying levels. (Executive Staff for 4 years, Coalition 2)

We have to be able to adjust to the community and it’s great to have [TA provider] that understands that but can also pull us back in if we get way off base. (PD for 1.5 years, Coalition 2)

[TA provider] was able to give us that direction and motion on how to approach some things and was always available to answer questions. (PD for 1.5 years, Coalition 2)

Planning.

Planning and structure was frequently mentioned as a particular helpful aspect of the TA provider:

By setting meetings and having [TA provider] come every 2 weeks, it forced us to focus on the data and preparing materials. (PD for 1 year, Coalition 2)

When asked what facilitated GTO use, planning by the TA provider was often cited:

The TA, both the knowledge and skill of the person providing it, but also the structure of regular meetings and guidance. (PD for 1.5 years, Coalition 2)

Capacity.

When asked about problems with the demonstration or ways to improve the TA, issues about capacity arose:

The hardest piece in continuing GTO will be manpower related. What will happen as there is staff turnover? Who will train the incoming folks who have not had exposure to GTO? (Executive Staff for 10+ years, Coalition 2)

The TA could have been improved only if I had had more time to prepare for those meetings and to more fully take advantage of my time with [TA provider]. (PD for 1.5 years, Coalition 2)

Identification of the GTO Steps.

Consistent with the results regarding the amount of TA delivered by GTO step, evaluation figured prominently in program staff responses regarding activities with which the TA was most helpful:

[TA provider] helped us set up databases and taught us database skills. [TA provider] also helped with collecting data and pushing us.” (PD for 1.5 years, Coalition 2)

[TA provider] spurred on the changes in the database that had to be made. (Clinical Staff for 5 years, Coalition 2)

TA Recipient Questionnaires.

Responses from 12 staff members were analyzed. The three executive-level staff persons from Coalition 2 were not asked to complete this questionnaire because of their lack of one-on-one meetings with the TA provider. Means and standard deviations on these items are presented in Table 1, organized by sections of the questionnaire. Given our small sample size and that the items were developed for use in this study, it was inappropriate to develop scale scores (Guadagnoli & Velicer, 1988). However, reliability coefficients (i.e., Cronbach’s alpha) are reported next to each of the five section headings, which suggest that, as might be expected, the “TA Experience” section is most likely composed of more than one construct, whereas some other sections (“Accessibility,” “Feedback,” and “Perceived Difficulty”) might represent one underlying construct.

Similar to the open-ended interview results, respondents from both coalitions reported their TA experience as very positive. Respondents from both coalitions perceived the TA provider as respectful, trustworthy, helpful, accessible, and open to feedback. Not surprisingly, practitioners from the coalition with less GTO experience reported greater difficulty with GTO activities and perceived increased difficulty in the year ahead. Practitioners from the coalition with more GTO exposure were less likely to agree that the TA interfered with other job responsibilities; this may indicate better integration of the GTO framework into their job duties over time than the practitioners from the coalition with less exposure had achieved.

DISCUSSION

This article is the first of its kind to provide detailed analyses of TA delivered in community-based organizations to build substance abuse prevention capacity. A mixed methods approach in both the data collection and analyses process was used. Data were integrated at several stages during the course of the research. During data collection, both open-ended (qualitative) and close-ended (quantitative) questions were used to gather TA recipients’ assessment of the TA. The TA notes were analyzed both qualitatively, by the generation of themes that denoted the main activities of the TA, and quantitatively, by analyzing the frequency of each theme. Moreover, consistent with mixed method approaches, a sequential process was used such that the primary data set, TA providers’ experience, informed the collection and analyses of a second data set, from the TA recipients (Creswell, 2003).

The results from this study show that the majority of the TA providers’ time was spent on communication with the TA recipients. As expressed by others studying TA to build prevention program capacity (Florin et al., 2006), we learned that brief and direct communication was highly valued by practitioners. The data from the TA notes and interviews suggest that an important component of the TA providers’ style was to serve as a motivator and guide to practitioners by setting meetings, giving direction, troubleshooting, and providing structure. The TA notes, interviews, and questionnaire data revealed that programs received the most assistance with evaluation activities. This result may have occurred because of the match between the TA providers’ skill set and program needs, the selection of programs used in this study, or that evaluation was the area within the GTO framework that was perceived as needing the most improvement by the TA provider and practitioners. The focus on evaluation and planning are consistent with findings from similar studies (Davis et al., 2000; Jolly et al., 2003; Kegeles et al., 2005; Stevenson et al., 2002).

High staff turnover and funding constraints have been reported as barriers to conducting program evaluation, and this was displayed in the results from this study. Because continued prevention funding often depends on documentation of quality implementation and effectiveness (O’Donnell et al., 2000; Stevenson et al., 2002), it is not surprising that the programs in this study requested assistance with evaluation. These findings indicate that structures are needed to support the building of evaluation capacities of community-based organizations. Possible mechanisms include the crafting of service provision grants that include support for TA and increased TA networks that use a consultation approach as outlined herein.

As noted in the interviews and questionnaire data, practitioners reported that it would be difficult to sustain GTO activities after the TA component ended. Other researchers have noted a reduction in prevention quality after TA ended (O’Donnell et al., 2000). Although the programs in our demonstration received a great deal of assistance during a fairly lengthy time frame (1 to 2 years), at this point, the long-term impact of this type of TA is not known; however, the high turnover in program staff, reliance on TA skills, and reported perceived difficulty provide evidence that the prevention capacity gains might not be easily sustained.

Implications for Practitioners and TA Providers

The findings from an analysis of the TA notes indicate that effective TA provision is a two-way street. The TA provider was not solely charged with disseminating information but also listened and learned from practitioners. Practitioners had the opportunity to communicate the details and status of their programs to the TA provider. The TA providers developed a relationship with program staff, and as a result, the TA providers were perceived as flexible, respectful, patient, and motivating by the participating program staff. Analogous to a clinical relationship, it is our belief that this relationship was the foundation for many of the gains made by the programs. Many other forms of TA do not use such an interactive approach and therefore may not realize the benefits found in this study.

To maximize the short-term benefits of TA, the findings from this study suggest that frequent, structured interactions are useful. For example, we learned that assessing needs early on helped to identify areas for the TA provider to target and plan more efficient onsite meetings. We also learned that documentation that specifies progress, to date, is helpful for both the TA provider and participating organization in keeping track of activities and communicating program improvements as well as strengthening prevention capacity for the longer term. Some of the areas that community-based prevention organizations may seek assistance from a TA provider are outlined in the “Results” section.

TA has costs as well as benefits for practitioners to consider (Florin et al., 2006). Participating in TA directs staff time and energy away from other activities. The interviews with practitioners suggest that staff were not always prepared to take advantage of the assistance. Any organization that intends to employ this type of TA should acknowledge the time and effort of participating staff. That is, in addition to the time set aside to meet onsite with the TA provider, the staff should have time to prepare for meetings and perform activities recommended by the TA provider.

Another important practitioner concern was how to handle the termination of the TA. Some of the GTO skills were successfully transferred to existing staff, but some of the evaluation work done by the TA provider was not. Securing funding to support a coalition-wide staff person to focus on GTO activities across multiple programs has been challenging. For one of the programs, federal grants that require an external evaluator have supported the TA provider to assist with performance reporting. In the other coalition, in-house funds are being used to support the TA provider to continue as a part-time grant writer and in-house evaluator. Our results indicate that instilling GTO activities into the organizational culture as the way the coalition does business was more apparent in the coalition with previous GTO experience, which suggests that it may require a long-term commitment and transition period.

In sum, this study found that an effective TA model consists of two-way interactions between TA providers and recipients that emphasize collaboration. Even with the support and satisfaction of this TA model throughout the demonstration period, communities may still need to consider hiring a grant writer or evaluator to perform these duties, which are traditionally outside the scope of program services staff workloads. The summary of the narrative sections of the TA notes give some indication of the kinds of activities TA providers may perform for prevention programs. More research is needed to determine the amount of TA needed to enhance prevention capacity.

Study Limitations and Ideas for Future Research

Certain limitations of this research should be noted. First, the GTO demonstration consisted of an annual training and delivery of a GTO manual along with the ongoing TA sessions. Although a strong correlation was found between the amount of TA received and the amount of improvement in prevention performance (Chinman et al., 2008), it is impossible to completely isolate the effect of TA from the manual and training. Future studies should consider examining varying levels of TA, training, and manual exposure to assess the unique impact of each demonstration aspect. Second, the data presented herein are limited by small sample sizes. Future research that employs larger sample sizes will provide the opportunity to more adequately test the effectiveness of TA and the reliability of the measures presented here. Third, although the findings were quite similar across two coalitions and six different programs, it is not clear that a GTO demonstration as implemented by other TA providers in different programs would result in similar findings. For example, the initial TA provider for Coalition 1 was not well used. We did not have the opportunity to analyze the factors that distinguished use across the TA providers because we did not anticipate the turnover. Finally, the ultimate goal of improving prevention capacity is for the benefit of improving program outcomes. However, we do not present any evidence of this link. Most of the programs we worked with had limited evaluation data at demonstration inception, and therefore, an assessment of changes during the course of the study was not possible. These limitations underscore the need for future research to evaluate TA delivered on a larger scale with comprehensive and rigorous design and measurement tools.

Summary

This article provides a detailed description of an effective form of TA to build prevention capacity of community-based organizations. The TA delivered in this study focused mainly on communication between TA providers and practitioners around planning and evaluation. This detailed examination of the TA process has begun to identify useful TA practices and contributed to the development of a theoretical understanding and research agenda for TA. We think that our results would translate to other health promotion programming, given that similar needs have been expressed in the literature about different prevention settings.