The Project Towards No Drug Abuse (TND) Dissemination Trial: Implementation Fidelity and Immediate Outcomes

Abstract

One of the important research issues in the emerging area of research on dissemination of prevention programs relates to the type and extent of training needed by program providers to prepare them to implement effective programs with fidelity. The present paper describes the immediate outcomes of a dissemination and implementation trial of Project Toward No Drug Abuse, an evidence-based prevention program for high school students. A total of 65 high schools in 14 school districts across the USA were recruited and randomly assigned to one of three experimental conditions: comprehensive implementation support for teachers, regular workshop training only, or standard care control. The comprehensive intervention was comprised of on-site coaching, web-based support, and technical assistance, in addition to the regular workshop. Students (n=2,983) completed self-report surveys before and immediately after program implementation. Fidelity of implementation was assessed with a classroom observation procedure that focused on program process. Results indicated that relative to the controls, both intervention conditions produced effects on hypothesized program mediators, including greater gains in program-related knowledge; greater reductions in cigarette, marijuana and hard drug use intentions; and more positive changes in drug-related beliefs. There were stronger effects on implementation fidelity in the comprehensive, relative to the regular, training condition. However, seven of the ten immediate student outcome measures showed no significant differences between the two training conditions. The implications of these findings for dissemination research and practice are discussed.

Introduction

One of our nation's most critical health challenges is to develop effective interventions that address priority health issues, and to assure that those interventions are being disseminated, implemented, and sustained once proven effective. The need for translation of effective interventions from research to real-world settings is especially noteworthy in the area of substance abuse prevention (Rohrbach et al. 2006). There is now substantial empirical evidence indicating that a number of school-based programs are effective in preventing or reducing adolescent substance abuse (Gottfredson and Wilson 2003; Tobler and Stratton 1997; Tobler et al. 2000); yet based on recent estimates, the prevalence of use of effective programs is relatively low (Dusenbury et al. 2003; Ringwalt et al. 2009).

At present, there are substantial gaps in our scientific knowledge about the replication, dissemination, and successful implementation of evidence-based prevention programs (Bowen et al. 2009; Elliott and Mihalic 2004; Glasgow et al. 2003, National Research Council and Institute of Medicine 2009; Woolf 2008). In general, we know more about the impact of prevention programs when they are implemented in a uniform fashion (i.e., efficacy trials) than when they are delivered under conditions in which researchers have less control over implementation (i.e., effectiveness trials; cf., Flay 1986). Overall, the science related to implementing these programs with fidelity and positive outcomes for target populations lags far behind the science related to developing the programs (Fixsen et al. 2005).

One framework that is useful to guide research on the process of translating or “scaling up” the use of effective prevention interventions is Glasgow and colleagues' RE-AIM model (Glasgow et al. 1999). The RE-AIM framework identifies five dimensions that are pertinent to evaluating the translatability of interventions, including: (1) reach into the target population, (2) effectiveness, or the intended results of the intervention, (3) adoption, or the uptake of the intervention among targeted settings and providers, (4) implementation, as measured by the quantity, quality, and consistency of program delivery, and (5) maintenance, which is the long-term results of the program and/or institutionalization of it in targeted settings. This framework emphasizes the importance of conducting research on both the individual (i.e., reach and effectiveness) and organizational (i.e., adoption, implementation, and maintenance) levels of impact in order to evaluate the overall public health impact of an intervention.

In the present paper, we focus on two dimensions of the RE-AIM model to evaluate the translation of Project Towards No Drug Abuse (TND), an evidence-based substance abuse prevention program (Sussman et al. 2002). First, we examine the short-term effectiveness of TND when it is implemented on a large scale in real-world settings. During the past decade, researchers have begun to focus more attention on testing the effectiveness of prevention programs with proven efficacy. Specific research questions have included, for example, whether the effects of the program generalize to minority, low income populations (e.g., Botvin et al. 2001) or students in different regions of the country (e.g., Ellickson et al. 2003); whether the evidence-based program is effective when implemented in real-world settings by providers different from those in the original efficacy trials (e.g., St. Pierre et al. 2005; Spoth et al. 2007); and whether adaptations of prevention programs for specific cultural groups produce program effects (e.g., Komro et al. 2006). Across these studies the evidence for effectiveness has been mixed, with some studies showing effects comparable to those in the efficacy trials of the same program (e.g., Botvin et al. 2001; Ellickson et al. 2003), and other studies showing null or weaker effects (e.g., Harrington et al. 2001; Komro et al. 2006; St. Pierre et al. 2005). The present study represents the first trial to examine the effectiveness of Project TND when it is implemented by regular classroom teachers in a variety of real-world high school settings.

The second element of the RE-AIM framework that we address in this study is the fidelity of implementation of evidence-based programs in real-world settings. Several reviews of the literature have demonstrated the importance of implementing prevention programs faithfully, or with fidelity to the original design (Derzon et al. 2005; Du Boise et al. 2002; Durlak and DuPre 2008; Dusenbury et al. 2003; Fixsen et al. 2005; Tobler 1986; Wilson et al. 2003). In substance abuse prevention trials, for example, low fidelity has resulted in smaller or no program effects on behavioral outcomes (Botvin et al. 1990, 1992; Derzon et al. 2005). However, while it is widely recognized that implementation fidelity matters, little is known about the most effective approaches for improving fidelity among prevention program providers. Thus, one of the key research questions in the emerging area of implementation science is: What is the extent and type of training needed in order to prepare evidence-based program providers to implement programs with fidelity? In the present paper we examine the relative effectiveness of two training approaches for Project TND, a standard teacher training workshop versus a comprehensive training and implementation support model, on implementation fidelity and immediate student outcomes.

Research in school settings has shown consistently that teachers who participate in training workshops deliver a greater proportion of evidence-based health education program components, and adhere more closely to program manuals, relative to untrained controls (Basen-Enquist et al. 1994; Connell et al. 1985; Dusenbury et al. 2003; Parcel et al. 1991; McCormick et al. 1995; Perry et al. 1990; Ross et al. 1991). Training workshops provide the skills needed to implement the program, generate enthusiasm and commitment for both initial implementation and sustainability (Gottfredson and Gottfredson 2002; Joyce and Showers 1980; Perry et al. 1990; Rohrbach et al. 1993), and have a positive effect on targeted mediators or short-term program outcomes (McCormick et al. 1995; Parcel et al. 1991; Ross et al. 1991; Taggart et al. 1990).

However, in light of the many barriers to successful implementation of evidence-based prevention programs in school settings (e.g., Fagan and Mihalic 2003; Mihalic et al. 2008), it has been suggested that program providers may need training that extends beyond workshops and takes place throughout the implementation phase, in order to achieve program fidelity and positive program outcomes (Backer 2000; Fixsen et al. 2005; Joyce and Showers 2002; Mihalic and Irwin 2003). For example, a recent evaluation of 42 communities that implemented evidence-based programs identified by the Blueprints for Violence Prevention initiative showed that high-quality technical assistance was positively related to program fidelity and sustainability (Mihalic and Irwin 2003). To date, only a few school-based trials have tested the effectiveness of a multi-component training approach that combines pre-implementation workshops with ongoing technical assistance, coaching, and/or internet-based support. Allison et al. (1990) found that a teacher training approach that combined a training workshop and coaching for implementation of a substance abuse prevention program was more effective than a training workshop alone in producing effects on students' alcohol use intentions. Abbott et al. (1998) found that a series of teacher training workshops, which included program background information, skills building exercises, and coaching, improved program implementation and outcomes. In contrast to these findings, a recent randomized trial conducted by Ringwalt and colleagues (2007), in which a standard workshop training condition was compared to one where teachers received workshop training plus coaching to prepare them to implement an evidence-based substance abuse prevention program, showed no effect of the coaching component on specified program mediators. Thus, although the literature provides some support for the effectiveness of a multi-component approach for training evidence-based prevention program providers, more research on training models is needed to guide efforts to disseminate and successfully implement prevention programs on a wide scale.

Previously, Project TND was evaluated in five randomized trials that demonstrated an impact of the program on the use of alcohol, tobacco, marijuana, and hard drugs at 1-year follow-up (Dent et al. 2001; Sun et la. 2008; Sussman et al. 1998, 2002, 2003b; Valente et al. 2007), and an impact on hard drug use at 5-year follow-up (Sun et al. 2006). In the fourth trial, it was shown that implementation fidelity and immediate program outcomes were comparable when high school teachers delivered the program, relative to project health education specialists (Rohrbach et al. 2007; Sun et al. 2008). In the present study, we hypothesized that a comprehensive training condition would produce stronger implementation fidelity and immediate outcomes relative to a standard training workshop, and both training conditions would produce positive immediate outcomes relative to the control condition.

Method

School Selection and Experimental Design

A total of 65 high schools from 14 school districts in the USA were recruited as a convenience sample for this study. In order to be eligible to participate, districts needed to have at least three high schools that were willing to be randomly assigned to experimental conditions. We approached school districts that had contacted us for information about purchasing Project TND and met the above criteria, to determine potential interest in the study. Our objective was to obtain a sample that was representative, in terms of region of the USA, of high schools that had already adopted Project TND. We recruited schools from four regions of the country (northeast, south, central, and west) in equal proportions to those represented in our database of organizations that had adopted Project TND since we began program dissemination.

Of the 65 participating schools, 58 were regular high schools and 7 were alternative high schools (known as “continuation” high schools in California). Within each school district, participating schools were randomly assigned to one of three experimental conditions: comprehensive implementation support for Project TND teachers (IMP-SUPPORT), regular workshop training only for TND teachers (REGULAR), or standard care control (CTRL). This resulted in a sample with 22 schools in the IMP-SUPPORT, 21 schools in the REGULAR, and 22 schools in the CTRL condition. Schools were blocked prior to assignment by school size, ethnicity composition, and the percentage of students receiving free or reduced price lunch. Specifically, each group of three schools was aligned using a linear composite of factor scores across a drug use inflate-suppress continuum (Graham et al. 1984), and schools within each triad were randomly assigned to the three conditions.

Within high schools in the IMP-SUPPORTand REGULAR conditions, at least one teacher was recruited to participate in the training intervention and deliver Project TND to his/her students. The school designated the subject area for program implementation (health or physical education). Delivery of the program took place in existing class groupings of students at all schools in the two program conditions. In the standard care control condition, students received only the drug abuse prevention activities, if any, provided directly by their school.

Students in all three conditions were administered a questionnaire at pretest and immediately after the program. The average time span between pretest and immediate posttest was 41.1 days (SD=2.8), depending on adjustments made to the programming or data collection schedule in order to meet the needs of the schools.

Intervention

Project TND Curriculum

The Project TND curriculum targets substance use and violence-related behaviors through the use of a motivation, skills, and decision-making approach (Sussman et al. 2004b). The curriculum is comprised of 12 classroom sessions, approximately 45 min each, which are designed to be implemented over a 4-week period. Utilizing interactive teaching techniques, the instruction to students provides cognitive motivation enhancement activities, information about the consequences of drug use, correction of cognitive misperceptions, communication and coping skills enhancement, and tobacco cessation techniques. (For more details about the curriculum sessions, see Skara et al. 2005; Sussman et al. 2004a).

Teacher Training Interventions

Teachers in the IMP-SUPPORT and REGULAR conditions participated in a one-day workshop conducted at the program sites by certified Project TND trainers. The workshop was designed to introduce the key concepts and skills required by the program, build teacher self-efficacy and comfort with the program approach, and generate enthusiasm and commitment to the program. Trainers presented an overview of the theoretical underpinnings and evidence base for the curriculum, provided detailed instruction about each program lesson, and provided opportunities to practice key program activities.

The additional components of the IMP-SUPPORT intervention were designed to promote full mastery of the skills required by the program through web-based support, on-site coaching, and technical assistance. Following the workshop, the IMP-SUPPORT teachers attended a 2-h technical assistance session that provided an overview of the web site and coaching components. In addition, these teachers received two coaching sessions that occurred immediately after delivery of curriculum lessons five and nine, respectively. Each coaching session lasted approximately one classroom period, and it focused on feedback from the certified TND trainer about delivery of the observed lesson, problem solving about issues pertinent to the teacher's program delivery context, and implementation tips designed to help the teacher prepare for the remaining lessons. The web-based component of the training provided teachers with access to a password-protected web site where they could participate in a discussion forum about implementation issues and download teaching tips, scientific articles, and additional information related to the program. Throughout the program delivery phase, additional technical assistance from the trainers and program developers was made available to the teachers via telephone and e-mail, on an as-needed basis.

Subjects

Students

Within the program schools, two classrooms of students taught by each participating teacher were randomly selected to participate in the evaluation. Within the control schools, we randomly selected two classrooms taught by a teacher in the same subject area as that designated by the district for program implementation in the intervention schools. A total of 4,351 students were enrolled in the classes selected for participation in the evaluation. Positive parental informed consent and student assent were required for participation. Of the 4,351 enrolled students, 3,751 were consented for participation in the study (86.2% of students in the selected classes). Of the 3,751 consented students, 3,346 took the pretest survey (89.2%). Among the 3,346 subjects that completed the pretest survey, a total of 2,983 (1,275 in the IMP-SUPPORT, 924 in the REGULAR, and 784 in the CTRL condition) also completed immediate post-program surveys (89.2% retention rate). The immediate program effects analysis described in the present paper was performed with pretest and posttest data collected from these 2,983 subjects.

Student subjects varied from 13 to 18 years of age (mean age=14.8 years, SD=1.0 years) at pretest. The sample was 46.7% male; 40.0% white, 28.2% Hispanic, 15.6% African American, 3.2% Asian, 7.2% mixed ethnicity, and 5.8% other. The majority of students (94.8%) were enrolled in regular high schools and 5.2% were enrolled in alternative/continuation high schools.

Teachers

A total of 60 teachers participated in the training intervention (33 in the IMP-SUPPORT, and 27 in the REGULAR condition); however, data analyses are based on the 54 teachers that had complete data. The sample of teachers was 53.7% female, 75.9% white, 5.6% Hispanic, 13% African American, and 1.9% Asian. The teachers had a mean of 15.7 years (SD=11.3) of teaching experience.

Data Collection Procedures and Measures

Student Assessment

Pretest and immediate posttest measures were collected from students using a self-report, closed-ended response survey. At each wave of data collection, surveys were administered over one class period by the research project staff. Those absent from the classroom on testing days were left absentee packets, and local school staff were asked to distribute the surveys to the appropriate students and mail the completed surveys to the research staff.

Student survey measures were adapted from previous studies of Project TND (Sussman et al. 2002). Constructs assessed in both pretest and posttest surveys include demographic variables, immediate outcomes of the program, and substance use behaviors. Process evaluation items were also included in the posttest survey for students in the program conditions.

Demographic items included age (in years), gender, ethnicity (coded as non-Hispanic white, Hispanic, African American, Asian, mixed ethnicity, and other), current living situation (with parents, alone, or other), and parents' education (mean response across father's/stepfather's and mother's/stepmother's educational levels, based on six categories ranging from ‘did not complete 8th grade’ to ‘completed graduate school’).

We assessed ten immediate outcomes of Project TND that are potential program mediators and have been associated with substance use in previous studies (Rohrbach et al. 2005; Sussman and Dent 1996; Sussman et al. 2003a). Four of the items assessed substance use intentions, including how likely the student was to use cigarettes, alcohol, marijuana, and hard drugs (cocaine, amphetamines, heroin, etc.), respectively, in the next 12 months (1 = definitely not to 5 = very likely). Addiction concern is an index of three items (Sussman and Dent 1996) that assess students' perceptions of the likelihood they will become a drug abuser, addict, and alcoholic, respectively (1 = not at all likely to 4 = very likely; α=0.74). Four items that assessed beliefs about the immorality of drug use (e.g., “How wrong is it to use drugs?”) were averaged to create an index (1 = not wrong at all to 4 = very wrong; α=0.92). The pro-drug-use myths index was comprised of seven items that assessed cognitive misperceptions about physical, emotional, and social aspects of drug use (α=0.70; see Ames et al. 1999; Sussman et al. 1996). For data analyses, a score was calculated representing the percentage of pro-drug myths with which the subject agreed. Health-as-a-value is an index that averages three items about the importance of health as a life value (1=not at all important to 4 = very important; α=0.73; see Lau et al. 1986). Negative coping strategy is an index of three items adapted from Wills (1986) that assess how often subjects used each of three coping strategies when they had a problem at school or home during the previous year (e.g., “I party with other youth;” 1 = never to 5 = almost always; α=0.71). Program-related knowledge was measured with 22 items designed to assess learning of the content of the curriculum. Items were scored as correct or incorrect, summed, and converted to a percentage correct score for analysis.

In the process evaluation section of the posttest, students in the intervention conditions rated the program on six positive adjectives (e.g., “enjoyable,” “believable,” “interesting”), using four-point scales (1 = definitely yes to 4 = definitely not). These items were averaged to comprise the program acceptance index (α=0.89). Also, students completed five items that evaluated the quality of program delivery (teacher encouraged student participation, showed respect toward students, demonstrated confidence, was prepared, and responded to students' comments). These items were averaged to comprise the index of the teacher's program delivery skills (α=0.92).

Assessment of Implementation Fidelity

Fidelity of program implementation was assessed with a classroom observation procedure that was used in previous TND trials (Rohrbach, et al. 2007). Our goal was to observe each teacher in both the REGULAR and the IMP-SUPPORT conditions twice, while he/she delivered the same TND lesson (#5) to two separate class groupings. For 11 of the 54 teachers, observation was possible during only one classroom period; thus, analyses of implementation fidelity data are based on a total of 97 observations. In Lesson #5, psychodrama techniques are used to simulate a talk show in which various negative consequences of drug abuse are presented. This lesson was selected for observation because it is highly interactive and the classroom observation procedure emphasizes assessment of program process. Observations were conducted by trained members of the Project TND staff, and observers were not blind to the experimental condition of the school.

The observation instrument assessed three domains of implementation fidelity: classroom process, quality of delivery, and perceived student acceptance. All of the fidelity items utilized seven-point rating scales that specified behaviorally anchored criteria for the end- and midpoints of the scales (e.g., Mowbray et al. 2003). The classroom process index averaged three items assessing how well the lesson went overall, the extent to which the teacher elicited student participation and responses, and whether the objectives of the lesson were met (α=0.88). The quality of delivery index averaged items assessing teacher enthusiasm and confidence, and the extent to which he/she treated students respectfully (α=0.92). The perceived student acceptance index averaged three items that measured how interested the students appeared to be, how much they seemed to like the teacher, and class control (α=0.93). Because these three indexes were highly inter-correlated (r=0.95), for data analyses they were averaged and standardized (mean=0; SD=1) to create a composite implementation fidelity score (α=0.96).

Data Analysis

Data analysis for program effects on immediate outcome variables was completed by using a generalized mixed-linear model (Murray and Hannan 1990) in the SAS statistical package (SAS 2004). Condition was considered a fixed effect variable; fixed at desired experimental levels (school). School, which was the unit for randomization, was considered as a random factor (within program conditions). This specification allows for both the statistical accounting of intraclass correlation within clustered units (school) on computed significance levels, and for the logical generalization of findings beyond the specific sample. The variables adjusted for in the analyses included age, gender, ethnicity, propensity-for-attrition (to be explained later), school district, and the time span between pretest and posttest assessment.

The second set of analyses was restricted to the two program conditions. First, using the mixed-linear model described above, the two conditions were compared in regard to students' self-reported evaluations of the curriculum and program delivery process. Second, to compare the two intervention conditions on implementation fidelity, again we used a generalized mixed-linear model with condition considered as a fixed effect. Class grouping and teacher were considered random factors, since the fidelity score was nested within teachers by class grouping (Murray 1998). Other variables adjusted for in the analyses included teachers' gender, ethnicity (white vs. non-white), subject taught (physical education vs. health), and teaching experience (in years).

Results

Immediate Program Effects on Students

Assessment of Attrition Bias

To assess potential sampling bias due to subject attrition at the immediate posttest, a comparison was made of the student sample that was lost at the posttest (n=363) to the retained sample (n=2,983) on 17 key baseline measures. Measures included: age, gender, ethnicity, whether the subject lived with both parents, education level of parents, subjects' intentions regarding use of substances, and the other immediate outcome measures. The comparisons utilized chi square or t-test models to indicate statistically significant differences (two-tailed p value at the 0.05 level). The analyses showed comparability between the lost-to-follow-up and retained subjects with regard to gender, ethnicity, beliefs in health-as-a-value, and parents' education. The following differences between the two groups were found: retained subjects were younger (14.9 vs. 15.0 years old, p=0.005), less likely to be enrolled in an alternative/continuation school (4.1% vs. 14.6%, p<0.0001), more likely to be living with both parents (62.3% vs. 50.1%, p<0.01), reported a lower perceived likelihood of addiction (1.13 vs. 1.20, p=0.003), reported stronger beliefs in the immorality of drug use (3.35 vs. 3.04, p<0.0001), had higher program-related knowledge (48% vs 46% correct, p=0.01), reported less agreement with pro-drug use myths (21% vs. 27%, p<0.0001), and were less likely to intend to use substances in the next 12 months (for all four substances, p<0.001).

Because of the incomparability between the retained and lost-to-follow-up groups, it is crucial to amend the situation by statistically adjusting for significant differences between them. Based on the formula predicting actual attrition status at the immediate follow-up from the set of 17 pretest variables, a propensity-for-attrition score was calculated for each subject in the analytic sample (Rosenbaum and Rubin 1984). This score represented the tendency toward attrition that was computed based on statistical rules summarized from all subjects at pretest and assigned to each retained subject. In the analysis estimating the immediate effects of the program, the propensity-for-attrition score was included as a covariate in each of the statistical models.

Baseline Compatibility Across Conditions

Table 1 presents a summary of variables of interest across experimental conditions. At baseline, there were no statistically significant differences in these variables among the three conditions.

Table 1.

Summary of variables of interest at pretest, by experimental condition

Variable		Experimental condition						pa
		IMP-support training		Regular training		Control
		Mean	SE	Mean	SE	Mean	SE
Age (years)		14.83	0.17	15.14	0.18	14.79	0.18	0.32
Gender (% male)		44.0	2.7	52.0	2.9	45.5	2.9	0.11
Ethnicity (%)	White (non-Hispanic)	40.2	7.3	32.6	7.5	38.3	7.3	0.75
	Hispanic	33.5	6.6	25.6	6.9	28.5	6.7	0.71
	Asian	3.4	1.2	4.3	1.3	3.4	1.3	0.87
	African–American	13.7	5.7	24.2	5.9	18.4	5.7	0.45
	Other	2.8	0.9	4.7	1.0	3.7	1.0	0.35
	Mixed	6.4	0.9	8.6	1.0	7.7	1.0	0.33
Live with both parents (%)		65.4	2.7	59.3	2.9	62.4	2.9	0.30
Parents' educationb		3.84	0.13	3.84	0.13	3.68	0.13	0.59
Program-related knowledge (% correct)		48.2	1.0	47.7	1.0	48.1	1.0	0.95
Substance use intentionsc	Cigarette use	1.54	0.07	1.62	0.08	1.46	0.08	0.33
	Alcohol use	2.39	0.11	2.35	0.11	2.24	0.11	0.62
	Marijuana use	1.58	0.09	1.72	0.10	1.63	0.09	0.56
	Hard drug use	1.19	0.03	1.21	0.03	1.17	0.03	0.61
Pro-drug use myths (% agreement)		20.0	1.0	24.0	1.0	21.0	1.0	0.19
Addiction concernd		1.14	0.02	1.12	0.02	1.15	0.02	0.55
Immorality of drug usee		3.38	0.06	3.29	0.06	3.36	0.06	0.58
Health-as-a-valuef		3.76	0.02	3.78	0.02	3.75	0.02	0.40
Negative coping strategiesg		2.53	0.05	2.55	0.05	2.47	0.05	0.51
Propensity-for-attrition score		2.57	0.21	2.03	0.21	2.45	0.21	0.18
No. of days from pretest to posttest		37.8	2.8	44.3	2.9	41.1	2.8	0.28
Retention at posttest survey (%)h		90.0	2.8	82.8	2.9	88.1	2.9	0.20

^ap value from F test for compatibility across three conditions

^bMean education across parents (1 = did not complete eighth grade to 6 = completed graduate school)

^cPerceived likelihood of using substance in the next 12 months (1 = definitely not to 5 = very likely)

^dPerceived likelihood of becoming a drug addict (1 = not at all likely to 4 = very likely)

^eFour-point scale; higher score denotes stronger anti-drug belief

^fFour-point scale (1 = not at all important to 4 = very important)

^gFour-point scale (1 = infrequent to 5 = frequent use of strategy)

^hN for retention rate is 3,346

Overall Immediate Program Effects

Table 2 presents the program effects on the immediate outcome variables, showing the net effects for each intervention condition relative to controls, the net effects for both intervention conditions combined relative to controls, and the difference in effects between the two intervention conditions. The overall program effects analysis indicated the mean program-related knowledge score failed to change for subjects in the CONTROL condition from pretest to post-test (β±SE 1.08±1.96), but it increased in the IMP-SUPPORT (15.03±2.0) and REGULAR conditions (11.44± 1.97). There was a significant positive net program effect on the designated knowledge items (change in program conditions combined minus change in control condition: 12.38+ 1.61, p<0.0001).

Table 2.

Program effects on immediate outcome variables

Immediate outcome variable	Difference from pretest to posttest						Net difference between conditions												ICCb
	IMP-support		Regular		Control		IMP-support vs. control			Regular vs. control			Any TND vs. control			IMP-support vs. regular
	Beta	SE	Beta	SE	Beta	SE	Beta	SE	pa	Beta	SE	pa	Beta	SE	pa	Beta	SE	pa
Program-related knowledge	15.03	2.00	11.44	1.87	1.08	1.96	13.95	1.77	0.000	10.36	1.89	0.000	12.38	1.61	0.000	3.59	1.87	0.061	0.109
Intentions regarding
Cigarette use	−0.03	0.05	0.03	0.05	0.06	0.05	−0.10	0.04	0.010	−0.03	0.04	0.415	−0.07	0.03	0.033	−0.06	0.04	0.085	0.000
Alcohol use	0.08	0.07	0.10	0.07	0.12	0.07	−0.05	0.05	0.378	−0.02	0.06	0.739	−0.04	0.05	0.458	−0.03	0.05	0.612	0.010
Marijuana use	0.01	0.05	0.04	0.05	0.09	0.05	−0.09	0.04	0.026	−0.05	0.04	0.262	−0.07	0.03	0.045	−0.04	0.04	0.310	0.006
Hard drug use	−0.01	0.04	0.06	0.04	0.09	0.04	−0.11	0.03	0.001	−0.04	0.03	0.286	−0.08	0.03	0.009	−0.07	0.03	0.034	0.007
Pro-drug myths	−0.74	1.96	0.86	1.89	4.29	1.94	−5.03	1.62	0.003	−3.43	1.76	0.057	−4.37	1.44	0.004	−1.60	1.69	0.350	0.034
Addiction concern	0.04	0.02	0.03	0.02	0.06	0.02	−0.02	0.02	0.182	−0.03	0.02	0.096	−0.03	0.02	0.092	0.01	0.02	0.608	0.009
Immorality of drug use	0.09	0.04	0.08	0.04	0.01	0.04	0.08	0.03	0.003	0.07	0.03	0.031	0.08	0.02	0.003	0.02	0.03	0.549	0.003
Health-as-a-value	−0.01	0.03	−0.01	0.03	−0.06	0.03	0.05	0.02	0.016	0.05	0.02	0.037	0.05	0.02	0.010	0.00	0.02	0.934	0.002
Negative coping strategies	0.02	0.06	0.01	0.06	0.11	0.06	−0.08	0.04	0.066	−0.09	0.05	0.063	−0.09	0.04	0.038	0.01	0.04	0.798	0.001

All models were adjusted for age, gender, ethnicity, time interval from pretest to posttest, school district, and propensity for attrition

^ap based on two-tailed t test

^bIntraclass correlation (school level) for the change score from pretest to posttest

With regard to substance use intentions, relative to the controls there was a significant decrease in the perceived likelihood of using cigarettes, marijuana, and hard drugs in the future for students in the IMP-SUPPORT condition (p's<0.05). When students in both of the intervention conditions combined were compared to control students, there was a significant net program effect on cigarette, marijuana, and hard drug use intentions (p's<0.05). In addition, there was a significant overall program effect on three of the four belief constructs, including pro-drug myths (p<0.01), immorality of drug use (p<0.01), and health-as-a-value (p<0.05), and a marginally significant effect on the fourth belief construct, addiction concern (p<0.10). There was an overall program effect on the skills construct, negative coping strategies (p<0.05).

Comparison Between Training Interventions

As is shown in Table 2, the IMP-SUPPORT condition produced a significantly stronger effect on hard drug use intentions (p<0.05), a marginally significant stronger effect on cigarette use intentions (p=0.09), and a marginally significant greater gain in program-related knowledge (p=0.06), compared to the REGULAR condition.

Students' Evaluation of the Program

The two training conditions were compared with regard to students' responses to the program. Relative to the REGULAR condition, students in the IMP-SUPPORT condition had a higher rating of their teacher's program delivery skills (on a seven-point scale, a mean of 6.18±0.10 in IMP-SUPPORT vs. 5.91±0.10 in REGULAR; p<0.05). Likewise, students in the IMP-SUPPORT reported higher levels of overall program acceptance (mean of 2.34±0.04 on a four-point scale) compared to those in the REGULAR group (2.21±0.04; p<0.05).

Effects of Training Interventions on Implementation Fidelity

To determine the baseline comparability of teachers in the IMP-SUPPORT relative to REGULAR training condition, we compared the two conditions on four teacher background characteristics, including gender, ethnicity (white vs. non-white), mean years of teaching experience, and core subject area (physical education vs. health). There were no significant differences in these characteristics between the two groups of teachers.

We compared the standardized composite implementation fidelity score of classes in the two intervention conditions. The statistical models included the four teacher background characteristics as covariates. On the basis of our directional hypothesis, we used a one-tailed test of statistical significance. The results indicated higher implementation fidelity in the IMP-SUPPORT condition classes (mean=0.15, SD=0.87) relative to the REGULAR classes (mean=−0.22, SD=1.14; p<0.05).

Discussion

The present study demonstrates that immediate effects of TND may be produced when the program is implemented in a wide variety of high schools by classroom teachers who have received program-specific training prior to program delivery. When we compared students in the TND program conditions (combined), relative to students in the control condition, we found significant overall program effects on eight of the ten immediate outcome variables. These results are consistent with those of Project TND efficacy trials, which have demonstrated immediate program effects when the program is taught by project health education specialists (Dent et al. 1998; Rohrbach et al. 2007; Skara et al. 2005; Sussman et al. 2002). Furthermore, the magnitude of change in immediate outcomes we observed in the present study appears to be comparable to that reported in our previous studies. For example, the net effect of Project TND on increases in program-related knowledge scores from pretest to posttest (i.e., percentage of knowledge items correct), in the treatment groups relative to controls, has ranged from a mean of 7% to 16% (Dent et al. 1998; Skara et al. 2005; Sussman et al. 2002). In comparison, the net mean increase in knowledge scores in the present study was 12%.

As outlined in the RE-AIM model, evidence that prevention programs are effective when implemented in real-world settings is one of the critical measures of the potential for these programs to achieve a significant public health impact. The present findings contribute to a small, but growing, body of evidence for the effectiveness of school-based substance abuse prevention programs that have been tested under real-world conditions (e.g., Botvin et al. 2001; Ellickson et al. 2003). However, because the results of the study focused on hypothesized mediators of TND program effects, including both implementation fidelity and immediate student outcomes such as substance-related beliefs and substance use intentions, they should be considered only suggestive of program effectiveness. In future publications, we will examine the effects of real-world TND program implementation on students' substance use, including comparing the effects to those of TND efficacy trials.

Another important issue addressed in this study is the extent and type of training that is required in order for classroom teachers to deliver evidence-based prevention programs with fidelity and achieve targeted outcomes. Although previous research has demonstrated the effectiveness of pre-implementation training workshops (e.g., Basen-Enquist et al. 1994; Connell et al. 1985; Dusenbury et al. 2003; Parcel et al. 1991; McCormick et al. 1995; Perry et al. 1990; Ross et al. 1991), to date, there have been few randomized evaluations of training models designed to supplement pre-implementation training and support program providers as they begin implementation of evidence-based programs. In the present study, we tested the effectiveness of a standard pre-implementation training workshop relative to a comprehensive training and support model that included workshop training, coaching during implementation, technical assistance, and web-based resources for teachers. We found that relative to the standard training, the comprehensive training intervention resulted in a significantly higher level of implementation fidelity among teachers, based on classroom observational assessments of program delivery. When we compared the two training conditions with regard to immediate student outcomes, we found that the comprehensive training intervention had a significantly stronger effect on hard drug use intentions (p<0.05), and a marginally significant effect on cigarette use intentions and program-related knowledge (p's< 0.10). However, there were no significant differences between the two training conditions on the seven remaining immediate outcome variables. With regard to students' evaluation of the program process, the ratings of program acceptance and quality of program delivery were significantly higher in the comprehensive training relative to the standard training condition. These results suggest that comprehensive training approaches may improve implementation fidelity, but these improvements in fidelity may not result in stronger program outcomes.

The nature of our study design does not allow us to determine which components of our comprehensive training intervention accounted for its effect on implementation fidelity. The literature on coaching suggests that teachers can learn about the theory, rationale, key content, and mechanics of delivery of an innovative curriculum or program during a face-to-face training workshop, but mastery of the skills required for teaching it is achieved only when teachers have opportunities for practice, followed by feedback on the practice (Fixsen et al. 2005; Joyce and Showers 2002). Technical assistance can enhance the quality of program delivery by providing just-in-time suggestions for improving implementation and strategies for addressing implementation problems that may arise (Fagan et al. 2008; Kam et al. 2003). In order to guide the development of cost-effective approaches for training providers of evidence-based prevention programs, future research should examine the relative effectiveness of the components that comprise a multi-component training model.

One limitation of the study is that while the immediate outcome variables we examined are hypothesized to mediate program effects, to date we have not conducted studies that identify specific mediators of Project TND. Of the immediate outcome variables we examined, addiction concern and drug use intentions have been shown to predict adolescent substance use in previous studies (Rohrbach et al. 2005; Sussman and Dent 1996; Sussman et al. 2003a), suggesting that these may be important mediators. In addition, student acceptance of prevention programs has been positively associated with implementation fidelity (Rohrbach et al. 1993). However, as we have acknowledged, the conclusions about the effectiveness of program dissemination and large-scale implementation are tentative until we determine effects on adolescent substance use behaviors.

A second limitation of the study is that our classroom observers were not blind to the experimental condition of the school. Thus, we cannot rule out the possibility of bias in their ratings of implementation fidelity. Nevertheless, the use of observer assessments of program fidelity is an important strength of the study. Observations are thought to be more objective, valid, and reliable assessments of implementation fidelity than are teacher self-reports (Dane and Schneider 1998; Dusenbury et al. 2003; Hansen and McNeal 1999). A final limitation of the study is that we did not collect data about the use of technical assistance by teachers. In future studies, it would be useful to assess the extent to which teachers take advantage of technical assistance that is available via telephone, e-mail, and the project's web site.

In summary, the results of the present study provided evidence for the effectiveness of Project TND on immediate outcomes when the program is implemented by regular high school teachers. Furthermore, we found that a comprehensive training model, which supplemented a pre-implementation training workshop with ongoing support during the implementation phase, produced stronger implementation fidelity relative to a training workshop alone. These results suggest that in order to maximize the impacts of wide-scale dissemination of evidence-based programs like Project TND, ongoing training and support during program implementation may be required.