Three Year Trajectory of Teachers’ Fidelity to a Drug Prevention Curriculum

Abstract

Little is known about the trajectories over time of classroom teachers’ fidelity to drug prevention curricula. Using the “Concerns-Based Adoption Model” (C-BAM) as a theoretical framework, we hypothesized that teachers’ fidelity would improve with repetition. Participants comprised 23 middle school teachers who videotaped their administration of three entire iterations of the All Stars curriculum. Investigators coded two key curriculum lessons, specifically assessing the proportion of activities of each lesson teachers attempted and whether they omitted, added, or changed prescribed content, or delivered it using new methods. Study findings provided only partial support for the C-BAM model. Considerable variability in teachers’ performance over time was noted, suggesting that their progression over time may be nonlinear and dynamic, and quite possibly a function of their classroom and school contexts. There was also evidence that, by their third iteration of All Stars, teachers tended to regress toward the baseline mean. That is, the implementation quality of those that started out with high levels of fidelity tended to degrade, while those that started out with very low fidelity to the curriculum tended to improve. Study findings suggest the need for ongoing training and technical assistance, as well as “just in time” messages delivered electronically; but it is also possible that some prevention curricula may impose unrealistic expectations or burdens on teachers’ abilities and classroom time.

INTRODUCTION

After a lengthy and repetitive process of program development and evaluation, the field of adolescent substance abuse prevention has identified a number of school-based curricula that have demonstrated positive effects over time, and are so recognized on both federally (Elliot, 1998; NIDA, 2003; SAMHSA, 2008) and privately (Drug Strategies, 1998) sponsored registries. The most recent prevalence estimates available suggest that 34.6% of the nation’s public middle schools had adopted one of these curricula as of 1999 (Ringwalt et al., 2002) and it is likely that more have done so in the interim, spurred by mandates from funding agencies (Hansen, 2001) and from No Child Left Behind legislation (Hallfors, Pankratz, & Hartman, 2007). However, drug prevention curricula are unlikely to achieve their objectives unless they are taught with fidelity. Fidelity, sometimes referred to as integrity, adherence, or quality of program delivery, is an important characteristic of program implementation (Dane & Schneider, 1998; Domitrovich & Greenberg, 2000; Dusenbury, Brannigan, Falco, & Hansen, 2002; Higgins & Hogan, 1999). It includes both the proportion of lesson content attempted, as well as modifications to that content (Jackson-Newsom, 2008). Hill and colleagues (2006) classified modifications as omissions, additions, and changes, of which omissions were the most prevalent modification type while changes were the least commonly observed.

There is now abundant evidence from a variety of sources that the quality of program implementation differs considerably across teachers (Dusenbury et al., 2002; Dusenbury, Brannigan, Hansen, & Walsh, 2005; Melde, Esbensen, & Tusinski, 2006; Pankratz et al., 2006; Ringwalt et al., 2003; Rohrbach, Graham, & Hansen, 1993; Tappe, Galer-Unti, & Bailey, 1995; Tortu & Botvin, 1989), which is hardly surprising given their orientation towards creativity and flexibility in the classroom (St Pierre & Kaltreider, 2004). The literature is equally clear that the effects of prevention curricula are likely to suffer as a result of adaptation (Botvin, Baker, Dusenbury, Botvin, & Diaz, 1995; Dane & Schneider, 1998; Drake, McHugo, Becker, Anthony, & Clark, 1996; Dusenbury, Brannigan, Falco, & Hansen, 2003). Further, effect sizes tend to attenuate as evaluations move from conditions of efficacy to program implementation in more naturalistic settings (Connell & Turner, 1985; Dodge, 2001; Glasgow, Lichtenstein, & Marcus, 2003; Tobler et al., 2000). However, there is a dearth of research concerning factors that generate successful program implementation (Dane & Schneider, 1998; Fagan & Mihalic, 2003; Wandersman et al., 1998).

Diffusion of innovation theory suggests that teachers’ fidelity to the prevention curricula they administer will deteriorate with repetition because implementers tend to make novel products their own through “reinvention”(Rogers, 2003). However, very little is actually known about the natural progression of fidelity over time within the context of teachers’ administration of classroom-based curricula. Indeed, most evaluations are based on initial implementations, and few studies actually track teachers past their first cycle of program delivery. McCormick and her colleagues (McCormick, Steckler, & McLeroy, 1995) found a decline in the proportion of curricula activities that teachers implement from one school year to the next, a process that Bond and colleagues (Bond, Evans, Salyers, Williams, & Kim, 2000) characterized as “program drift.” Similarly, Connell & Turner (1985) found that in their second year of implementation teachers administered fewer components of their curriculum, adapted it more, and taught it for fewer hours. Gingiss and Hamilton (1989) reported that their teachers’ initial implementation of a curriculum tended to be mechanical, while in later administrations the teachers relied less on curriculum guides and became more spontaneous. While the limited evidence available suggests that fidelity decreases overtime, Rogers and others (e.g., Berman & McLaughlin, 1976) assert that reinvention is positively associated with program sustainability, though not necessarily effectiveness, as practitioners use and modify the innovations they have adopted. To date, no one has linked different types of curriculum modifications to the program’s outcomes.

Hall and his colleagues (1987) have developed a framework titled the “Concerns-Based Adoption Model” (C-BAM) that provides specific guidance as to changes that teachers and other practitioners are likely to make over time when they administer novel curricula or programs. C-BAM is now widely accepted in the field of education (Bailey & Palsha, 1992; Cheung, 2002) and has been called the most “robust” and “empirically grounded” theory to understand the process of implementation of educational innovations (Anderson, 1997; Hall, Loucks, Rutherford, & Newlove, 1975). The model’s developers suggest that teachers’ “Level of Use” of an educational innovation – that is, their behaviors and skills in regard to the curriculum (Hall & Hord, 1987; Hord, 1987)– will evolve or develop systematically with each iteration through a hierarchy from initial orientation (Level I) to mastery (Level VI) (Shotsberger & Crawford, 1999). The first two stages, orientation (Level I) and preparation (Level II), are those at which a teacher takes the initiative to learn about the innovation and develops definite plans to use the innovation. Level III, “Mechanical use,” is the behavioral stage of most pertinence to teachers who are administering a curriculum for the first several iterations (Hall & Hord, 1987), when their use is likely to be disjointed, uneven, superficial, and characterized by managerial problems (Cheung, 2002; Hall et al., 1975). These problems may relate to the organizational, administrative, or logistical mechanics of curriculum implementation, namely its content, materials, lesson planning, and any new teaching skills or classroom instructional or management behaviors required (Anderson, 1997). At this level, teachers may feel awkward and inadequate; like cooks confronted with a new recipe, they are focused on simply coping with demands of the new material (Huberman & Miles, 1984). Insofar as they make any changes to the curriculum, these changes are likely to reflect the teacher’s logistical or organizational requirements, as opposed to their students’ learning needs. That is, the purpose of their adaptations will be to decrease the challenges they face as a result of curriculum delivery (Hall & Hord, 1987).

As teachers progress to the next level of familiarity with the curriculum, they establish a routine pattern of behaviors that they find satisfactory (Level IV(a)), which is characterized by “equilibrium,” “stability” and full fidelity to the curriculum. Many teachers never advance beyond this level, while others may then begin to adapt it (Level IV(b)), based on their understanding of its effects on their students (Level IV(b)) (Hord, 1987; Horsley & Loucks-Horsley, 1998). Of particular note, the developers’ experience with the model has suggested that at least three iterations over as many years are required before teachers gain sufficient mastery of the curriculum that its implementation is both efficient and effective, although experience per se does not necessarily lead to the competence characterized by this level (Hall et al., 1975; Loucks-Horsely, 1996).

The purpose of this study is to examine changes to the fidelity with which teachers implemented a novel drug prevention curriculum three times over the course of three successive years. We hypothesized that we would see considerable variation in fidelity in teachers’ performance relative to the content specified by the curriculum guide. We hypothesized further that teachers’ fidelity to the curriculum would improve during this period, as they became more familiar with the requirements of its administration and made the transition from Level of Use Stage III “Mechanical” to Stage IV(a) “Routine.” We believed it was unlikely that teachers would progress beyond Stage IV(a) over the course of three implementations; however, to the extent that they did, the C-BAM model suggests that their fidelity would then begin to deteriorate.

METHODS

The All Stars Curriculum

All Stars is an evidence-based prevention program designed to reduce adolescent substance use, sexual behavior, and violence through changes in specific mediating variables (Harrington, Cheah, Norling, Hoyle, & Duvall, 2002; McNeal, Hansen, Harrington, & Giles, 2004). The mediating variables targeted by All Stars include normative beliefs, personal commitments not to use substances, perceptions that substance use may interfere with personal values and lifestyles, bonding to school, and perceptions of positive parental attentiveness. The curriculum consists of 24 sessions, of which 13 are required and are administered to the entire class during classroom time. The program includes interactive and cooperative learning activities such as debates, games, and general discussion. Each session is designed to affect at least one of the curriculum’s five mediating variables specified above. All Stars has been extensively implemented and evaluated, and has repeatedly yielded evidence of effectiveness (Hansen & Dusenbury, 2004; Harrington et al., 2002).

For the purposes of this paper, we coded Lessons 4 and 11, which we selected because they required extensive interactivity between teachers and students, and thus were expected to demonstrate variability among teachers’ ability to successfully administer the program. Given the importance of interactivity to effective prevention program, we also thought that coding a highly interactive lesson would produce the most useful information to advance the field. The sequence of these lessons in the curriculum also provided an indication of fidelity of lessons located toward the beginning and end of curriculum delivery. The primary purpose of Lesson 4 is to motivate students to set goals for their future, while the function of Lesson 11 is to encourage students to commit to developing good habits. Lesson 4 contains a wealth of instructional detail and requires individual, small group, and then large group work, strategies that are collectively designed to lead students through a process of thinking about their wanted (and unwanted) futures. The bulk of Lesson 11 constitutes a discussion and summary steps that pertain to understanding commitments and good habits. It also includes a worksheet exercise designed to help students share their understanding of good habits to develop.

Study Participants

Forty-four schools participated in our randomized trial testing the effect of personal coaching on the fidelity with which teachers implemented the All Stars curriculum (Ringwalt et al., 2007). Within each school, one teacher per year implemented All Stars to a class of 7^th graders for up to three years. In total, 12 teachers implemented it once, 13 implemented it twice, and 23 implemented it all three times. Most teachers who only taught the curriculum once or twice did so because they either left their school (44%) or took a new position within it that did not allow them to continue teaching All Stars (20%). Two teachers (8%) were resistant to completing a second implementation and three (12%) faced significant personal and professional challenges that did not allow them to teach a third time. The remaining four (16%) replaced an earlier teacher who left the study and only had the opportunity to teach one or two iterations of All Stars.

To examine if there were any differences between those teachers that completed only one or two iterations of All Stars versus those that were able to complete all three, we regressed teacher characteristics and baseline fidelity on the number of All Stars iterations completed (one or two versus three). Those completing all three iterations of All Stars were less likely to be classroom teachers than their counterparts (57% vs. 80%) (see Table 1). They also omitted more content (33.4% of steps attempted versus 26.1%) and used new methods less frequently (12.2% of steps attempted versus 18.5%) in Lesson 4 than their counterparts (see Table 2). There were no differences between those who completed one or two iterations of All Stars and those who completed three for any other characteristic or fidelity measure assessed.

Table 1.

Teacher demographics and baseline adherence by number of times the teacher implemented All Stars

		Implemented One or Two Times (N = 25)	Implemented three times (N = 23)
Teacher Demographics		%	%
Age, Mean (SD)		41 (10)	39 (10)
Gender (male)		28	13
Race/ethnicity
African-American		52	44
White		36	52
Other		12	4
Hispanic/Latino		16	0
Graduate degree		48	57
Classroom teacher		80†	57
Years experience in education, Mean (SD)		9 (7)	11 (7)
Experienced in teaching substance use prevention		28.0	47.8
Baseline Adherence		Mean (SD)	Mean (SD)
Attempts	Lesson 4	74 (13.4)	75 (19)
	Lesson 11	77 (16.5)	76 (16)
Omissions	Lesson 4	26 (12.4)†	33 (14)
	Lesson 11	35 (18.1)	42 (11)
Additions	Lesson 4	32 (14.5)	25 (15)
	Lesson 11	69 (16.9)	70 (15)
Changes	Lesson 4	15 (12.6)	13 (18)
	Lesson 11	15 (13.1)	26 (15)
New methods	Lesson 4	19 (9.3)*	12 (9)
	Lesson 11	10 (8.9)	12 (7)

^†p<.10

^*p<.05

^**p<.01

^***p<.001

Table 2.

Teacher fidelity by implementation year (n=23)

		Baseline Fidelity	Year 2 Fidelity	Year 3 Fidelity
	Lesson	Mean (SD), Median, Range	Mean (SD), Median, Range	Mean (SD), Median, Range
Attempts	4	75 (19), 75, 31–100	68 (20), 69, 25–94	66 (25), 75, 6–94
	11	75 (16), 77, 38–92	71 (20), 77, 23–92	80 (15), 85, 38–100
Omissions	4	33 (14), 33, 8–67	25 (21), 23, 0–100	28 (19), 23, 7–100
	11	42 (11), 42, 18–60	51 (16), 50, 27–100	50 (15), 50, 25–80
Additions	4	25 (15), 21, 7–60	26 (12), 23, 0–50	25 (21), 21, 0–100
	11	70 (15), 71, 44–100	50 (18), 55, 17–80	42 (21), 39, 0–80
Changes	4	13 (18), 8, 0–80	12 (16), 7, 0–50	15 (22), 11, 0–100
	11	22 (15), 18, 0–60	16 (15), 13, 0–60	13 (14), 9–40
New methods	4	12 (9), 13, 0–33	13 (9), 11, 0–33	11 (8), 9, 0–25
	11	12 (7), 10, 0–30	13 (20), 9, 0–100	10 (5), 10, 0–20

Teachers that implemented all three times were predominately female (87%), classroom teachers (57%), and master’s level professionals (57%) (see Table 1). On average, they were 39.3 years old and had 10.5 years experience in education. They predominately identified themselves as White (52%) or African American (44%). Nearly half had experience in substance use prevention (48%).

All teachers were exposed to the standard All Stars two day in-person training, received lesson specific teaching tips via email, and had access to follow-up consultation from the lead All Stars trainer upon request. Follow-up consultation was rarely used. Intervention teachers were also assigned a personal coach who provided specific feedback on their implementation. Because the intervention was not effective in changing either teachers’ fidelity or student behaviors (Ringwalt, et al., 2007), it is not further considered in this manuscript. Most teachers completed their first implementation during the spring or fall of 2004, their second implementation in 2005, and their third implementation in 2006.

As a criterion for recruitment into the parent study, teachers agreed to videotape each lesson they implemented by placing the camcorder we provided in the back of their classrooms and focusing it towards the front. We identified videotapes only by unique teacher identification numbers and the study year. The videotapes, which were mailed directly to study staff and were not viewed by anyone in the school system, were then rated by trained coders using the observation form described below. All study procedures were reviewed and approved by PIRE’s Institutional Review Board for the Protection of Human Subjects.

Coder Observation Form

The coder observation form was created through a highly iterative process of reviewing and coding numerous videotapes of teachers’ delivery of All Stars lessons, and evaluating the extent to which our instrument and coding instructions adequately documented the complex nature of our observations. Coders checked whether teachers attempted each step in each lesson. Steps constituted the smallest unit of instruction that teachers completed, for which there were specific goals, concrete instructions for teachers to follow, and specific prompts for student-centered questions. Coders rated a step as attempted if any of the material presented in the step was delivered by the teacher, regardless of the amount. For each step attempted, coders then checked whether there were adaptations to content (the substantive component of the material) and methods (the instructional strategies by which the content was delivered to students). In total there were 16 possible steps in Lesson 4, and 13 in Lesson 11

We coded three mutually exclusive categories of adaptations to curriculum content: changes, omissions, and additions. Changes included any rewording of material as written, including modifications to statements, questions, or instructions (beyond simple rephrasing). This category of modifications was limited to alterations to content but excluded alterations involving teaching method or strategies. Omissions included any deletion of content within a step, and additions included any new material presented that was not specified in the curriculum. To assess modifications to teaching methods, coders indicated whether or not any new strategies were used during the delivery of the material within each step. New methods were defined as any change in teaching strategy as prescribed by the curriculum

Coding Process

Coders were graduate students who received the standard training in the All Stars curriculum, which was followed by extensive training in the coding process. All tapes of teachers’ initial implementation of both Lessons 4 and 11 were rated by two coders to establish inter-rater reliability (Jackson-Newsom et al., 2008), and 20% of subsequent tapes were double coded to facilitate continuing assessment. A codebook was created to capture decision rules that we made as common modifications were identified and codified; and the third author reviewed ratings throughout the process, provided detailed feedback to each coder, and served as a “gold standard” when there was a discrepancy between coders. We then used a multi-pronged approach to assess level of agreement between the two raters in which we assessed concordance with log odds ratios, evaluated marginal homogeneity with McNemar’s tests, and reported proportions of agreement. Percent agreement was above 80% for all categories.

Measurement

We began the creation of fidelity variables by calculating the percent of steps attempted. We then summed the number of steps with omissions, additions, changes, or new methods, respectively, and then divided this by the total number of steps attempted. The resulting fraction created a variable that indicated the percent of steps attempted that contained modifications. For each of the two lessons coded, this procedure yielded 5 ratings.

Data Analyses and Results

We were interested in modeling teachers’ trajectories of fidelity to the curriculum for all those who implemented the curriculum three times. Ideally we would have liked to use a multilevel model with random intercepts and slopes to address this research question. However, the sample size was too small to support a multilevel model that included both random intercepts and slopes. As an alternative, we computed fidelity means, standard deviations, medians, and ranges by year and then implemented a “brute force” multilevel model. To do so, we estimated a simple linear regression of fidelity on year of implementation for each teacher in our sample:

$${\mathit{Adherence}}_{ij}={\beta }_{0j}+{\beta }_{1j}{\mathit{Year}}_{ij}$$ (1)

In the above equation, the subscript i represents year of implementation. There are three observed years of implementation, j (I = 3) for 23 teachers (J = 23). Instead of using a multilevel modeling statistical package (e.g., SAS Proc Mixed or HLM) that would have utilized all of the available data from the sample to estimate the trajectories for each individual teacher¹, we used SAS Proc GLM to estimate simple linear regressions. We selected this procedure because our data were slightly skewed and Proc GLM does not require conditional normality of the dependent variable to yield unbiased parameter estimates. Standard error estimates are biased if normality conditions are not met when using GLM; however, our “brute force” model does not make use of the standard error estimate, so normality is not essential. Individual estimates for the intercepts (β_{0 j}) and slopes (β_{1 j}) were outputted for each type of fidelity, and we then calculated the mean and standard deviation of each parameter and their inter-correlations. These results are approximately equivalent to the results that would have been obtained had we been able to employ a random intercept, random coefficient multilevel model. That is, we were not interested in each teacher’s parameter estimates, but in both ‘fixed effects’ (the mean of the intercepts, β̄_{0 j} and the mean of the slopes, β̄_{1 j}), and ‘random effects’ (the variance of the intercepts and slopes).

Table 2 displays the fidelity means, standard deviations, medians, and ranges by study year. Table 3 then summarizes the results from simple linear regressions of each type of fidelity on time for the 23 teachers who implemented All Stars three times. The first column indicates the mean baseline fidelity (i.e., the fidelity intercept), and its standard deviation shows individual teachers variation around the mean. The second column indicates the degree of change in each type of fidelity per year (i.e., the fidelity slope), and its standard deviation show the individual variation in fidelity trajectories. For this, a negative number for attempts means that fidelity declined over time, while a positive number means that fidelity increased. For all other fidelity categories (i.e., omissions, additions, changes, new methods), a negative number indicates greater fidelity over time and a positive number indicates less fidelity. The correlation between the intercept and the slope indicates the relationship between teachers’ baseline fidelity levels and their trajectories in regards to fidelity over time. For this, a negative number indicates that teachers who started out with above average fidelity tend to decline over time, and that teachers who started out with below average fidelity tend to improve over time. The results presented in this table constitute a small-sample alternative to a multilevel model with random slopes.

Table 3.

Longitudinal Trajectories of Fidelity for Teachers Administering All Stars Three Times Over Three Years (n=23)

Type of fidelity	Lesson	Baseline fidelity (fidelity intercept)		Change over time (fidelity slope)		Correlation between baseline fidelity and change over time
		Mean	SD	Mean	SD
Attempts	4	.77***	.22	−.03	.09	−.63**
	11	.70***	.22	.02	.06	−.50*
Omissions	4	.42***	.26	−.08*	.14	−.94***
	11	.39***	.21	.05*	.09	−.91***
Additions	4	.29***	.21	−.02	.08	−.85***
	11	.82***	.30	−.14***	.15	−.93***
Changes	4	.17*	.27	−.02	.10	−.93***
	11	.24***	.23	−.04	.09	−.84***
New methods	4	.14***	.13	−.01	.06	−.84***
	11	.16***	.14	−.02	.06	−.72***

^*p<.05,

^**p<.01,

^***p<.001

Table 2 shows that there is substantial variation in fidelity by teacher. A comparison of mean and median fidelity scores reveals that, while there is some skewness to our data, there are no obvious outliers that would lead to parameter bias. Table 3 shows that baseline fidelity rates are all significantly different from zero, with teachers most likely to attempt steps and least likely to use new methods within the steps they attempt. While few of the time trends were significantly different from baseline, teachers did make fewer omissions within the Lesson 4 steps attempted over the three year period, and more omissions (but fewer additions) within the Lesson 11 steps attempted. The standard deviations suggested that this degree of change varied from one teacher to another. The most interesting finding from our analyses was that, for all types of fidelity and for both lessons, intercepts and slopes were strongly negatively correlated. That is, teachers wo started out making more modifications decreased the number of these over time, whereas teachers who initially made fewer modifications tended to make more of these over time. We found a similar pattern for teachers’ lesson attempts over the three year period. Clearly, teachers tended to regress to the mean on all measures.

DISCUSSION

This study is the first to track changes in the fidelity with which middle school teachers administer substance use prevention curricula over time. We were not surprised to find a high degree of variation in the fidelity with which teachers initially implemented the All Stars curriculum. However, we did not expect to find that, regardless of their initial level of proficiency, all teachers regressed toward a mean level of fidelity in subsequent iterations of the curriculum. As such, our findings provided only partial support for the hypotheses we derived from the Level of Use model, which led us to believe that curriculum fidelity would generally improve over the period that we observed teachers’ implementation. Like many theories of behavior change, the Level of Use model may thus be somewhat reductionistic and need further elaboration.

Although the nature of the curriculum content varied across the two lessons we examined, the overall pattern for both suggests that teachers struggled to deliver all of the prescribed content in each lesson. For instance, teachers implemented only three-fourths of the curriculum steps for both All Stars Lessons 4 and 11 the first time they taught the curriculum. In regards to Lesson 4, which included a substantial amount of instructional detail, teachers tended to cover fewer steps over time, but omitted less material in those steps they did cover. For Lesson 11, which has a higher proportion of discussion-oriented steps, teachers tended to cover more steps over time, but may have compensated for increases in the time required to do so by omitting more material in the steps they did deliver. In other words, for Lesson 4, teachers did less over time, but they improved the quality of what they attempted to do. In Lesson 11, teachers tried to do more, but the quality of what they attempted seems to have suffered as a result. In each case, fidelity might be enhanced by allowing more time for instruction. However, given the many competing demands for instructional time in the nation’s schools, and the generally fixed length of class periods, any expansion of the time allotted for teaching drug prevention curricula seems unlikely.

A post hoc review revealed that difference noted between the two lessons in the precise pattern of attempts and modifications over time can be partially attributed to differences in the nature of the content they contained. Similar to what is found with other curricula (Tricker & Davis, 1988; Tappe, et al., 1995; Tortu & Botvin, 1989; Botvin, et al., 2001; McCormick, Steckler, & McLeroy, 1995; Rohrbach, et al., 1993), it appears that teachers may have struggled initially to deliver all of the instructions specified in Lesson 4, but that in subsequent iterations they became increasingly comfortable with these instructions. However, even as they omitted less material within the steps they completed, their fidelity of implementation may have suffered because they still ran out of time to cover the lesson’s summary steps.

In Lesson 11, it appears that teachers made more of an effort over time to provide complete instructions for the lesson’s worksheet exercise. As teachers included more of this exercise, they tended to omit some of the summary points and questions specified for the lesson’s didactic and interactive steps. Many of these steps comprised multiple questions or summary points, and teachers may have come to believe that they did not need to cover each point in order to convey the desired message to their students.

While differences in teachers’ fidelity to the prescribed content of the two All Stars lessons are interesting, we believe that our study’s most important finding is that regardless of their initial proficiency in teaching the curriculum, all teachers tended to regress to the baseline mean fidelity score over time. That is, teachers who initially demonstrated a high degree of fidelity seemed to learn what content they could successfully modify in their second or third iterations of the curriculum. The C-BAM model suggests teachers can be expected to deliver a curriculum that is new to them in a mechanical fashion, in that they will follow the curriculum guide closely. Then, as they gain experience with the curriculum, they should feel more comfortable with the material and begin to make changes that they believe to be constructive. Conversely, those teachers who started out with below average fidelity may have realized that their modifications were unsuccessful, and they may then have sought to implement the curriculum more conscientiously in subsequent iterations. In both cases, study findings suggest that all teachers need continual support and reinforcement if they are to implement evidence-based curricula with fidelity. One cannot assume that teachers who perform well in their initial implementation year will continue to do so.

The study has several limitations. First, our ‘brute force’ estimates of each teacher’s individual fidelity trajectory in an OLS regression did not take the standard errors of the parameter estimates into account. Multilevel modeling, using empirical Bayes estimation, draws individual trajectory estimates closer to the sample mean in a manner that is proportional to the precision of the estimates. Because shrinkage from empirically derived Bayes estimates is greatest for small samples, our individual intercept and slope estimates are probably further from the mean than they would have been had we been able to estimate a multilevel model (Raudenbush & Bryk, 2002). Second, since we examined a relatively large number of outcomes – 10 in all – the findings we reported may have capitalized on chance. Third, we were able to code, with existing resources, only two of the All Stars curriculum’s total of 14 lessons; an examination of the remaining lessons might have revealed additional patterns of fidelity trajectories, and validated (or refuted) what we reported. Data coding, we found, constituted a highly repetitive and resource intensive process; we estimate that a complete coding of each of the two lessons for the entire set of teachers required 750(!) hours of raters’ and supervisors’ time.

Of further concern, it is possible that teacher attrition from the study’s first year to its third may have biased our results. To examine this potential confound, we compared the demographic characteristics and baseline fidelity scores of teachers with only one or two iterations of All Stars to those with all three. Overall the groups were similar, which is not surprising given that most of the attrition was due to position changes and not to any lack of desire to continue teaching the curriculum. Lastly, the schools that participated in the study were selected following a focused recruitment effort on the part of a community-based organization with which we contracted for this purpose. In that sense, the process was not “natural” as the C-BAM model typically presumes: that is, the impetus for and process of program adoption did not come from inside the school itself. However, the coaching intervention we were testing did not affect either teacher fidelity or student outcomes (Ringwalt, et al., 2009); as such, it is likely that fidelity trajectories are similar to those that could be expected in a more naturalistic context.

Study findings have implications for the ongoing support of teachers confronted with a novel curriculum, and particularly one that demands interactive skills and classroom management practices with which they are unfamiliar or uncomfortable. There is now a considerable literature that addresses the need for coaching to support teachers charged with the administration of drug prevention curricula (Ringwalt et al., 2009). However, many observers have expressed concern that in-service training alone may not result in positive and lasting changes in teaching practice (Bailey & Palsha, 1992). C-BAM offers both a conceptual and pragmatic framework by which to match teachers’ developmental level of concerns and competence in regards to a novel curriculum with the timing and content of the coaching they receive. Of particular importance is the need for technical assistance to continue over multiple cycles of use (Hall, Loucks, Rutherford & Newlove, 1975).

Unfortunately, this recommendation is at odds with the common practice of teacher preparation in the field of school-based drug prevention curricula, for which teachers are fortunate if they receive a one- or two-day training. This practice is entirely understandable, given both the financial and temporal challenges involved in bringing teachers together periodically for further training, and the likelihood that teachers’ needs for ongoing assistance are idiosyncratic. Personalized coaching or mentoring has met with only modest success (Rohrbach, et al. 2008; Ringwalt, et al., 2009) and is also beyond the means of many schools. However, investigators are studying the effects of a variety of other strategies, including providing teachers with access to a trainer by telephone, and sending “just in time” tips by e-mail that arrive immediately preceding the implementation of a particular lesson. There are now a number of studies devoted to developing an instrument to assess the level of use that individual teachers have achieved, as conceptualized by the C-BAM model (e.g., Bailey & Palsha, 1992); the results of these studies could potentially be used to tailor messages to teachers’ Level of Use.

The results of this study also highlight issues pertaining to the linearity of change, one of the lingering areas of confusion in the C-BAM model. The model suggests that practitioners will progress in incremental fashion through a set of clearly definable stages, although their exact number is still unresolved (Bailey & Palsha, 1992). On the other hand, it may be equally as appropriate to consider change to be nonlinear and dynamic, since teachers will begin at different starting points (Anderson, 1997) and may then diverge along a variety of pathways. It may thus be unrealistic to expect to find much consistency in trajectories of fidelity over time. Indeed, our study’s findings suggest that the process by which teachers master an unfamiliar curriculum is personal and idiosyncratic (Anderson, 1997).

In conclusion, we strongly recommend that further attention should be paid to how teacher fidelity to prevention curricula unfolds over time, and the implications to the prevention field of any trajectories found. Systematic changes in fidelity in an undesirable direction may suggest areas of improvement for developers in crafting both initial teacher training and follow-up reminders. Such changes may also suggest the need to modify curricula or their associated guides, especially if the demands placed on the teachers come to be recognized as unrealistic.

The implications for evaluators of changes in fidelity over time are equally salient. To the extent that fidelity improves over time, as the Level of Use model suggests, it would seem inappropriate to evaluate curricula that are administered by novice teachers. Regardless, an understanding of what teachers are – and are not – doing in the classroom, and how their implementation changes as a function of program repetition, is essential to efforts to disseminate and sustain drug prevention curricula in a manner that ensures their continued effectiveness.