Styles of Adaptation: The Impact of Frequency and Valence of Adaptation on Preventing Substance Use

Abstract

Purpose:

To be effective, evidence-based programs should be delivered as prescribed. This suggests that adaptations that deviate from intervention goals may limit a program’s effectiveness. This study examines the impact that number and quality of adaptations have on substance use outcomes.

Design:

We examined 306 video recordings of teachers delivering ‘All Stars’, a middle school drug prevention program. Multiple observers coded each recording, noting the number and type of adaptation each teacher made. Each adaptation was given a valence rating. Adaptations that were deleterious to program goals received negative valence ratings; positive ratings were given for adaptations that were likely to facilitate achievement of program goals; neutral ratings were given to adaptations that were expected to have neither a positive nor negative impact on program goals.

Findings:

All teachers made adaptations. Teachers were consistent across time in the types of adaptations they made, suggesting each teacher has a personalized style of adapting. Those who made few adaptations, and whose average adaptation was rated as being positive had a higher percentage of students who remained non-drug users. In contrast, teachers who made many adaptations, whether their average valence rating was positive, neutral or negative, failed to have as many students remain non-drug users. Measures of fidelity, including quality of delivery and teacher understanding were related to valence of adaptations, with better performance related to making positive adaptations.

Practical Implications:

Through training and supervision, teachers should be guided and encouraged to follow programs directions, making few adaptations and ensuring that adaptations that are made advance the goals of intervention. Programs should define acceptable and unacceptable ways they may be adapted.

Value:

This study provides significant evidence about the challenges that face disseminated evidence-based programs.

INTRODUCTION

Adolescent substance use continues to be a problem. After declines between 2002 and 2008, past 30-day illicit drug use among 12-17 year olds has risen from 9.3% in 2008 to 10.1% in 2010 (SAMHSA, 2011). Of notable concern is that daily marijuana use among 12^th graders is at its highest rate since the early 1980’s, reflecting a trend in an upsurge of marijuana use at younger grade levels and a deterioration of attitudes regarding the risks involved in smoking marijuana (Johnston et al., 2010). The US Office of National Drug Control Policy (2011) estimates that the cost of drug use in terms of lost productivity, health care, and others costs is $6,124 per second and totals $193 billion per year.

Rates of use in the UK present a longitudinal different pattern (Fuller, 2011). Fewer students in the 11 to 15 age range reported ever having used tobacco in 2010 (27%) than reported doing so in 1982 (53%). However, the prevalence of alcohol use has increased, with 39% reporting having had alcohol some time in their life in 2003 and 55% having done so in 2010. Among young adults in the 16 to 24 age group, the use of marijuana has declined steadily since 1998, from a peak of 27% having used marijuana in the past year to 17% reporting use in 2009/2010 (Head and Lodwick, 2011). Specific drugs may trend up or down. Whatever the situation, primary prevention has been consistently heralded as among the most cost-beneficial strategies that can be instituted (Miller and Hendrie, 2009).

Numerous meta-analyses and reviews elucidate the value of school-based substance use prevention programs in reducing alcohol, tobacco, and drug use as well as other conduct problems (Cuijpers, 2003; Dobbins et al., 2008; Durlak et al., 2010; Gottfredson and Wilson, 2003; Wilson et al., 2001). Programs that are built upon social competency approaches, cognitive behavioral theories, and social influence models, and utilize interactive delivery mechanisms that target mediating variables associated with risk for substance use are particularly effective at preventing problem behaviors (Botvin and Griffin, 2007; Cuijpers, 2003; Ennett et al., 2003; Hansen et al., 2009; Hansen and McNeal, 1996; Wilson et al., 2001).

In a review of 30 school-based problem behavior prevention programs, including meta-analysis, mediating variable interventions, and program feature comparisons (e.g., boosters, community components) Cuijpers (2003) found that effective programs included any of the following characteristics: presence of an interactive delivery style, employment of a social influence model, inclusion of community intervention and peer leader components, and a focus on mediating variables that include normative beliefs, commitments and intentions to not use substances, and life skills. In a systematic review of 29 randomized controlled trials of school-based substance use prevention programs (Faggiano et al., 2008), similarly found that skills-based programs were successful in reducing marijuana use and improving decision-making, self-esteem, and resistance skills compared to usual curricula. More recently, Durlak and his colleagues (2010) conducted a meta-analysis of 75 studies from 69 different after-school programs and found that children and adolescents who participated in these programs experienced significant gains in achievement and personal, social, and other academic benefits.

As prevention science and theory have evolved over the past three decades, it has become apparent that the success of research-based programs is contingent upon implementation fidelity, or the extent to which program delivery is congruent with the intentions of the people who developed the programs (Dusenbury et al., 2003). Programs that are delivered with high fidelity are typically associated with improved student outcomes (Burke et al., 2011; Durlak and DuPre, 2008; Kalafat et al., 2007). For example, in a study of a classroom management intervention conducted by Burke and colleagues (2011), high fidelity of implementation was associated with greater academic engagement and fewer student suspensions. High fidelity has also been associated with significant improvements in theoretical knowledge (Skara et al., 2005), decision-making skills, perceived negative consequences of substance use, and expectancies regarding alcohol use (Sloboda et al., 2009), and desirable changes in program mediators and intentions to use substances (Rohrbach et al., 2010), among others.

Unfortunately, once disseminated, research-based programs that produce meaningful effects in efficacy trials are often delivered with poor fidelity (Ennett et al., 2011), which lessens the potential of these programs to be effective. Often fidelity is thought of in two senses, fully covering prescribed program content and delivering content using prescribed methods. Ennett and her colleagues (2003) found that while 62% of providers taught prescribed content, only 17% delivered the programs using prescribed methods and only 14% implemented both prescribed content and prescribed methods of delivery when implementing a program. These results are consistent with other research (Sterling et al., 2009) in which 37% of implementers reported modifying the length or content of a smoking cessation program for reasons that included time constraints, multiple risk factors of teens that impacted tobacco use, and low participation enrollment.

In early discussions, adaptation was often thought of as a counter-balance to fidelity. That is, adaptations, which are deliberate or accidental changes made to a program were thought to detract from fidelity. However, more recently (Backer, 2001; Dusenbury, Brannigan, Hansen et al., 2005), the degree to which teachers introduce adaptations into their program delivery has been considered a separate feature. Indeed, Dusenbury and her colleagues (2005) observed that there were teachers who had both high fidelity (in that they taught a program so that content was covered and methods adhered to) and high levels of adaptation. Some researchers (e.g., Barrera et al., 2011; Castro et al., 2010; Fagan et al., 2008) have argued that changes in the manner with which a program is delivered may require adaptation in order to meet a specific group’s needs. Indeed, there are increasing numbers of research reports that suggest program adaptations have the potential to helpful and should not be considered to be unequivocally harmful (Card et al., 2011; Castro et al., 2004; Kumpfer et al., 2008). Some researchers are willing to go further. As Morrison and her colleagues (2009) conclude, slavish fidelity may result in an intervention that is faithful in form, but fails to achieve intended goals because the program’s structure does not reach the audience to which it is delivered. Others concur that implementation should balance fidelity and adaptation (Ozer et al., 2010; Yang et al., 2004). Moreover, there is a need to find balance as recent research has demonstrated that all teachers adapt prevention programs (Dusenbury et al., 2005; Hill et al., 2007; Larsen and Samdal, 2007).

Some adaptations are undoubtedly the result of culture, geography, the nature of resources available to the implementation, and other justifiable circumstances require change (Bowen et al., 2010; Morrison et al., 2009; Shen et al., 2008). Other adaptations may reflect the underlying philosophy of a program and may indeed maximize effectiveness (Lee et al., 2008; Shen et al., 2008). It is likely that adaptations will reflect the training, personality, or other characteristics of the implementer. However, it is also likely that adaptations may reflect a lack of understanding or a lack of specific skills called for in implementing a program. For example, Blakely and his colleagues (1987) found that high-fidelity adapters had more effective implementation than did low-fidelity adapters. Thus, it may not be whether an adaptation has occurred that is important, but how that adaptation aligns with a program’s goals. There remains a dearth of research on how programs are actually adapted, why they are adapted and how adaptations affect program outcomes (Rohrbach et al., 2006). As of yet, this issue has not been resolved and is the focus of the current research.

In this study, we attempt to answer four questions. First, how do teachers vary in their styles of adapting a program as it is taught? Second, to what extent is an observed style of adaptation consistent across time? Third, how do styles of adaptation relate to other qualities that define fidelity of implementation? Fourth, what are the implications of these findings for promoting effective dissemination of prevention programs? The study we report is correlational in nature; no experimental manipulations were instituted.

METHODS

Subjects and Setting

We recruited 43 teachers to deliver All Stars Core to sixth or seventh grade students in Chicago Public Schools. All Stars focuses on preventing the onset of gateway drugs such as alcohol, tobacco, marijuana, and inhalants, as well as aggression and premature sexual activity. All Stars includes activities designed to change mediators of behavior to prevent substance use and other risky behaviors. The program focuses on the following mediators: (1) increasing perceived incongruence between drug use and other high-risk behaviors and one's desired lifestyle (lifestyle incongruence), (2) correcting erroneous and exaggerated normative beliefs about drug use and other high-risk behaviors (normative beliefs), (3) building strong personal commitments to avoid drug use and other high-risk behaviors (commitment), (4) increasing attachment between youth and the school or a prosocial community group to which they belong (bonding), and (5) increasing positive parental attention, including increased communication, monitoring and supervision, and clarified standards for behavior.

Schools and teachers were asked to teach at least one section of All Stars per year for three successive years, 2002-2005. Teachers video recorded all sessions. Among the 23 teachers who taught all three years, nine were selected who represented a diverse set of outcomes. We identified three teachers whose students consistently scored in the top tertile in terms of changing targeted mediators, three teachers whose classes scored in the middle tertile, and three teachers who were, based on class outcomes, ineffective at changing mediators. The potential to affect changes in mediators was thought to be a key indicator of overall quality of program delivery because of their key roles as mediators in program effectiveness (Hansen and McNeal, 1996; McNeal, Hansen, Harrington and Giles, 2004). All teachers were assigned ID numbers. Top teachers were 49a, 02a and 36a, middle of the road teachers were 7a, 32a and 18a and those ineffective at changing program mediators were 33a, 45a and 13a.

Students were also included in this research. Analyses were based upon 597 students participating in All Stars. On average, each class had 22.1 students (S.D. = 4.3). On average 48.6% (S.D. = 22.9%) of classes were female. Three classes were composed entirely of female students and one class was composed entirely of male students. The average student was 12.4 years of age at pretest (S.D. = 0.3). Overall, 33.5% of students were Hispanic, 29.1% of students were Black, 20.1% of students were White, and the remainder were Asian (3.0%), or Native American (2.0%).

Procedures

A manual to guide the coding of adaptations was developed from a prior pilot study in which two sessions of All Stars were coded by two Ph.D. investigators.⁵ During the pilot study, each addition or change in the intervention was described using narrative text. Texts were then examined and general types of adaptations were classified. This resulted in a draft coding schema of nine categories of adaptations which was then used in the current analysis. Five categories addressed changes in methods. Four categories addressed changes in messages.

Method adaptations included changed lesson structure (how things were done), changed instructions (what students were told to do), added steps (doing something definably different than what was called for), added or altered questions (asking a new question or changing a question that altered meaning from what was prescribed), and added examples or stories (relating a personal experience or anecdote not called for by the curriculum).

The first message category included teachers’ statements that addressed norms and attitudes about risky behaviors, such as using drugs, having sex, delinquency, or violence that were otherwise not called for in the curriculum. The second category applied when a teacher added a message that emphasized the importance of some aspect of the program. The third message category applied when teachers introduced a new skill for students to develop. The fourth message category, new concepts, applied when a teacher introduced information about a topic that was not intended to be discussed.

Valence coding was based on earlier work (Dusenbury et al., 2005). The goals of each session and objectives of each relevant activity in the curriculum manual served as the basis for making these judgments. That is, raters made valence ratings in response to the question, “What effect will this adaptation have on the goals of this session or the objectives of this activity?” when they made judgments. For each adaptation, observers gave a valence score. These scores were intended to reflect the potential of that adaptation to enhance or detract from program success. Positive scores (+1, +2) reflected an anticipated increased effectiveness whereas negative scores (−1, −2) reflected an assessment that the adaptation would detract from program success. Adaptations that were not judged to have an impact were scored as zero to indicate that there was no impact on outcomes expected.

Additional data were also collected about each session, including adherence, percent of the program that was skipped, the degree to which objectives were achieved, dosage, student engagement, quality of teaching, teacher understanding of the program, potential for impact on targeted mediators and behaviors, and classroom management. Specifically, in reference to components of the program that were skipped, we are aware that some researchers consider omissions to be a form of adaptation (Hill et al., 2007). In the early discussion about this among the investigators, including results of a pilot study, two facts underscored our decision to examine only additions and changes. (1) Adaptations as additions or changes are substantive and involve the teacher doing something that is observable and can therefore be easily coded. For example, it was easy for observers to note the time when an addition or change was initiated. (2) Omissions were often less easily documented. In other words, whether a teacher completed or failed to complete a given step within an activity was often open to debate. "How much needed to be done to say something was completed?" became a question that was extremely difficult to answer. For example, the time the omission occurred was nearly impossible to code. We therefore ended up coding omissions based on an estimate of the overall amount of material skipped and considered skips to be substantively and theoretically different from additions and changes. Therefore, when teachers skipped activities or steps within activities, these were not coded as adaptations. In the end, the reason for focusing primarily on additions and changes in this paper is that we think it they are more theoretically interesting. However, it should be noted that on the form used to code adaptation details, we also allowed and encouraged raters to code the skips they observed.

Four raters were employed by the project to complete analyses. Three held masters degrees and one held a bachelors degree. Raters were blind to the ranking of teachers’ mediating variable and behavioral outcomes. Raters were trained by three Ph.D. level investigators. The training process involved having all seven participants independently rate six video recordings from session 8 (a middle session) of the program. Each observation was then processed as a group with agreements and disagreements noted and used as the basis of discussion and instruction. Discussion resulted in a final consensus rating for each video. Prior to coding each of the remaining 11 sessions in All Stars, the group of four trainers and at least one of the Ph.D.-level investigators first coded and discussed three videos from that session. Thus, overall, 39 videos were used to train coders.

The remaining 16 videos in Session 8 along with the other 251 videos from the other sessions were rated by pairs of coders. Pairs of raters had 72.8% agreement in rating the occurrence of structural adaptations, 81.4% agreement for rating changes in instructions, 80.6% agreement when rating added steps, 81.0% for when teachers added questions, 80.3% for when teachers added examples and stories, 94.9% for messages about norms, 76.7% for motivational messages, and 77.4% agreement for when new concepts were introduced. There was only one case in which a new skill was added. Overall agreement between pairs for the positive, neutral, and negative valences that were assigned for method adaptations was 70.3% and agreement for the valence of message adaptations was 72.0%.

In the end all 306 video recordings were independently coded by at least two and in some cases as many as seven coders. A consensus rating was agreed upon between both coders and used for the final fidelity coding for each session. There were occasions when initial ratings differed, either an adaptation was not observed by one of the raters or both raters coded the adaptation but disagreed about its classification or valence rating. In such cases, coders discussed the adaptation, referring to the coding manual for definitions, until they came to consensus. On occasion, disagreements were resolved with a discussion with one or more Ph.D. researchers. Only consensus ratings for were used for outcome analysis.

Student and Classroom Measures

Students were administered pretests before participating in All Stars and posttests upon completion of the program. Data were collected as part of a prior research study (Ringwalt, Pankratz, Hansen et al., 2009). Students answered yes/no questions about recent (past 30-day) alcohol use, drunkenness, smoking cigarettes, using smokeless tobacco, marijuana use, and inhalant use. An index of continued non-use was created based on the proportion of non-users at pretest who remained non-users at posttest. To create this index, students who had used any substance at pretest were removed from the data that were used, leaving only pretest non-users. The proportion of these students who then remained non-users constituted the percent of the class that remained non-users. We adopted this approach to creating classroom-level data because the primary goal of All Stars is to prevent non-users from experimenting with substances and becoming users. As a consequence, cessation by pretest users is ignored.

RESULTS

We examined teachers’ styles of adaptation by first examining the types of adaptations teachers made and then examining raters’ valence scores.

Types of Adaptations

Our first method of exploring the different styles teachers had for making adaptations was to examine the types of adaptations they made. On average, teachers made 5.84 (S.D. = 2.34) adaptations per session. Most adaptations were methodological in nature (Mean = 4.89; S.D. = 1.81); fewer adaptations were made that included message content (Mean = 0.95; S.D. = 0.66). The percent of each category of adaptation attributable to each teacher was calculated. Table 1 presents the adaptations made per category by each of the nine teachers. This allows the relative emphasis of each teacher’s style of making adaptations in terms of adaptation type to be revealed.

Table 1.

Adaptations by Category Attributable to Each of Nine Teachers (Percentages are Calculated within Each Row)

	Teacher*									Total per Category
Category of Adaptation	13a	02a	32a	49a	36a	33b	18a	07a	45a	N	Percent
Method: Change in Structure	12.1%	12.6%	12.1%	14.2%	11.6%	12.6%	11.1%	8.2%	5.4%	388	100%
Change in Instructions	4.8%	16.3%	15.4%	10.6%	15.4%	17.3%	11.5%	6.7%	1.9%	104	100%
Added Steps	9.6%	16.9%	23.3%	7.6%	7.9%	11.2%	12.6%	7.0%	3.9%	356	100%
Added Questions	18.4%	14.6%	7.9%	13.6%	13.6%	11.1%	8.4%	7.1%	5.4%	560	100%
Added Examples and Stories	36.5%	17.4%	5.2%	7.0%	8.7%	9.6%	1.7%	3.5%	10.4%	115	100%
Message: Added Normative Message	62.5%	4.2%	0.0%	12.5%	8.3%	0.0%	12.5%	0.0%	0.0%	24	100%
Added Motivational Message	10.1%	10.8%	27.0%	15.5%	17.6%	4.1%	6.8%	4.1%	4.1%	148	100%
Introduced New Concept	26.7%	15.8%	7.5%	20.8%	5.8%	11.7%	0.8%	2.5%	8.3%	120	100%
Total Number of Adaptations per Teacher	293	264	245	228	210	200	163	117	95	1815	100%
Percent of Adaptations per Teacher	16%	15%	13%	13%	12%	11%	9%	6%	5%

^*Teachers are ordered by the cumulative number of adaptations made.

Two examples will serve to elaborate on how these percentages can be interpreted for individual teachers. Teacher 13a made relatively more adaptations than all other teachers (16% of all observed adaptations). His style was characterized by an unusually large number of added examples and uncalled-for normative messages. He accounted for over one-third of all observed example adaptations and made nearly two-thirds of all uncalled-for normative messages. On the other hand, he made relatively few changes in the instructions he gave to students.

In contrast, teachers 45a and 07a made relatively few adaptations overall, each accounting for only 5% and 6% of observed changes, respectively. Neither provided any uncalled-for normative messages and made practically no changes in instructions to students (1.9% and 6.7%, respectively). Of all teachers, these two teachers made the fewest changes in program structure (5.4% and 8.2%, respectively), added the fewest steps (3.9% and 7.0%, respectively), and provided the fewest motivational messages (4.1% each).

Adaptation Quality

A second way to calculate style of adaptation is to examine valences or quality of adaptations associated with each adaptation type that was observed. Table 2 presents the average valence for each adaptation category by teacher. The table also presents the total number of adaptations and the average overall adaptation valence for each teacher. This allows the relative quality of each teacher’s adaptations to be revealed. It should be noted that when a teacher made no adaptations in a given category, valence scores are not calculated.

Table 2.

Mean Valences Associated with Adaptations Attributable to Each of Nine Teachers

	Teacher*
Category of Valence	13a	02a	32a	49a	36a	33b	18a	07a	45a	N	Overall Mean
Method: Change in Structure	−0.21	−0.39	0.21	−0.11	0.13	−0.51	0.14	−0.03	−0.29	388	−0.12
Change in Instructions	−0.80	−0.65	0.25	−0.73	0.44	−1.17	0.83	0.29	−1.00	104	−0.22
Added Steps	0.06	−0.02	0.80	−0.22	0.61	−0.23	0.47	0.60	−0.29	356	0.28
Added Questions	−0.06	0.15	0.66	0.05	0.46	−0.07	0.43	0.20	0.03	560	0.18
Added Examples and Stories	0.07	−0.05	0.83	−0.13	0.20	−0.18	0.50	0.50	−0.17	115	0.06
Message: Added Normative Message	−0.47	1.00		−1.00	1.00		−0.33			24	−0.33
Added Motivational Message	0.33	0.13	1.05	0.39	0.85	0.17	0.80	1.00	−0.17	148	0.63
Introduced New Concept	−0.03	−0.16	0.33	−0.44	0.14	−0.29		−0.33	−0.60	120	−0.18
Total Number of Adaptations per Teacher	293	264	244	228	210	200	163	117	95	1815	0.11
Average Valence of Adaptations	−0.06	−0.08	0.65	−0.10	0.44	−0.32	0.40	0.26	−0.21

^*Teachers are ordered by the cumulative number of adaptations made.

Average valences for teachers ranged from −0.32 to +0.65 and the valence ratings for individual categories among teachers ranged from −1.17 to +1.05. Four teachers (07a, 18a, 32a, and 36a) had overall positive valence ratings. Teachers 02a, 13a, 33b, 45a, and 49a had generally negative valence ratings. These findings were confirmed with a 2-step cluster analysis whereby teachers with negative valence scores were consistently grouped together into one cluster and teachers with positive valence scores were consistently grouped together into a second cluster. However, when adaptation type was examined, these teachers were not grouped together in the cluster analyses. For example, 07a, 18a, 32a and 36a were all placed into different cluster solutions. Thus, there appear to be two ways of examining style of adaptation, one based on the types of adaptations made and another based on adaptation quality, or observers ratings of valence (positive and negative).

Teachers’ valence scores varied across categories and observations. Valence was uncorrelated (r = .02, n.s.) with number of adaptations made. Thus, a teacher’s quality of adaptation appears to be independent of the type and frequency of adaptation they made.

Adaptation and Prevention

Each type of adaptation and valences of adaptations was examined as predictors of preventing drug use onset. As can be seen in Table 3, raters’ assessments for three adaptation categories yielded significant correlations. Two valence score ratings were positively correlated with program outcomes. When structure and instructions were enhanced because of what teachers did, they improved drug use prevention effects. When teachers added stories and examples, this was related to a decreased proportion of students who remained non-drug users. None of the message adaptations, either when examined for frequency of occurrence or valence, were related to drug use outcomes. However, while not significant, valences associated with added steps, added questions, and added motivational messages generally trended in a direction to indicate that when these were positive, student outcomes fared better.

Table 3.

Correlations between the Relative Frequency of Adaptation, Magnitude of Valence and the Percent of Students in a Classroom Who Remained Non-Users of Any Drug

	Frequency			Valence
Category	r	p	N	r	p	N*
Method:
Change in Structure	0.201	0.315	27	0.433	0.024	27
Change in Instructions	0.032	0.876	27	0.507	0.013	23
Added Steps	−0.216	0.279	27	0.340	0.082	27
Added Questions	−0.235	0.237	27	0.321	0.103	27
Added Examples and Stories	−0.458	0.016	27	−0.043	0.843	24
Message:
Added Normative Message	−0.122	0.545	27	−0.012	0.973	11
Added Motivational Message	−0.261	0.188	27	0.321	0.117	25
Introduced New Concept	−0.103	0.608	27	0.137	0.522	24

^*Valence Ns vary because teachers did not make each kind of adaptation each year.

Both total of number of adaptations made and rated quality of adaptation (valence scores) were used to examine a teacher’s style of adaptation. Each teacher’s three classes were considered separately in these analysis (total N = 27) because outcome data were class-specific. Total number of adaptations, number of method adaptations, and number of message adaptations per classroom were calculated. Teachers were classified as high adapters if the made more adaptations than the median and low adapters if they made fewer adaptations. Teachers were classified as positive adapters or negative adapters based on their average valence scores (total, method, and message) for each of their classes. Table 4 displays classification results and average rates of continued non-use of all six substances use categories for associated classes. For the model in which total numbers of adaptations were considered there was an interaction between number of adaptations and valence of adaptations (F(1,23) = 11.79; p = .002), with low frequency adapters whose adaptations were positive having significantly more of their students remain non-users than all other groups. Low frequency adapters generally had a higher proportion of students who remained non-users (F(1,23) = 4.73; p = .04) than was the case among high frequency adapters.

Table 4.

Percent of Students Remaining as Non-Users for Four Groups Based on Median Split for Frequency of Making Adaptations and Positive versus Negative Average Valences

	Total-Average		Method		Message
Group	N per Group	Non- Users	N per Group	Non- Users	N per Group	Non- Users
Frequent-Negative Adapting	7	62.0%	8	60.1%	5	61.3%
Frequent-Positive Adapting	6	52.4%	5	56.9%	8	55.3%
Infrequent-Negative Adapting	5	55.9%	6	59.1%	6	61.6%
Infrequent-Positive Adapting	9	79.7%	8	78.2%	8	78.4%

All tests of significance examining differences between high- and low-frequency method adapters who had positive and negative valences failed to reach significance at the .05 level. However, when message adaptations were considered, both the main effect of frequency of adaptation (F(1,23) = 4.50; p = .045) and the interaction of frequency of adaptation and valence of adaptation (F(1,23) = 4.30; p = .049) were significant. In all cases, those who made few adaptations whose adaptations were generally positive had better results than other teachers.

Consistency of Adaptation Style

Do teachers persist in making the same kinds of adaptations and make adaptations of similar quality across time? There were more adaptations observed in years 2 and 3 of the project than in year 1 (year 1 =563, year 2 = 627, year 3 = 625). With only one minor exception, the relative frequency with which categories of adaptations were made was consistent for all three years of the project.

The degree to which teachers persist in making the same kinds of adaptations year-after-year speaks to a consistency of style of adapting. Our approach to examining this issue was to create proximity matrices of Euclidean distances (SPSS 20). Euclidean distance is the square root of the sums of squares of the absolute differences between each teacher’s profile across all years. Each teacher’s yearly adaptation profile (the percent of a category of adaptation each teacher’s adaptations represented) was used to calculate entries in the adaptation style matrix. For the valence matrix, only variables for which all teachers made adaptations during all years were included. The valence matrix is therefore based only on three valences; structural changes, added steps, and added questions, If a teacher were perfectly consistent from one year to the next, his or her Euclidean distance would be zero.

Excluding a teacher’s own averages, the overall average Euclidean distance between adaptation styles based on the frequency with which adaptations occurred was 0.404. This represents the similarity of style between each teacher and all other teachers. The average Euclidean distance for the nine teachers (the proximity of each teacher with him or herself) was 0.259 (see Table 5). In other words, teachers’ year-to-year adaptations were more similar to each of their other adaptations than they were to all others by an average factor of 0.144. As seen in Table 5, there were differences among teachers in how consistent they were. No teacher was perfectly consistent between any two years; however, most teachers were generally consistent across time in the types of adaptations they made. Indeed, all teachers but one (36a) were more consistent year-to-year with themselves than with other teachers and six teachers (18a, 07a, 32a, 02a, 45a, and 33b) had proximity values that suggested they deviated very little from year-to-year in the kinds of adaptations they made.

Table 5.

Consistency of Types of Adaptations Across Time (Smaller Proximity Values Indicate Greater Consistency)

Teacher*	Average Proximity across Years	Standard Deviation	Difference from the Overall Average (0.404)
18a	0.109	0.007	−0.295
07a	0.124	0.024	−0.280
32a	0.162	0.011	−0.241
02a	0.173	0.025	−0.231
45a	0.180	0.087	−0.224
33b	0.199	0.001	−0.205
49a	0.291	0.077	−0.112
13a	0.378	0.169	−0.026
36a	0.718	0.339	0.314
Average	0.259	0.082	−0.144

^*Teachers are ordered by average proximity score.

Teachers’ valence ratings were also consistent from year-to-year (see Table 6). Teachers who made poor adaptations (adaptations with negative valences) did so consistently. Teachers who made good adaptations (those judged to be beneficial to the program) also did so consistently. Because the metrics presented in Tables 5 and 6 are not identical, a direct comparison of consistency in type and quality of adaptation is not possible. Nonetheless, the relative magnitude of difference between teachers’ year-to-year consistency versus own-to-others’ consistency is approximately the same.

Table 6.

Consistency of Valence Ratings Across Time (Smaller Proximity Values Indicate Greater Consistency)

Teacher*	Average Proximity across Years	Standard Deviation	Difference from the Overall Average (0.763)
13a	0.274	0.071	−0.489
33b	0.282	0.115	−0.481
49a	0.282	0.143	−0.481
07a	0.317	0.095	−0.446
18a	0.397	0.026	−0.367
02a	0.433	0.096	−0.331
32a	0.444	0.147	−0.319
45a	0.515	0.169	−0.248
36a	0.624	0.230	−0.139
Average	0.396	0.121	−0.367

^*Teachers are ordered by average proximity score.

Adaptation and Fidelity

Our final analysis was to examine the relative contribution of valence and frequency of adaptation as predictors of other ratings of fidelity, such as adherence, the quality with which student-centered objectives were achieved, percent of program content that was skipped, student engagement, the overall judged quality of teaching, the judged degree of teacher understanding, the judged potential for effectiveness, and ratings of classroom management. Table 7 presents data for these analyses. With the exception of the test examining classroom management, teachers who made positive adaptations scored significantly better on their fidelity scores than teachers who were judged to have made negative adaptations. For example, positive adapters average score for Overall Quality of Teaching (scaled 1–5) was 3.79 (S.D. =0.85) versus 2.26 (S.D. = 0.60) for negative adapters. Similarly, Degree of Teacher Understanding (scaled 1–5) was 4.14 (S.D. = 0.89) for positive adapters and was 2.53 (S.D. = 0.67) for negative adapters.

Table 7.

Analysis of Variance Results (F_1,23) for Eight Measures of Quality of Delivery

	Positive vs. Negative Adapters	Frequent vs. Infrequent Adapters	Valence × Adaptation
Adherence (Activities Attempted)	9.60‡	1.19	0.14
Adherence (Percent Skipped)	10.42‡	4.59*	0.30
Student-Centered Objectives	16.09†	0.23	0.16
Student Engagement	9.45‡	0.14	0.41
Overall Quality of Teaching	25.44†	0.74	1.49
Degree of Teacher Understanding	26.82†	0.83	3.29*
Judged Potential for Effectiveness	19.16†	0.88	1.13
Classroom Management	2.32	0.40	3.08*

^*p < .10

^‡p < .005

^†p < .001

There was only one marginally significant main effect related to fidelity for number of adaptations made. This related to the percent of steps skipped, with teachers who made relatively fewer adaptations skipping fewer steps than those who made more adaptations. There were also two marginally significant valence × frequency of adaptation interactions; one for raters’ assessments of the degree to teachers managed their classroom well. In the case of teacher understanding, when teachers made negative adaptations, it made no difference if they were also low frequency or high frequency adapters (mean of ratings = 2.37 [S.D. = 0.63], and 2.64 [S.D. = 0.72], respectively). On the other hand, low frequency positive adapters scored much better on how raters’ judged their understanding (mean of ratings = 4.47, S.D. = 0.49) than if the positive adapters were also found to be high frequency adapters (mean of ratings = 3.65, S.D. = 1.16). Interestingly, low frequency negative adapters had the best classroom management scores (mean of ratings = 4.19, S.D. = 0.32), whereas high frequency negative adapters, low frequency positive adapters, and high frequency positive adapters were all judged to be generally less adept at managing their classrooms (mean ratings of 3.75, 3.58, and 3.79 and standard deviations of 0.51, 0.40, and 0.60 for these respective groups).

DISCUSSION

Generally speaking, both how frequently teachers adapt and the quality of the adaptations they make have profound effects on program effectiveness. While this non-experimental study is limited to description, rather than inference, the overall, the implications of this research on health education should be to encourage teachers to adapt sparingly. When teachers do adapt, they should be encouraged to only make adaptations that clearly improve the potential of intervention to achieve its goals and objectives.

Our findings provide answers to the first and second questions posed at the outset of this article: “How do teachers adapt?” and “How consistently do teachers adapt over time?” All teachers adapt programs as they teach. Results of our analyses demonstrate that teachers are highly individualistic in how they adapt a program. Some teachers are prone to adapt various methodological features — how activities are delivered, instructions they give to students, what components they add, what questions are asked to prompt discussion, and what examples and stories they add. At the same time, there are teachers whose adaptations include features related to the messages that are delivered. We identified three primary sets of message adaptations — messages that prescribe or proscribe various norms and attitudes, added messages that are intended to motivate students, and messages that promote concepts not otherwise intended by the intervention.

Even though this research is exploratory in nature and includes a limited number of teachers, the varieties of combinations of adaptations made from teacher-to-teacher were remarkably varied. We were able to differentiate between frequent and infrequent adapters both generally and when specific types of adaptations were considered. We also demonstrated that, in nearly all cases, adaptation type and quality is consistent across time. This finding replicates other research, where the same numbers of adaptations were implemented in the second iteration of a prevention program as the first (Ozer et. al., 2010).

The third question we asked was about the relationship between fidelity and adaptation. Our findings suggest that there is a link between fidelity and adaptation, but that this is primarily a relationship between the valence or quality of a teacher’s adaptations and several markers of fidelity, including nearly all markers of fidelity that we assessed. It is possible that, as raters made judgments about each quality that they auto-correlated each of the values. Thus, it may have been the case that there was a tendency to make ratings about such qualities as teacher understanding and quality of teaching in part as a response to the valence ratings they made. Or, conversely, they may have ascribed valences based on the overall quality of teaching, etc. This is further complicated by the fact that, having coded numerous sessions for any given teacher, raters may have had expectations of what might be observed even before they viewed some video recordings. Nonetheless, even should such influences exist, the fidelity and valence of adaptation linkage hypothesis still appears to generally be supported, in part because there was significant session-by-session variability in what was coded. For now, the notion that poor adaptations are correlated with poor fidelity continues to be a viable expectation.

This study adds to the methodology for assessing adaptations. We trained raters to make judgments about the valence of adaptations. The involved using an assessment of the impact of each adaptation on prescribed goals and objectives. We recognize that there are many subjective elements that are involved in this process. Making judgments about the valence of specific adaptations often required considerable thought and discussion. Thus, there may be additional research required to develop a less subjective system for ascribing valence to adaptations. As it currently stands, we have no way of verifying that any given valence rating had the enhancing, neutral or detracting impact we predicted. We can simply state that, in aggregate, adaptations that were judged to be positive or negative appeared to carry weight when correlated with student outcomes and with other measures of fidelity.

When the valence of adaptations was considered, there were generally two types of teachers; positive adapters and negative adapters. Those who made positive adaptations to the structure of the program and the instructions given to students had a positive effect on student behavioral outcomes in that a higher percentage of students remained non-users of drugs. In contrast, those whose adaptation quality was negative had a lower percentage of students who remained non-users. Because of the large number and wide variety of adaptations in each category, understanding the nature of positive and negative changes introduced will ultimately require additional research. However, it appeared to be generally the case that positive changes in structure and instructions simplified tasks students were asked to perform, allowing the messages inherent in the curriculum to be more clearly communicated. Negative valences generally reflected cases where teachers complicated tasks and made it difficult for the messages intended by the curriculum to be delivered. While the number and type of adaptation did not correlate with behavioral outcomes, the quality of adaptations made, reflected in the valence scores, was predictive.

Most teachers exhibited a tendency to make the same kinds of adaptations and all teachers were consistent in the valence attributed to their adaptations. In many respects, this is to be expected. People generally tend to behave in consistent ways and, as has long been known, the best predictor of future behavior is past behavior. When behavior is inconsistent, one intuitively suspects either something unstable within the person or that external pressures are being exerted. The character of classrooms may be one factor that influences how teachers adapt. Some classrooms of students may be less engaged or comprehending, prompting the teacher to make adaptations with the hope of increasing involvement or understanding. However, when teachers make adaptations, they might be expected to do so within the repertoire of approaches they already use, which would suggest that teachers may dial-up or dial-down the rate at which they make adaptations, but not change the essential character of what they do. We were not able to determine whether specific adaptations were planned in advance or in response to situational demands. It is probably the case that a teacher will make adaptations based on the style with which they are accustomed, using the repertoire of adaptation tools they are familiar with and avoiding the tools they have not previously considered.

Not all categories of adaptation yielded relationships with drug use outcomes. Only when teachers improved the structure of the intervention or add useful instruction were specific types of adaptations valuable in terms of reducing the onset of drug use. In these two cases, the opposite is also true. That is, when teachers made deleterious adaptations to the structure of the program or when they added confusing or inappropriate instructions, drug use outcomes suffered. Our assessment suggests that both positive and negative adaptations to structure and instructions were made. Each of the adaptations was recorded in narrative form. It appears that positive structural and instructional changes generally simplified the intervention and made it clearer to the students whereas negative adaptations added complexity and confusion.

One interesting finding that requires some speculation has to do with the negative relationship observed between adding examples and stories and student outcomes. One teacher (13a) accounted for most of the examples and, as a teacher who had poor outcomes, appeared to determine the negative statistical relationship that was observed. As such, it may not be that examples and stories are inherently harmful for program goals. Rather, this teacher’s examples and stories may have contributed to students’ poor outcomes. Given the number of observations made of this teacher, there was plenty of context to suggest that teacher 13a’s stories, which often dealt with personal drinking experiences and attitudes, may have significantly colored the inherent message that was embedded in what was said. Thus, it may be that this teacher’s stories may have had an exceptionally negative effect. Interestingly, the valence ratings of these stories did not reflect this strength of relationship and this interpretation is therefore not fully supported.

Our findings were specific to All Stars. It is not clear how any of these findings would generalize to other interventions. Programs vary in their structure and specificity. In comparison to other substance use prevention programs that were recently reviewed (Hansen et al., 2007), All Stars ranks relatively high in the specificity with which the curriculum manual is structured, providing detailed step-by-step instructions. Adaptation may be suppressed under such circumstances. On the other hand, for programs with more vague structures, the idea of an adaptation may be more difficult to define as all instruction might vary considerably from teacher-to-teacher and yet fit within the intended framework. Nonetheless, we expect a teacher’s style of instruction and adaptation to be definable and consistent across time. We also expect that negative outcomes will be observed more frequently among those who deliver the program in ways that fail to fulfill the program’s goals.

The fourth question we posed asked about what implications these findings might have for promoting effective dissemination of prevention programs. Research has yet to reveal the full extent to which adaptation affects program outcomes. It may be extremely difficult to conduct truly experimental studies in which either the type of adaptation or the valence of an adaptation is controlled and manipulated. However, even with the correlational evidence that is available from this study, it is clear that some adaptations may be beneficial and improve program effectiveness, whereas others may either be neutral in effect or deleterious. Extensive adaptation is expected to fundamentally change a program’s character. Teachers who are prone to adaptation, i.e., those like teachers 13a, 02a, and 49a who make extensive revisions to a program, require evaluation results to be examined carefully. Moreover, whatever the type and frequency with which adaptations are made, it is essential to assess quality. We found that the quality of structural changes and changes in instructions were most clearly associated with effects on program outcomes. However, there may a similar tendency for the quality of questions, added steps, and motivational messages to have the same effect.

A second set of implications revolves around the recruitment, training, and retention of teachers. It will be ideal to select teachers who, by their natures, understand the importance of fidelity, have demonstrated ability to stick to the script, and understand and have the teaching skills needed to make needed adaptations that can be expected to benefit the program. Formal training alone may not ensure that these qualities are present. Some form of an audition with the ability to compare observed adaptations to those of other successful and unsuccessful teachers may be one method for assessing the appropriateness of a teacher’s style. Once selected and trained, continued observation with an eye to assessing the frequency, type and quality of adaptations may be required. When teachers show evidence of frequent or poor adaptation styles, they may require further training or, in worst cases, dismissal.

Finally, there are implications for program design. Programs should be sufficiently structured to make it clear to teachers what is expected for delivery. Perhaps even more important might be warnings about what adaptations to avoid. When a program is well developed and continually updated to be as current in its approach, fewer adaptations should be needed. At the same time, it must be recognized that differing styles of delivery and varying contexts may require some adaptation. Programs currently provide little in the way of guidance for how they can be adapted. Improving communication about this will require a new line of research.