Factors Influencing Teachers’ Implementation of an Innovative Tobacco Prevention Curriculum for Multiethnic Youth: Project SPLASH

Abstract

BACKGROUND

The effectiveness of school-based tobacco use prevention programs depends on proper implementation. This study examined factors associated with teachers’ implementation of a smoking prevention curriculum in a cluster randomized trial called Project SPLASH (Smoking Prevention Launch Among Students in Hawaii).

METHODS

A process evaluation was conducted and a cross-condition comparison used to examine whether teacher characteristics, teacher training, external facilitators and barriers, teacher attitudes, and curriculum attributes were associated with the dose of teacher implementation in the intervention and control arms of the study. Data were collected from a total of 62 middle school teachers in 20 public schools in Hawaii, during the 2000-2001 and 2001-2002 school years. Sources included teacher questionnaires and interviews. Chi-square test and t test revealed that implementation dose was related to teachers’ disciplinary backgrounds and skills and student enjoyment of the curriculum.

RESULTS

Content analysis, within case, and cross-case analyses of qualitative data revealed that implementing the curriculum in a yearlong class schedule and high teacher self-efficacy supported implementation, while high perceived curriculum complexity was associated with less complete implementation.

CONCLUSIONS

The results have implications for research, school health promotion practice, and the implementation of evidence-based youth tobacco use prevention curricula.

Tobacco use prevention education beginning in middle school or earlier is critical for delaying the onset and decreasing the incidence of youth smoking, and for preventing smoking into adulthood.^1,2 Studying factors related to the implementation of school health programs, including smoking prevention, are important because difficulties in the implementation process are common.^3-6 Studies have found that in real-world settings, teachers do not always implement tobacco use and substance abuse prevention lessons according to specified guidelines. Hallfors and Godette found that only 19% of school district coordinators implemented evidence-based curricula with fidelity.⁷ Similarly, the School-Based Substance Use Prevention Program Study found that only 15% of teachers followed substance abuse prevention curricula guides very closely.⁸

Evidence-based youth tobacco use and substance abuse prevention programs use specific content, teaching strategies, and dosage to effectively influence and sustain changes in anti-use knowledge, attitudes, and behavior. Effective programs have reduced students’ use of tobacco and other substances from 20% to 75%,^9,10 with some sustaining changes in antismoking knowledge, attitudes, and behavior for up to 6 years after the end of the program.^9,11,12 Content for effective youth smoking prevention programs may include resistance skills training^9,10,12,13 and social influences.^9,13,14 When programs use effective teaching strategies that involve direct peer interaction, their effects have been shown to be stronger than programs using only effective content.^9,10,15 Evidence-based smoking and substance abuse prevention programs also involve an average intensity of 16 hours of programming^9,16 and are interactive.¹⁶

Studies of the implementation of smoking prevention programs have identified several factors related to this process: teacher training, teacher characteristics, teacher attitudes, organizational factors, and curriculum attributes. Several studies have evaluated the relation of these factors to use, fidelity, and dose of curriculum implementation.^15,17-28 However, until recently, factors related to implementing tobacco prevention programs have received less attention than factors related to a school's decision to adopt (ie, uptake, initiation, commitment, acceptance) these programs.^4,5,15 Identifying strategies to successfully implement tobacco use prevention curricula contributes to an understanding of how such programs may be translated to the real-world setting.

The purpose of this study was to identify organizational and individual factors associated with teachers’ implementation of an innovative tobacco use prevention curriculum targeting youth in Hawaii. Two research questions guided the study:

1. To what extent did teachers implement the recommended dose of the intervention and control curricula?

2. What factors were associated with teachers’ implementation of the curriculum?

METHODS

Subjects

Implementation of Project SPLASH (Smoking Prevention Launch Among Students in Hawaii) and Towards No Tobacco Use (TNT) Hawaii was assessed for process evaluation. The study sample of 62 teachers was obtained from the schools participating in the larger randomized trial. The dose and reach of both programs were evaluated. Factors associated with teachers’ implementation of the programs were also identified.

Instruments

Quantitative and qualitative data were collected from teacher and student questionnaires, teacher interviews and training evaluation questionnaires, and the Project SPLASH database (Table 1). Depending on the variables, quantitative, qualitative, or both types of measures were obtained to minimize data collection burden on teachers. The teacher questionnaires and interviews were used to inform the independent and outcome variables. These instruments were reviewed by the research team and pilot tested with 3 teachers to establish their face and content validity. Tests of internal consistency for multi-item measures yielded high Cronbach's alphas of 0.74-1.00.

Table 1.

Data Collection Procedures

Data Collection Procedures	Process Evaluation Purpose	When Collected	Sampling Procedure	Sample Size
Teacher questionnaire	Evaluate teacher characteristics, teacher attitudes, curriculum attributes, and teacher implementation	End of seventh-grade school year; end of eighth-grade school year	All teachers	62
Teacher interview	Evaluate teacher characteristics, teacher training, external facilitators and barriers, teacher attitudes, and teacher implementation	End of seventh-grade school year/beginning of eighth-grade school year; end of eighth-grade school year	All teachers	62
Teacher training evaluation questionnaire	Evaluate teacher training and teacher attitudes	Beginning of eighth-grade school year	All teachers	30
Student surveys	Evaluate reach	Beginning and end of eighth-grade school year	All students aggregated at level of teachers’ classes	4884
Student feedback interviews	Evaluate reach	End of eighth-grade school year	Subsample	725
Student homework	Triangulate with other data sources and exploratory research to evaluate student reach	End of eighth-grade school year	Subsample	Varied*
Project SPLASH database	Evaluate teacher characteristics and external facilitators and barriers	Ongoing, database maintained by the main study	All teachers	62

^*Sample size depended on the type of homework evaluated ranging from n = 20 to 2678.

Four questionnaires were developed, 1 each for seventh-grade SPLASH and TNT teachers and eighth-grade SPLASH and TNT teachers, and all teachers were given questionnaires. Three constructs were evaluated through the teacher questionnaires. Teacher attitudes and curriculum attributes were measured by 4 items. Teacher characteristics were measured by 9 items on the seventh-grade SPLASH and eighth-grade TNT questionnaires, while 8 items on the seventh-grade TNT and 10 items on the eighth-grade SPLASH questionnaires were used to measure this construct.

Teacher interviews were conducted to provide additional information not obtained from the teacher questionnaires. An interview guide with open-ended questions (24 and 28 for SPLASH across the 2 years, and 21 for TNT) guided the procedure. Most interviews were conducted in-person, while some were conducted by phone due to scheduling constraints. Selected questions from the interview guides are provided in Figure 1. (The complete interview guides are available on request from the first author.)

Figure 1.

Sample Teacher Interview Questions

Procedures

This study used process evaluation data from a randomized trial called Project SPLASH, which tested the effectiveness of a curriculum that emphasized student involvement on smoking rates in Hawaii middle school students, compared to a standard smoking prevention curriculum (Table 2). Project SPLASH, the experimental curriculum, comprised of 3 innovative components: computer lessons, a drama education program, and youth advocacy training.²⁹ The control curriculum was based on Project TNT, an evidence-based, school-based smoking prevention program.^30,31 This program was less interactive than the intervention program and primarily used social influences approaches.

Table 2.

Description of Project SPLASH and TNT Hawaii Curricula

Grade	Lesson Number	Number of Days	Title and Description
Project SPLASH: 45 minutes for each lesson
7	Computer 1	1	Behavioral Aspects of Tobacco: Smoking rates and issues
7	Computer 2	1	Immediate & Long-Term Health Consequences: Health, financial, and social consequences of tobacco use
7	Computer 3	1	Quitting Smoking: Helping Your Family, Friends, & Yourself: Benefits of quitting smoking, how to quit, and how to help others quit
7	Drama 1-5	5	Developing, Practicing, and Performing Anti-Tobacco Education Skits: Students work with drama educators to create anti-tobacco skits that are videotaped
8	Computer 4	1	Uncovering the Truth About Tobacco Ads: How tobacco companies target youth through advertising and marketing
8	Computer 5	1	Malama ka ‘Aina (Care of the Land): Impact of tobacco use on the environment
8	Advocacy 1	1	Introduction to Youth Advocacy: Historical, national, and local advocates and how to become an advocate
8	Advocacy 2	1	Media Literacy: Identifying how ads glamorize tobacco use and the truth behind those messages
8	Advocacy 3	2	Policy Change & the Legislative Process: day 1—learning about the state legislative process and tobacco control laws; day 2—participate in a mock hearing led by the drama educators
8	Advocacy 4	1	Advocating for a Change: Develop activities to voice opinions and motivate others
TNT Hawaii: 45 minutes for each lesson
7	1	1	Effective Listening & Tobacco Information: Importance of being active listeners
7	2	1	Tobacco History: History of tobacco
7	3	1	Course & Consequences of Tobacco Use: Stages of nicotine addiction and decision-making strategies
7	4	1	Self-Esteem: Practice techniques to improve self-esteem
7	5	1	Being True to Yourself & Changing Negative Thoughts (Peer Pressure): How to deal with peer pressure and changing thoughts about what may appear to be threatening situations
7	6	1	With a Little Help From Friends: Role play scenarios to say “no” to tobacco
8	7	1	Clearing the Smoke About Cigarettes: Conduct group research on current tobacco facts and information
8	8	1	Being Tobacco-Free: Addiction and how to convince others to quit smoking
8	9	1	Effective Communication: Discussion and role plays on communicating effectively
8	10	1	Assertiveness Training & Refusal Skills Practice: Discuss and practice ways to be assertive
8	11	1	Tobacco Advertising: Worksheets on replacement smokers, media, and tobacco companies

The process evaluation was based on a conceptual model with teacher implementation as the primary outcome (Figure 2). Factors proposed to influence teacher implementation were teacher characteristics, teacher training, external facilitators and barriers, teacher attitudes, and curriculum attributes.

Figure 2.

Process Evaluation Conceptual Model

According to the Theory of Reasoned Action, it was hypothesized that the influence of teacher characteristics, teacher training, and external facilitators and barriers was mediated by teacher attitudes. This theory indicates that a person's behavior is determined by factors related to knowledge, attitudes, and beliefs about the behavior.³² Variables outlined in the conceptual model addressing teacher attitudes cover the knowledge, attitudes, and beliefs related to the behavior, that is, teacher implementation.

Curriculum attributes was a construct based on Diffusion of Innovations and was hypothesized to moderate the influence of teacher characteristics, teacher training, and external facilitators and barriers on teacher implementation. According to this theory, an innovation's compatibility (fit) and complexity (difficulty of use) are also attributes that affect the speed and extent of the diffusion process. Thus, teacher implementation was proposed to be moderated by the characteristics of the curriculum, that is, teaching compatibility, classroom compatibility, and complexity.³³

The primary outcome variable was implementation dose, defined as the extent of intervention units delivered (as reported by teachers) and comprised of 2 components: quantity and completeness.³⁴ A composite teacher implementation measure was created by computing the average of the percent lessons taught and finished within class time dichotomized into high, 66.7-100%, and low, that is, 0-66.6%, of lessons implemented for SPLASH and TNT (Table 3). Other outcome variables such as reach and fidelity of teacher implementation provided additional descriptive results.

Table 3.

Dose Items and Measures

Variable	Measure	Variable Type	Program/Grade	Questionnaire Items	Response Format/Coding
Quantity: lessons taught	% Lessons taught	Dichotomous: high/low	SPLASH seventh	Indicate the number of SPLASH computer classes you taught	No. classes taught
			SPLASH eighth	For the computer lessons, indicate whether you taught both lessons	0 = no, 1 = yes
				Indicate the number of youth advocacy lessons you taught	No. lessons taught
			TNT seventh	Indicate the number of TNT classes you taught	No. classes taught
			TNT eighth	Indicate the number of TNT lessons you taught	No. lessons taught
Completeness: lessons finished	% Lessons finished	Dichotomous: high/low	SPLASH seventh	Indicate the number of classes that were not finished within class time	No. classes not finished
			SPLASH eighth	Indicate the number of lessons that were not finished within class time	No. lessons not finished
			TNT seventh	Indicate the number of classes that were not finished within class time	No. classes not finished
			TNT eighth	Indicate the number of lessons that were not finished within class time	No. lessons not finished

Independent variables were teacher characteristics, teacher training, external facilitators and barriers, teacher attitudes, and curriculum attributes. Quantitative measures for teacher characteristics, teacher training, and external facilitators and barriers were operationalized as categorical or dichotomous variables, while measures for teacher attitudes and curriculum attributes used ordinal scales. Variance, standard deviation, skewness, kurtosis, and normality were examined for each variable. Upon data screening, necessary transformations were conducted on selected variables.

Data Analysis

A cross-condition comparison was used to identify facilitators and barriers that influenced self-reported teacher implementation, that is, dose of the curricula, from the process evaluation results. Bivariate and multivariate analytical procedures were used. Quantitative analysis involved chi-square test, Fisher's exact test, and t test to determine if characteristics differed significantly between the SPLASH and the TNT teachers and to determine significant associations between the dependent and the independent variables. Logistic regression was used to determine the independent contributions of each variable. Variables selected for the logistic regression were those that showed significant results in bivariate analyses.

Qualitative analysis involved content analysis. The variables identified in this study served as general coding categories to initially guide the analysis. For example “teacher training” “SPLASH staff assistance” would be a general code. The constant comparative method was used to obtain more specific codes. Specific codes for “SPLASH staff assistance” would include “questions,” “materials,” and “visits.” A codebook was maintained, and NUDISTVivo was used to manage and analyze the qualitative data. Descriptive results were displayed using matrices based on the codes identified from the coding process. These matrices served to further summarize results and examine patterns of responses. Findings obtained from the different data sources were triangulated to determine convergence of results.

RESULTS

Response Rates

A total of 60 teachers completed either a questionnaire or an interview, with 39 (63%) completing both. The average response rate for the teacher questionnaires from both programs and grades was 86.2%. The average response rate for the teacher interviews from both programs and grades was 81.3%. Respondents were from seventh (n = 27) and eighth (n = 25) grades, and from the SPLASH (n = 34) and TNT (n = 18) study arms.

Implementation Dose

Results on teacher-reported implementation dose were analyzed to address the first research question: “To what extent did teachers implement the recommended dose of the intervention and control curricula?” Of the 56 teachers who responded to questionnaires (the source of quantitative measures on dose), 53 answered the items used to compute teacher-reported dose. Results indicated that most teachers in both programs implemented most of their lessons, with 71.4% of SPLASH and 72.2% of TNT teachers reporting high implementation of their curriculum.

Factors Associated With Implementation

Results from bivariate analyses identified variables that were significantly associated with the primary outcome variable, teacher-reported implementation dose, to answer the second research question: What factors are associated with teachers’ implementation of the curriculum? Measures of the independent variables for the bivariate analyses were combined for the 2 programs because program type was not significantly associated with implementation levels.

The level of significance to determine associations between the independent variables and teacher-reported dose was set at p ≤ .10 because of the small sample size. Sample size influences need to be taken into account when setting a significance level.³⁵ A nonsignificant result may be more likely if the number of observations is small, even if there is a large real effect.³⁶

Bivariate analyses revealed that teacher characteristics were significantly associated with teacher-reported dose (Table 4). Being physical education certified, pursuing the Hawaii state health teacher endorsement—a newly implemented state health education certification—receiving preservice health training, and using the Internet daily outside SPLASH were significantly associated with low implementation dose. Teachers pursuing the Hawaii state health teacher endorsement were more likely to have taken health courses (37.5%, p = .024) than those who were not pursuing the health endorsement, yet were associated with a lower implementation rate of lessons.

Table 4.

Independent Variables Significantly Associated With Implementation Dose

	Low (0-66.6%)			High (66.7-100%)			Chi-Square/Fisher's Exact Test
Independent Variables	n	Frequency	%	n	Frequency	%	Value	df	p
Teaching experience*
Physical education certified	15	9	60.0	38	11	28.9	4.14	1	.036
Purse Hawaii health endorsement	15	8	53.3	38	3	7.9	13.50	1	.001
Taken health courses†	7	4	57.1	15	2	13.3	4.62	1	.054
Internet experience (SPLASH only)	10			24			2.84	1	.098
Never-once/week		4	40.0		17	70.8
Daily		6	60.0		7	38.2

^*Responses were “yes/no” for each category of teaching experience, that is, 5 dichotomous variables.

^†Results reflect eighth-grade teachers.

Logistic regression results revealed very large confidence intervals. Since this multivariate analysis was unstable for meaningful interpretation, results were not used to identify factors influencing teacher-reported implementation.

An examination of the qualitative data revealed additional factors that may be related to teacher implementation. In both programs, proportionally more teachers who taught in yearlong class schedule formats and those who indicated having high self-efficacy fully implemented their lessons. In both programs, more teachers who indicated that their curriculum was complex only partially implemented their lessons. These results suggest that in addition to the factors identified from the bivariate analysis, factors obtained from the qualitative results may be associated with teacher implementation.

DISCUSSION

Teachers’ Background, Attitudes, and Skills

Results revealing that teachers who were pursuing the Hawaii state health teacher endorsement were more likely to have taken health courses than those who were not pursuing the certification but implemented less lessons than those who were not pursing a certification were unexpected. Teachers explained these results by noting that because they had to comply with the state health teaching standards to obtain their endorsement, they only implemented lessons that addressed those standards.

Findings that prior health course work and physical education certification were inversely associated with implementation are consistent with results suggesting that the degree of implementation may be related to whether teachers have specifically taught smoking and substance abuse prevention curricula. Previous experience and/or familiarity teaching the content and using the educational strategies recommended for smoking and substance abuse prevention curricula, but not general familiarity with health topics, may determine the extent that teachers implement new curricula. In essence, relevant and more specific professional training, rather than taking course work in general, may positively influence the implementation of lessons.

Qualitative findings support the conclusion that teachers who were not formally trained in health education may not have had sufficient training on effective health education pedagogy to prevent students’ tobacco use. The issues teachers discussed relating to low self-efficacy suggested that those who may not have training in health education may not be familiar with novel teaching strategies found in smoking prevention education. A SPLASH teacher explained, “I left out the self-esteem part at the beginning because I couldn't make the connection between the two [smoking and self-esteem].”

Daily Internet usage was also significantly associated with low implementation dose, another counterintuitive finding. Examining classroom teachers’ computer experience in relation to health education has not been previously studied. Further exploration regarding teachers’ computer experience in relation to technology-based smoking prevention curricula is needed since technology-based health education continues to be developed.

Teacher Training

Previous studies found that having received training on a smoking prevention or health education curriculum was associated with implementation dose.^6,19,25,37 This study evaluated type of training, and not whether training was received, to investigate whether training type was related to self-reported implementation dose. Workshop, individual, or no training was analyzed in relation to implementation dose, and none of these training types was found to be associated with dose.

The different delivery formats used to train teachers on implementing SPLASH and TNT may not have been distinct enough to influence implementation dose, despite previous findings that in-service training is strongly associated with dose. These studies found that “extensive”³⁸ training strategies to teach a curriculum, for example, developing an implementation plan³⁷ or practicing teaching methods,⁶ may be critical in influencing teachers’ implementation of the curriculum. Indeed, many teachers who attended workshops or received individual trainings to teach SPLASH or TNT commented that they did not find their training helpful because it did not provide additional assistance than if they reviewed the curriculum on their own.

Class Schedule Format and Curriculum Attributes

Factors external to teachers, for example, time limitations, were also found to be associated with implementation. The qualitative results on class schedule format and implementation levels suggested that a shorter class period duration or school term, that is, semester or quarter, instead of yearlong was a time barrier toward full curriculum implementation. A SPLASH teacher explained, “. . . but also it was time . . . because I only have these kids for a semester. If health was a year course, I probably could've finished it more completely.”

Curriculum complexity and scheduling difficulties were other factors that may have impeded implementation of both programs. A SPLASH teacher described implementation difficulties as follows:

I like the plan of this, but there's so many things that have to be coordinated. You gotta get the survey coordinated. Then you gotta get the computer lab time where they're open. Then you gotta coordinate it with when Bill's [drama artist, not actual name] going to be free or somebody else is going to be free to lead them [students], and then when somebody can tape it. It all has to be pretty much drop anything else you're doing and stick this in there.

TNT teachers also thought that the lesson plans and instructions to carry out activities were complex.

The only thing that was difficult was that it was really long. Six lessons was not six days.

Study Strengths and Limitations

This study compared the extent to which the content and teaching strategies of 2 programs were implemented. Evaluating the implementation of a smoking prevention program in relation to both study conditions revealed what teachers tend to “naturally” implement, regardless of their lesson plans, and also helped explain study outcomes.

The qualitative results provided insight into the implementation of smoking prevention curricula and how particular factors may influence implementation. This study's internal validity and reliability were also strengthened through the use of multiple data sources and analytic techniques that indicated an overall convergence of findings across data sources.

This study also revealed additional factors, such as teaching background and professional skills, related to teacher implementation of smoking prevention curricula not found in previous studies. Findings from this study may provide additional criteria for quality teacher implementation of evidence-based smoking prevention and substance abuse prevention programs.

The methods to ensure the development of valid and reliable measures of the process evaluation outcomes, especially fidelity, should be given critical focus in future process evaluation studies. The extent to which teachers maintained fidelity in teaching their curricula was not evaluated in this study. Defining “fidelity,” or what constitutes quality of delivery and how to measure this concept, has been challenging in process evaluations.^4,34 When operationalizing process measures, careful attention should be given to assure that the highest possible degree of validity and reliability is obtained for such measures.

The study's lack of power (n = 60) limited the use of the multivariate analysis. Only results from the bivariate analysis between the independent variables and the implementation dose were used, limiting this study's ability to identify directionality and to control for confounding variables.

Conclusions—Implications for Research and Practice

Smoking prevention curricula need to be developed with practical considerations of teachers’ time limitations to teach such programs, given their teaching expertise, curriculum requirements, and length of school terms. Class schedule format needs to be considered keeping in mind that teachers on a yearlong class schedule would be more likely to completely implement lessons. Effective smoking and substance abuse prevention curricula require a specific “dose” involving number and duration of lessons to be effective.¹⁶ Additionally, schools may also offer health as a yearlong subject to allow more health promotion topics to be fully implemented.

Schools should play a role in supporting teachers’ implementation of smoking prevention programs by ensuring that teachers have the appropriate background. Schools in Hawaii should be aware that smoking prevention education in their schools may not be fully implemented since most health teachers in Hawaii were found to have a physical education background and not health certification. Curriculum complexity was also found to relate to partial implementation. Because effective smoking and substance abuse curricula employ interactive strategies, they are often considered complex and are therefore less likely to be implemented.^18,23,27

The finding that teachers with high self-efficacy were more likely to fully implement their curriculum confirms the importance of increasing teachers’ confidence in their ability to fully implement a smoking prevention curriculum. If teachers do not have the appropriate background to proficiently implement smoking prevention curricula, schools should support teacher in-service training to implement such programs by allowing paid time off to receive adequate training. Training that involves informing teachers of the critical strategies of an evidence-based curriculum and increasing their ability to implement such strategies, as found in previous studies, may encourage teachers to more likely implement those innovative, yet complex components.^20,25,39

Further research on the extent teachers need to adhere to smoking prevention curricula guidelines without compromising their effectiveness should be conducted. From a research standpoint, adherence to curriculum protocols is key to achieving the intended effects of effective smoking prevention curricula. In practice, when teachers modify curricula, they may be attempting to address local, cultural, teaching, and/or student needs. If the critical elements of a curriculum are identified, teacher modifications may be encouraged as long as the key elements of the program are delivered.

Program developers should systematically consult with the teachers in their research to develop and identify effective smoking and substance abuse prevention and other health promotion curricula. Issues that exist in the school setting are sometimes in conflict with the critical elements of smoking prevention programs. For example, effective tobacco use prevention curricula require a particular dose of lessons, and their full implementation may be more feasible within longer school terms. Additionally, although interactive teaching strategies are a key feature of effective smoking and substance abuse prevention curricula, this study indicated that such aspects were barriers during implementation since teachers considered those strategies complex.

Participatory approaches in health promotion research to develop interventions may help address the challenging aspects of program implementation. Program developers may obtain feedback on teachers’ implementation needs and the feasibility of implementing a curriculum in their school setting. Program implementers, as key “agents of change,” should be involved in the stages of research to develop an intervention protocol. Such an approach ensures better implementation in and translation to the practice setting. Finally, by including the populations being served in research and program planning, the unique needs of diverse settings and communities are respected, which is key in conducting health promotion research and delivering interventions.