Using a Systematic Conceptual Model for a Process Evaluation of a Middle School Obesity Risk-Reduction Nutrition Curriculum Intervention: Choice, Control & Change

Abstract

Objective

To use and review a conceptual model of process evaluation and to examine the implementation of a nutrition education curriculum, Choice, Control & Change, designed to promote dietary and physical activity behaviors that reduce obesity risk.

Design

A process evaluation study based on a systematic conceptual model.

Setting

Five middle schools in New York City.

Participants

562 students in 20 classes and their science teachers (n=8).

Main Outcome Measures

Based on the model, teacher professional development, teacher implementation, and student reception were evaluated. Also measured were teacher characteristics, teachers’ curriculum evaluation, and satisfaction with teaching the curriculum.

Analysis

Descriptive statistics and Spearman’s Rho Correlation for quantitative analysis and content analysis for qualitative data were used.

Results

Mean score of the teacher professional development evaluation was 4.75 on a 5-point scale. Average teacher implementation rate was 73%, and student reception rate was 69%. Ongoing teacher support was highly valued by teachers. Teachers’ satisfaction with teaching the curriculum was highly correlated with students’ satisfaction (p <.05). Teachers’ perception of amount of student work was negatively correlated with implementation and with student satisfaction (p<.05).

Conclusions and implications

Use of a systematic conceptual model and comprehensive process measures improves understanding of the implementation process and helps educators to better implement interventions as designed.

INTRODUCTION

Nutrition education interventions are often complex, with various components. To accurately measure the outcomes of intervention, it is important to ensure that the implementation is completed as originally designed. Evaluating how an intervention is actually implemented is called a process evaluation, and helps us to avoid the error of drawing a conclusion about the effectiveness of the study without knowing whether or not the program has been adequately implemented (i.e. Type III error)¹. Process evaluation research has been done in a number of different settings and using both quantitative and qualitative methods ^2–7. Such studies have been useful for behavioral interventions by providing information that helps in interpreting the outcomes of the interventions and identifying barriers and facilitators of implementation. For example, the CATCH and 5-a-Day program studies ^2,8 provided considerable insight on the importance of the process evaluation and identified key process components. A book by Steckler and Linnan in 2002³ on process evaluation in the health promotion arena established that process evaluation is an important part of any public health intervention research and listed the following as key components that a process evaluation should examine: context, extent of reach of the program, dose of intervention delivered, fidelity to the intervention as designed, implementation (a combination of reach, dose delivered, dose received, and fidelity), and the nature of the recruitment process. Process evaluations have used terms differently for the different components, such as extent for dose, or exposure for reach, and many others. Baranowski and Jago⁹ emphasized that, in particular, behaviorally focused and theory-based health-related interventions need to be fully implemented if the desired outcomes are to be achieved and hence barriers and facilitators of implementation need to be identified and addressed. They developed a comprehensive conceptual model for evaluating the implementation process that incorporates components such as those listed above and links them in a stepwise fashion.

Examining issues of implementation is particularly critical in the design of school-based nutrition education interventions because numerous preventive intervention programs are delivered in the classrooms by classroom teachers^5,6,10–19. The extent to which teachers faithfully deliver a particular curriculum or incorporate instructional strategies emphasized by an intervention, especially those related to theory-based psychosocial variables that are motivators and facilitators of behavior change, is a critical issue for the overall program evaluation. As school-based nutrition education studies become increasingly complex, the field can move forward by applying systematic and comprehensive frameworks from the literature to examine the implementation process components ^3,9.

Large multi-component, and often multi-site, nutrition interventions have commonly conducted comprehensive school-wide process evaluations^4,6,16,20. Smaller scale nutrition curriculum studies have also begun to include process evaluation in their study designs^5,15. However, these studies have relied mostly on lists of process components without further investigating the relationships among those components or linking them with intervention outcomes. A sequentially laid out conceptual framework, such the one that Branowski and Jago⁹ proposed, may help us understand which factors hinder or facilitate the intervention process and where the barriers occur. Since Baranowski and Jago have suggested their conceptual model on multiple process pathways and relation of process to mediators and outcomes of intervention, to our knowledge, no study has been reported applying the model to practice.

The current study used a conceptual model, derived from the literature to examine the process of implementation of a middle school science and nutrition education intervention, Choice, Control & Change. Using a conceptual model to understand the process of delivering a curriculum intervention in a setting such as this one, within the classroom, is significant as it will help nutrition educators, teachers, and schools better implement, in similar classroom settings, nutrition education interventions as they were designed.

Overview of Choice, Control & Change

Based on social cognitive theory (SCT) ^21,22 and self-determination theory (SDT), the Choice, Control & Change curriculum was designed to encourage the adoption of behaviors related to obesity risk reduction, or energy balance related behaviors²³. These behaviors were increasing water consumption, fruit and vegetable consumption, and walking; and decreasing intakes of sweetened beverages and processed packaged snacks, and eating at fast food restaurants, and decreasing leisure screen time. Theory-based psychosocial variables were used as potential mediators for behavioral change^23,24.

The curriculum addressed selected national middle school science standards in biology, in particular dynamic equilibrium or energy balance. Specific educational activities of the curriculum were based on the use of a systematic science-inquiry procedure (or scientific method) that provided a meaningful scientific rationale for taking action ^23,24. The procedure involves a five-phase learning cycle. This cycle, called QuESTA begins with the students Questioning by thinking about what they already know and what questions they have about the topic of study. Next the students conduct hands-on Experiments to explore the questions. This is followed by Searching, where students research what other scientists already know, through readings provided in the curriculum, interviewing others, and exploring books and the internet. The next phase provides students the opportunity to synthesize what they have learned through Theorizing and creating their own ideas about the topic so that they can Apply what they learned in their lives.

The curriculum consisted of 24 lessons taught by science teachers over a period of about 8–10 weeks. Teacher professional development involved an intensive three-hour workshop before the curriculum and a second workshop about half way through the curriculum. Additionally, two implementation coordinators met with each teacher weekly in order to trouble shoot and provide on-going support and guidance.

The purpose of the current study was to examine the implementation of a middle school curriculum intervention and to use and review a conceptual model of process evaluation derived from the literature. Specific objectives were to describe in detail the process evaluation components in each pathway of the model and to evaluate each process component. In addition, further objective was to determine the relationships among multiple process components.

METHODS

Study Design and Participants

The present study was the process evaluation of Choice, Control & Change, for which the outcome evaluation consisted of a cluster-randomized controlled trial. Ten middle schools in underserved, low-income neighborhoods within the same school district in a large city were matched on school size, race/ethnicity, free/reduced lunch percentage, and reading and math test scores. One school of each matched pair was randomly assigned to the intervention and the other into the comparison condition. The school sample was a convenience sample, where the schools were identified and recruited in collaboration with the school district’s science education coordinator. The student and teacher demographics, however, were similar to those of the district as a whole ^25,26. A detailed description and the outcome evaluation study of the Choice, Control & Change curriculum have been reported elsewhere^23,24.

The data for the current study were from all the intervention schools: five schools and eight teachers, each teaching multiple classes, with a total of 20 classes and 562 students. Table 1 shows the description of the study population and school context. The mean age of students was 12 years and 51% of the students were male. About 26% were African-American and 73% were Hispanic students. The rate of free and reduced lunch was 83%. The proportion of students who met standards on the state reading and math tests were in between 19–39%. The present study was approved by the Institutional Review Boards of the Teachers College Columbia University and New York City Department of Education.

Table 1.

Description of school, teacher, and student characteristics in a middle school nutrition curriculum intervention, Choice, Control & Change.

	Description
Schools	n=5
Number of classes	20
Average class size	25.7
Ethnicity	31.0% African American 67.0% Hispanic 2.0% Other
% free and reduced lunch	82.8%
Reading score (1–5) a	1.6 (.5) b
Math score (1–5) a	2.0 (.7) b
Teachers	n=8
Total years of teaching c	2.29
Age category d	1.43
Students	n=562
Grades	9% 6th grade 91% 7th grade
Gender	51% boys

^aReading and math scoring systems represent proportion of students who met standards on the state tests: 1= 0–19%, 2 = 20–39%, 3 = 40–59%, 4 = 60–79%, 5 = more than 80%

^bValues represent mean (standard deviation)

^cResponse options: 1 = ~1 year, 2 = 2 years, 3 = 3 years, 4 = 4 years, 5 = > 5

^dResponse options: 1 = 20s, 2 = 30s, 3 = 40s, 4 = 50s, 5 = 60s

Conceptual Model for the Process Evaluation

The conceptual model for the current process evaluation was modified from the model proposed by Baranowski & Jago⁹. Their comprehensive model described the process components as a series of steps in a probabilistic chain and combines process evaluation components and outcome evaluation components. For the purposes of the present study, only the process evaluation portion of the model was used and this was simplified after some pilot testing to suit the needs of evaluating a classroom curriculum (see Figure 1). The process evaluation in the Choice, Control & Change consists of several components. Teacher professional development consisted of workshops and on-going teacher support to ensure that the teachers could successfully deliver the Choice, Control, & Change curriculum in middle school classrooms. Teacher implementation in this study was seen as a combination of faithfulness to the curriculum and lesson completion. These features are often referred to as fidelity and dose in other literatures ^{2,3,5,8,14,16,20,27,28}. Student reception in this study refers to how well students received the curriculum activities in class. The extent to which students actively engaged in the lesson activities (students’ engagement) and the extent to which any classroom management issues interfered with curriculum delivery determine the level of student reception of the curriculum. Measuring individual student satisfaction is another way to examine whether or not students were receptive to the curriculum.

Figure 1.

Conceptual model for the process evaluation of the Choice, Control, & Change curriculum

Barriers refers to the obstacles faced by teachers in implementing the curriculum as designed. Competing programs refer to similar nutrition or physical activity programs that might interfere with our curriculum implementation. External factors/school context such as average class size or standard test scores at the school level might also impact curriculum implementation in class.

Teacher characteristics such as teachers’ total years of teaching, age, and regular attendance at professional development sessions, teachers’ opinions about the curriculum, and their satisfaction with teaching it might influence how they implement the curriculum as well, and were therefore included in the model.

Measures

The current study used both quantitative and qualitative methods. Several instruments were developed to measure each process component described in the conceptual model. A professional development evaluation form, teachers’ online survey, class observation form, lesson completion form, and student survey were used in the study. Table 2 shows the process evaluation components and the corresponding measures for the components.

Table 2.

A list of measures used to evaluate process components of the Choice, Control & Change curriculum

Process components		Measures	Type of Data	Completed by	Timing of data collection
Teacher professional development	Workshops	First workshop: evaluation form	Quantitative: 5-point Likert scale	Teachers	Right after the workshop
		The second workshop: Teachers’ online survey	Quantitative: 5-point Likert scale	Teachers	Post intervention
	On-going teacher support	Teachers’ online survey	Qualitative: an open-ended question	Teachers	Post intervention
Teacher implementation	Faithfulness	Classroom observation form.	Quantitative: 5-point scale	Research staff	Throughout the intervention
	Completion	Lesson completion form	Quantitative: 5-point scale	Research staff	Weekly during intervention
Student reception	Student engagement	Classroom observation form	Quantitative: 4-point scale	Research staff	Throughout the intervention
			Qualitative: field notes	Research staff
	Classroom management	Classroom observation form	Quantitative: 3-point scale	Research staff
			Qualitative: field notes	Research staff
	Student satisfaction	Student survey	Quantitative: 4-point Likert scale	Students	Post intervention
Barriers		Teachers’ online survey	Qualitative: an open-ended question	Teachers	Post intervention
Competing programs		Student survey	Quantitative: Yes/No question	Students	Post intervention
External factors/school context		www.insideschools.org	Quantitative: secondary data	Secondary data	Beginning of the intervention
Teacher characteristics		Teachers’ online survey	Quantitative: categorical multiple choice	Teachers	Post intervention
Teachers’ curriculum evaluation		Teachers’ online survey	Quantitative: 5-point scale	Teachers
Teacher satisfaction with teaching the curriculum		Teachers’ online survey	Quantitative: 5-point scale	Teachers

Teacher professional development

Two workshops were provided to intervention teachers. To evaluate the first workshop, a one-page teachers’ professional development evaluation form was adapted from the CATCH study ²⁹. There were 4 scales in this questionnaire: (1) Curriculum overview; Curriculum activities; Curriculum materials; and (4) Overall evaluation of the professional development workshop. Response options involved a 5-point scale (1 = poor; 2 = not so good; 3 = satisfactory; 4 = good; 5 = excellent; N/A = not attended). Cronbach’s alpha values for the internal reliability test for these scales were .86, .95, .90, and .74, respectively. The second workshop and on-going teacher support were measured by the teachers’ online survey developed by the research team and completed by teachers at the end of the curriculum. Teachers answered the question “Please evaluate the overall quality of the SECOND professional development session, as you recall it” and the same response options were applied as described above. On-going teacher support was measured by one question “Was the weekly meeting with implementation coordinator helpful? Please explain.” Space was provided for teachers to write comments.

Teacher implementation

Faithfulness to the curriculum was measured by trained research staff with a classroom observation form, which included a quantitative scale. An initial score of 5 was assigned and if anything was altered, omitted, inserted, or replaced, then one point for each alteration was deducted from the total score. Therefore, the possible range for this scale is from 1 to 5.

Lesson completion was measured with a form that was specific to each lesson with a checklist listing each activity and QuESTA learning cycle component within the lesson. Each activity completed was checked off on the list, and then a lesson completion score was given based on a 5-point scale: 1= none; 2=small portion; 3=half; 4=most sections; and 5=all. The completion scores then were converted into percentages. In addition, the percentage of completion for each phase of the QuESTA learning cycle was calculated based on the number of activities within each phase covered by teachers, divided by total number of activities in each phase in the entire curriculum. The Theorizing and Applying to Life phases of the QuESTA cycle are especially important because through these phases, children refine their abilities to construct explanations and theories about what they have learned from their exploring and experimenting and to apply their learning to their everyday lives. The implementation coordinators completed the form with the teachers during the weekly meetings.

Student reception

Student engagement was also measured by trained research staff on the classroom observation form using a quantitative 4-point scale and qualitative open-ended field notes. The response options were: 1 = uninterested (overall, less than 1/3 of the participants involved); 2 = few/some involved (overall, between 1/3 and 2/3 of the participants involved); 3 = most involved (overall, more than 2/3 of the participants involved, but not all); and 4 = all involved (during the entire class period, all of the participant actively involved). Observers completed field notes under “Describe specific situations to explain your choice of score”.

Classroom management was measured by trained research staff on the classroom observation form using a 3-point scale and qualitative open-ended field notes. Response options were: 1 = major problems (where extensive disciplinary issues throughout the class period hindered delivery of the lesson); 2 = minor problems (when the class was disturbed by student(s)’ behavioral problems, but the teacher was able to handle the problems and keep doing the lesson); and 3 = no problems (when the lesson was completed without any classroom management issues). Observers completed field notes under “Describe specific situations to explain your choice of score”.

Student satisfaction with the curriculum was measured using one question in the student survey “Did you like Choice, Control & Change curriculum?”, and 4 response options were, 1= not at all; 2= a little; 3= somewhat; and 4= mostly.

Implementation barriers

One open-ended question in the teachers’ online survey asked what teachers’ perceived implementation barriers were.

Competing programs and external factors/school context

One question in the Choice, Control & Change student survey that was given at the end of the curriculum asked whether or not students had received any similar programs. External factors and school context information were collected from one of New York City school information online resources.

Teacher characteristics, teachers’ curriculum evaluation, and satisfaction with teaching the curriculum

Teacher characteristics collected for the study included teachers’ age, total years of teaching, and participation in professional development activities. For the teachers’ curriculum evaluation, teachers answered 16 questions. Examples of the questions included “compare the Choice, Control & Change program with other science curricula you have taught in the following areas: amount of science content, amount of science process…etc.” The response options were on a 5-point scale, from 1= less than others, 3= same as others, and 5= more than others (Cronbach’s alpha value for internal consistency reliability of scale = .92). Satisfaction with teaching the curriculum was measured with two questions: “Did you like teaching the Choice, Control & Change curriculum in your class(es)?” using a 5-point scale, from 1= not at all to 5= completely, and an open-ended question asking whether or not teachers would teach Choice, Control & Change again the next year. All the questions in this section were asked in the teachers’ online survey.

All instruments were revised several times through pilot tests, and content validation was performed through a review by several science teachers and nutrition intervention experts.

Data Collection Procedures

The professional development evaluation form was administered to the teachers right after the first workshop session. The teachers’ online survey was conducted to collect some quantitative data including the second workshop evaluation, teacher characteristics, teachers evaluation of curriculum content, and teacher satisfaction and some qualitative data such as their assessment of the on-going teacher support and of implementation barriers. The survey link was emailed to teachers after they finished implementing the curriculum, and then weekly reminders were sent. If teachers did not respond after two months, then data were treated as missing.

Classroom observation forms were completed by three trained research staff to measure teacher implementation and student reception. The observation schedule was made so that the research staff could attend at least one third of the 24 lessons for each of the 20 classes. The protocol was for each class to be observed 8 times so that there would be 160 direct observations. To ensure the consistency of the observation among observers, a series of inter-rater reliability tests was performed. Two observers at a time observed a few classes together and examined the percent agreement. The scoring systems were adjusted until the percent agreement between observers reached 95%.

Lesson completion forms were completed by implementation coordinators at the weekly teacher interviews. Implementation coordinators scheduled weekly meetings with all intervention teachers, and went over each lesson activity together and checked if the teachers covered them all. The protocol was to have lesson completion data for all 480 sessions (24 lessons in each of 20 classes).

The student survey was administered at the end of the curriculum to measure student satisfaction with the curriculum and if students participated in any competing programs similar to Choice, Control & Change. This one-page survey also included several other questions related to goal setting, which were not included as part of the current study. The timing of the data collection is described in Table 2.

Statistical Methods

Descriptive statistics were used to describe the process evaluation components. To investigate the relationship between variables, Spearman’s rho correlation was used as a non-parametric statistic method due to the small number of teachers and class sample size. Statistical Package for the Social Sciences (SPSS version 17.0 for Windows, SPSS Inc., Chicago, IL, 2008) was used for data analyses. Content analysis method was used for all qualitative data from observers’ field notes and open-ended questions.

RESULTS

Teacher Professional Development

Seven out of eight total intervention teachers attended the first workshop and the teacher who did not attend had a private session with an implementation coordinator. Four teachers attended the second workshop session, and all teachers received on-going teacher support.

Results from the questionnaire evaluating the first workshop showed that the means were: 4.8 on a 5-point scale for the curriculum overview and 4.7 for the curriculum activities, curriculum materials, and overall session (response options were 1 = poor; 2 = not so good; 3 = satisfactory; 4 = good; 5 = excellent). The mean score for the second workshop evaluation question was 4.25 (the same response options as above).

According to the content analysis results from the open-ended question on on-going teacher support, all teachers who participated in the survey reported positively about the support they received from the implementation coordinators. They appreciated the on-going nature of the support and opportunity to discuss issues and difficulties as well as share success stories. They also reported that the support assisted them with the implementation of the program by providing advice and keeping them focused.

Teacher Implementation & Student Reception

On average, 43% of the lessons (about 10 lessons out of 24 lessons) were observed by classroom observers who recorded teacher implementation and student reception levels for each lesson.

Teacher implementation

Table 3 shows the results: the mean of faithfulness to the curriculum was 76%, and the mean of lesson completion rate was 70%. Levels for faithfulness to the curriculum and lesson completion were divided into three categories as: below 33% = low implementation, 33–67% = medium implementation, and greater than 67% = high implementation. These categories were based on those suggested by previous studies, which found that fidelity and lesson completion levels ranged from less than 33% to over 90% ^{8,14–16,20,27–29}.

Table 3.

Teacher implementation results for Choice, Control & Change, a middle school nutrition curriculum intervention (n=8 teachers, 20 classes).

Measure		Range	Mean	Level e
Faithfulness to the curriculuma	Score c	3.1 – 4.7	3.8 (.4)d	High
	Percentage	62 – 93%	76%
Lesson completionb	Percentage	60 – 93%	70%	High

^aFrom classroom observation form completed by researcher

^bFrom lesson completion form completed by researcher in conjunction with the teacher

^cScoring system: 5 = if nothing was changed; the score was reduced by 1 when any of the following cases happened: teachers altered sequence of the curriculum, omitted materials, inserted materials, or replaced authentic assessments with conventional tests.

^dMean score (Standard deviation)

^eLevel categories: low < 33%, medium = 33–67%, and high > 67%.

The percentage of each of the QuESTA Learning Cycle phases covered in the classroom is following: Questioning (57%); Experimenting (89%); Searching (81%); Theorizing (71%); and Applying to Life (52%).

Student reception

The classroom observation data showed that the mean of student engagement was 72%, and the mean of classroom management was 66.5% (Table 4). Content analysis results from qualitative data showed that the students were highly engaged in hands-on activities but less so in reading, writing, or discussions. In particular, the students were very engaged in activities such as burning a peanut to demonstrate energy in food, measuring fat and sugar in popular food, demonstrating blood flowing through clear and fat-clogged (play dough) blood vessels (plastic tubing), and tasting substances to explore human’s biologic preferences for sweet, sour, bitter, and salty tastes. What differentiated well-implemented from poorly implemented classrooms was whether or not the students not only participated actively in the hands-on activities but also were able to theorize by enthusiastically discussing and writing about the activity. In addition, teachers used different management styles to discipline students and to gain students’ attention or to calm them down. A coding manual had been developed that was used by the implementation coordinators to code their field notes about teachers’ management styles. Part of the coding manual is provided in Table 5, and examples of the field notes of the observers describing two different management styles (as how they are coded) are as follows:

“Much yelling! Confrontations between teacher and several students. One student sent out of the classroom and brought back in and out over and over.” (Code = 1.2.5)

“One student is chatting with her neighbors instead of doing her work. The teacher asks her to stand up…the teacher talks to them about self-control and how to be respectful in class…the teacher asks them to write down specific rules that they will have to follow…to help them remember to behave.” (Code = 1.2.2)

Table 4.

Student reception results for Choice, Control & Change, a middle school nutrition curriculum intervention (n=20 classes, 562 students)

Measure	Range	Mean	Level d
Student engagement (1–4) a	2 – 4	2.9 (.6) c	High
Percentage	48.8 – 100%	72.3%
Classroom management (1–3) b	1 – 3	2.0 (.6) c	Medium
Percentage	33.3 – 100%	66.5%

^aScoring system: 1 = uninterested, 2 = few/some involved, 3 = most of them involved, 4 = all actively involved

^bScoring system: 1 = major problem, 2 = minor problems, 3 = no problems

^cMean score (Standard deviation)

^dLevel categories: low < 33%, medium = 33–67%, and high > 67%.

Table 5.

Part of the coding manual for the classroom management observation field notes in Choice, Control & Change

General domain/indicators	Specific indicators
Teacher’s management style	1.1Have no control 1.2Discipline methods 1.2.1Stop lesson to keep them in control 1.2.2Talked to students to be respectful or have a serious talk with students 1.2.3Threat with detention 1.2.4Yelling 1.2.5Sending students out of classroom 1.2.6Wait until students get quiet 1.2.7Ignore the problem 1.2.8Punishment

Based on an analysis of the coordinators’ field notes, an effective management style was the teacher having a serious talk with students about their behaviors and attitudes.

The individual student satisfaction rate was analyzed separately because the unit of measurement was different from classroom observation. The mean student satisfaction rate was 2.9 ± .9 (n=379) on a 4-point scale.

Other Factors Related to Program Implementation

Teacher characteristics

Demographics

Most participating teachers (5 out of 8 or 63%) were in their 20s and three teachers were in their first year of teaching. Mean years of teaching was 2.3 years. Three teachers attended professional meetings at least several times per year, but the other teachers did not participate in any. Teacher characteristics were not significantly correlated with teacher implementation, student reception, or student satisfaction.

Teachers’ curriculum evaluation

The mean score of comparing the Choice, Control & Change curriculum with other science curricula was 3.23 on a 5-point scale. Among 16 items, teachers’ perceived amount of individual work such as worksheets and writing had significant negative correlations with teacher implementation (r = −.81; p<.05) and student satisfaction (r = −.80; p<.05).

Teacher satisfaction with teaching the curriculum

The mean score on whether teachers liked teaching the curriculum was 4 on a 5-point scale (score range was from 3 to 5), and all teachers said they would teach Choice, Control & Change again the next year.

Teachers reported that they liked the hands-on experiments, its relevance and applicability to students’ lives, having a specific student workbook, the readings provided to students involved innovative concepts, and the goal setting activities. Some teachers stated there was too much introspective writing required of students. There was a significant positive correlation between teacher satisfaction with teaching the curriculum and student satisfaction (r = .76; p<.05).

Implementation barriers

Students’ misbehavior was teachers’ biggest challenge. Other barriers included conflicts with other tasks in schools and language barrier for Spanish speaking students.

Competing programs

There were no competing programs reported.

External factors/School context

As described above, Table 1 provides a description of school context. The relationship between school context and other process components was examined. Non-parametric correlation statistic results showed that there were statistically significant associations between ethnicity and teacher implementation, and between ethnicity and student satisfaction rates: (1) having more students of African American ethnicity in school was associated with higher implementation and student satisfaction rates (p<.01) and (2) having more students of Hispanic ethnicity in school was associated with lower implementation and student satisfaction rates (p <.05). However, it should be noted that the sample size was very small and there is no indication of a causal relationship.

DISCUSSION

The current study makes an important contribution to our understanding of process evaluation for school-based nutrition interventions. Using a conceptual model to conduct a systematic process evaluation for the Choice, Control & Change intervention helped to determine which implementation components were important to evaluate and how to organize the collection of data. Use of the framework also led to new understandings about relationships among specific implementation components. Many recent school-based studies incorporate a process evaluation component but do not use a model to show how the process elements are related to each other and to outcomes ^5,6,15,19. The model used here was adapted from a more comprehensive and complex model proposed by researchers⁹. Careful review of the school setting and the nature of a classroom curriculum intervention suggested that the original model needed to be simplified considerably. The modified model conceptualized the major process components as being in a series of steps linked in a probable chain. This study found this simplified model to be useful for identifying the relevant implementation components and for understanding how these were linked.

The teacher professional development component consisted of workshops and on-going support. The evaluation results for the teacher professional development workshops showed that the overall evaluation score was high, indicating that the professional development workshop was generally successful. Teachers reported that the curriculum overview, concepts, activities, and materials of the Choice, Control & Change curriculum were clear, held teachers’ attention, provided new information, and offered useful practice opportunities.

These positive results are similar to those found in other studies. Many school-based nutrition intervention studies provide professional development and most have shown that professional development sessions are successful and attendance records high ^4,6,16,17,28. However, intervention implementation levels have varied from 30% to over 90%, despite high attendance at the professional development sessions. This raises the question as to what kind of professional development may be most effective for curricula to be implemented as designed. A review of elements of effectiveness showed that teacher professional development was more effective when sessions were spaced across time, suggesting that consistent implementation support for teachers is necessary ³⁰. For example, in the Gimme 5 study, even though attendance at the professional development event was high (95%), 82% of the teachers reported that they would like someone to be available at the school to answer questions during curriculum implementation ²⁷. The overall implementation score for Gimme 5 was about 47%. Therefore, on-going teacher support may be crucial when an intervention is implemented not by the researchers but by school teachers. Classroom observation results from another school-based intervention among American Indian children—the Pathways study—also indicated the need for regular classroom visits by intervention staff members to provide teachers with assistance and support in implementing the Pathways curriculum ¹⁶.

These prior findings were the basis for the on-going support provided in the current study and such support may have contributed to the relatively high teacher implementation levels. The qualitative data from the current study indicated that teachers felt that the on-going support helped them with implementing the program by providing advice and keeping them focused. This of course, raises the issue of cost. Can such on-going support be maintained long-term? Some curriculum interventions have found ways to fund on-going support through grants or programs such as the Supplemental Nutrition Assistance Program – Education (SNAP-Ed) for low-income schools. With the rise of the internet, on-going support can be provided by web-sites with online teacher support materials and activities, such as web-embedded videos demonstrating the curriculum activities, background information, interactive worksheets, and more.

An important contribution of the current study is the focus on the conceptualization of the classroom implementation process as consisting of not only what the teacher does (teacher implementation) but also what the student actually experiences or receives (student reception). These are sometimes described as dose delivered and dose received. Often these are conflated into one variable, dose.

As in some other studies, the teacher implementation component is considered to be made up of faithfulness to the curriculum (fidelity) and completion of the lessons. The faithfulness to the curriculum was 76% and lesson completion was 70%, which are both considered “high.” However, the range was from about 60% to 93% for both, suggesting that while implementation levels were very encouraging for all, the on-going support did not have the same impact on all teachers. Other studies have found that the range of the implementation level was as low as 30% to as high as 97%^{8,14–16,20,27–29}. Clearly additional work needs to investigate how to motivate and support all teachers.

This study also found that the activities of the curriculum that fell into the Questioning and Experimenting categories were more likely to be implemented (89% for hands-on experimenting, for example) whereas those that were more behavior-change focused --Applying to life -- involving goal-setting and self-monitoring activities, were implemented less frequently (52%). This is similar to the findings of other studies, which have found that the more traditional or content aspects of the curriculum were considerably more fully implemented than the behavior-change aspects of the curriculum²⁷. The fact that Choice, Control & Change was implemented in inquiry-based science education classrooms meant that teaching traditional science content and conducting experiments were more familiar. Focusing on behavior change was less likely to be seen as relevant to science education. Professional development and ongoing assistance should thus focus on the behavior change aspects of a curriculum to ensure that teachers are comfortable and competent in these aspects.

The student reception component in this study is made up of student engagement, classroom management, and student satisfaction. While other studies have evaluated dose reception, they have measured it primarily in terms of student liking of, or satisfaction with, the curriculum, or students following instructions or appear interested^6,18,19. In this study, besides student engagement and student satisfaction, classroom management was also examined in detail because often there were many student behavioral issues that distracted the class from fully receiving the intervention. The current study identified specifically on how teachers managed classroom obstacles. For example, observations in the classroom found that what was effective was when teachers talked seriously with students about their behaviors and about being respectful of each other and what was least effective as when teachers yelled at students. Classroom observation results showed that the student reception of the program varied by classroom. Because how well the students received the classroom activities was influenced by teachers’ classroom management abilities, a discussion of teachers’ management styles may be important at the initial professional development workshops.

In terms of teachers’ curriculum evaluation, when teachers thought that the amount of individual work required by the Choice, Control & Change curriculum and the number of discipline issues that arose during class were greater than when they taught some other science curriculum, the implementation rate of the program was lower. Interestingly, student satisfaction rates were then also lower. This could be because teachers’ sense of being overwhelmed by the number of student worksheets and by the discipline issues they faced in implementing the curriculum made students feel less enthusiastic about it.

On the other hand, teacher satisfaction with teaching the curriculum was significantly and highly correlated with the student satisfaction rating. This could be interpreted that when teachers enjoyed teaching the curriculum and thus showed enthusiasm, this produced a better learning environment for the students. It is possible as well that the students could feel more emotionally connected to the teacher who is more engaged and be influenced positively thereby ³¹. Teachers’ enthusiasm was investigated via process evaluation in another study. The TEENS study found that teachers’ ability or interest in teaching a behaviorally based nutrition curriculum was limited, and researchers commented that this may have been related to the non-significant outcomes¹⁴.

A positive association was noted between the percentage of African American students in school and teacher implementation as well as student satisfaction. The percentage of Hispanic students in school was negatively associated with teacher implementation and student satisfaction. However, ethnicity information was gathered from only three schools: two schools had a greater African American population and had relatively high implementation levels and student satisfaction rates, and the remaining school was comprised mostly of Hispanic students. It was one of the least implemented programs and it reflected low student satisfaction scores. Without having a larger sample size and more information related to other factors, such as whether students had language barriers or acculturation difficulties if they were from immigrant families, it is hard to conclude that there is a strong relationship between ethnicity and curriculum implementation. Other cultural factors may also have been present that the current study did not measure.

The Planet Health study, which was an obesity reduction research for middle school students, examined evidence regarding the reduction in obesity among different ethnic groups¹¹. The results from the Planet Health study showed that the largest intervention effects were seen among African American girls, with obesity prevalence significantly reduced in intervention compared to control participants. The intervention effect among white girls was similar to the overall results while among Hispanic girls, the results were statistically insignificant. However, no differences were found for boys. The Planet Health study did not examine whether any process components were related to this ethnic outcome discrepancy among girls. Further research to investigate intervention effects among different ethnic groups is needed.

Using a clearly articulated conceptual model for the process evaluation was one of the strengths of the study. In particular, the conceptual model used here was parsimonious and streamlined, with the components laid out in step-wise logical order. The average percentage of classrooms observed was 43%, which is exceptionally high as was the detailed observation notes compared to other studies, thus allowing us to understand in-depth program the implementation process. In this study, most teachers in a given school taught several classes. Thus, although a limitation was having a small sample size of teachers (with limited variability in age and years of teaching) and schools, the numbers of classes and students were relatively large. Because 10 out of 24 lessons were observed for each of 20 classes, there were about 200 direct observations. Lesson completion data were collected for all 480 sessions. However, the limitation of small number of teachers needs to be kept in mind when generalizing the specific findings to other settings and populations. Another limitation was that only half of the teachers attended the second professional development workshop. However, many believed they did not need to attend because the implementation coordinators provided sufficient on-going teacher support. Nevertheless, attendance may have increased the overall implementation rates even higher.

IMPLICATIONS FOR RESEARCH AND PRACTICE

This study has implications for both research and practice. As nutrition education interventions become increasingly complex, a clear conceptual model for the implementation process, specific assessments for each component of the model, and an examination of relationships among components is important. The conceptual model used in this study is parsimonious and captures the key process components and can be used in future implementation research studies in other similar samples and settings. The model can be modified and improved through research for use in other specific situations. Other models can also be investigated. In terms of practice, the model can help educators and nutrition practitioners who work with classroom teachers to think about the various process components that need to be considered as they implement a curriculum. The study found that in addition to professional development sessions, on-going teacher support seems important to ensure faithfulness to the curriculum and lesson completion. Student reception of the program and active engagement by students depends on teachers’ classroom management skills and also on teachers’ satisfaction with teaching the intervention curriculum, which in this study was strongly related to student satisfaction with the curriculum. Therefore, enhancing the quality of teacher materials by incorporating teachers’ feedback through extensive pilot testing may help to ensure their satisfaction with them. The current study can provide nutrition educators with a model for how to conduct systematic process evaluations for nutrition programs in similar classroom settings and thereby contributing to the increase in desired outcomes.