Linking implementation process to intervention outcomes in a middle school obesity prevention curriculum, ‘Choice, Control and Change’

Abstract

This study investigates the link between process evaluation components and the outcomes of a school-based nutrition curriculum intervention, ‘Choice, Control and Change’. Ten New York City public middle schools were recruited and randomly assigned into intervention or control condition. The curriculum was to improve sixth to seventh grade students’ energy balance related behaviors, based on social cognitive and self-determination theories, and implemented during the 2006–2007 school year (n = 1136). Behaviors and psychosocial variables were measured by self-reported questionnaires. Process components were evaluated with classroom observations, teacher interviews, and a student questionnaire. Using ‘Teacher Implementation’ (dose delivered) and ‘Student Reception’ (dose received) process data; intervention group was further categorized into medium- and high-implementation groups. Analysis of covariance revealed that, compared with control group, only high-implementation group showed significant improvement in students’ behavior and psychosocial outcomes. Hierarchical linear models showed that ‘Teacher Implementation’ and ‘Student Reception’ significantly predicted students’ sweetened beverage outcomes (P < 0.05). ‘Student Satisfaction’ was also greater when these implementation components were higher, and significantly associated with behavior and psychosocial outcomes (P < 0.05). Implementation process influenced the effectiveness of the ‘Choice, Control and Change’ intervention study. It is important to take into account the process components when interpreting the results of such research.

Introduction

Implementing field-based intervention programs is influenced by multiple factors and often encounters various unexpected challenges. Without information on the implementation process, it is difficult to judge whether a program’s failure or success is due to program design or to how well the program was implemented as planned [1–6]. Therefore, documenting program implementation through a process evaluation is crucial for field-based intervention studies. Among the interventions for health outcomes, a comprehensive review study that examined over 500 intervention programs directed at children and adolescents concluded that the implementation process influences program outcomes [7].

As the obesity epidemic has become a serious public health issue, interventions targeting eating and physical activity behavior have increased. There have been increased calls for measuring and reporting the implementation process in programs directed at diet and physical activity [3, 8–11]. In particular, school-based nutrition interventions to prevent childhood obesity have led to inconsistent results and small effect sizes. Discovering specific contributors to the level of implementation and their impacts on the program health outcomes would be one of the keys to improving school-based intervention research. Even though process evaluations of school-based nutrition interventions have increased [9, 10, 12–20], only a small percentage of published studies have investigated the influences of implementation on program outcomes [10, 18–20].

One of the early school-based nutrition intervention studies examining the relationship between implementation components and study outcomes was the Child and Adolescent Trial for Cardiovascular Health (CATCH). The study examined the relationships among teacher characteristics, measures of curriculum implementation, competing programs and student outcomes [18]. In particular, the study used a conceptual model to analyse the mediating and moderating effects of the measures of program implementation on the measures of study outcomes. Among the intervention schools, the study found that the percentage of classroom sessions modified by the teacher, as a measure of the fidelity, was associated with increasing student self-efficacy and knowledge outcomes.

The Lifestyle Education for Activity Program (LEAP), a school-based intervention designed to promote physical activity for high school girls, examined the primary study outcomes of self-reported vigorous physical activity among high implementation, low implementation and control schools [10]. Pair-wise comparisons showed that girls in the high-implementation schools had a higher prevalence of participation in vigorous physical activity than girls in control schools. The test for a linear dose-response was also significant, indicating that there was a dose effect in proportion to participation in vigorous physical activity from control, low implementers to high implementers.

Because linking implementation to study outcomes aids in interpreting study results and it is a much-needed research area, this study aimed to investigate the link between the implementation process and the study outcomes of a middle school nutrition education curriculum intervention to improve energy balance related behaviors (EBRBs), ‘Choice, Control and Change’. The objectives of the study are to examine (i) how the effectiveness of the curriculum differs based on their high- medium- and low-implementation process level compared with the control group and (ii) linear relationships between process components and student outcomes.

Methods

‘Choice, Control and Change’ study description

The study design was a cluster randomized controlled trial conducted in the 2006–2007 school year. Within the same school district, 10 public middle schools in underserved, low-income neighborhoods in New York City were recruited and matched on school size, race/ethnicity, percentage of students eligible for free or reduced price lunch and reading and math test scores. One school of each matched pair was randomly assigned to the intervention (5 schools, 20 classes and 562 students) and the other into the comparison condition (5 schools, 21 classes and 574 students). There were no baseline characteristic differences between intervention and control groups. Table I shows the baseline characteristics of the study setting and participants. The mean age of the students was 12 years and 51% of the students were male. The majority of students were Hispanic or African American.

Table I.

Baseline characteristics of the schools, teachers, and students in the ‘Choice, Control and Change’ project

	Intervention	Control	U valueb (significance)
School	n = 5	n = 5
Class size	25.7	27.6	6.5 (0.385)
Ethnicity	25.9% African American	44.8% African American	4.0 (0.297)
	72.7% Hispanic	52.5% Hispanic
% students eligible for free and reduced price lunch	85.2%	72.5%	5.0 (0.221)
Reading score (1–5)a	1.6 (0.5)	1.6 (0.5)	12.5 (1.0)
Math score (1–5)a	2.0 (0.7)	1.6 (0.5)	8.5 (0.339)
Teacher	n = 8	n = 7
Age categoryb	1.4	2.0	16.5 (0.460)
Total years of teaching	2.3 (1.6)	3.3 (1.9)	17.0 (0.319)
Student	n = 562	n = 574
Grades	9% sixth grade	100% seventh grade	NA
	91% seventh grade
Gender	51% boys	51.7% boys	χ2 = 0.058 (0.809)c

^aReading and math scoring systems represent proportion of students who met standards on the state tests: 1 = 0–19%, 2 = 20–39%, 3 = 40–59%, 4 = 60–79%, 5 =>80%. ^bResponse options: 1 = 20 s, 2 = 30 s, 3 = 40 s, 4 = 50 s, 5 = 60 s. ^cResponse options: 1 = ∼1 year, 2 = 2 years, 3 = 3 years, 4 = 4 years, 5 = >5. ^dMann-Whitney U-statistic (significance). ^eChi-square statistic (significance).

The intervention was a science and nutrition education curriculum designed to impact middle school students’ EBRBs: eating more fruits and vegetables, drinking more water, increasing physical activity and decreasing intakes of sweetened beverages and packaged snacks, eating at fast food restaurants and leisure screen time [21]. Social cognitive theory and self-determination theory together served as the behavioral change theoretical framework. The curriculum addressed theory constructs such as self-efficacy, autonomy and competence. The curriculum consisted of 24 lessons and was taught by science teachers in science classes most school days over 8–10 weeks, between September and December 2006. The control schools received regular science curriculum during the same period, and received the delayed intervention in spring 2007. The study was approved by the Institutional Review Boards of the Teachers College Columbia University and New York City Department of Education.

Evaluation model

The evaluation model for ‘Choice, Control and Change’ conceptualizes the major process evaluation components as being in a series of steps linked to the intervention outcomes (Fig. 1). The sequence of the model has been revised slightly from the original version that was published in 2013 [22]. The description of each component, however, remains the same. The process components in this model are as follows. School teachers receive ‘Professional Development’ that consists of ‘Workshops’ and ‘On-Going Support’ to ensure that teachers understand and successfully deliver the curriculum. The degree to which teachers implemented the curriculum (‘Teacher Implementation’) as designed can be examined by the level of ‘Faithfulness to the Curriculum’ and ‘Lesson Completion’. ‘Student Engagement’ and ‘Classroom Management’ determine whether or not students receive the curriculum (‘Student Reception’). ‘Student Satisfaction’ is about how individual students like the curriculum. ‘Barriers’ are the factors that may hinder implementation or reception of the curriculum. ‘Competing Programs’ refers to similar nutrition or physical activity programs that may also interfere with the curriculum. There can be any ‘External Factors/School Context’ such as average class size or standard test scores at the school level that interact with the curriculum implementation as moderating factors. ‘Teacher Characteristics’ such as teachers’ total years of teaching, teachers’ opinions about the curriculum (‘Teachers’ Curriculum Evaluation’), and ‘Teacher Satisfaction with Teaching the Curriculum’ are the components that may interact with ‘Teacher Implementation’, barriers and student reception. Detailed descriptions of each process component and relationships among these components have been published elsewhere [22].

Fig. 1.

The conceptual model for program evaluation of the ‘Choice, Control and Change’ curriculum.

Note: the current version is modified from the one in Lee et al.(2013) [21].

The outcome evaluation components in the model consist of mediating variables and behavioral outcomes. Changes in behavioral outcomes are proposed to be the result of change in mediating variables, and changes in the mediating variables are induced by implementing programs [8].

Measures and data collection

Baseline data were measured in September 2006, and the post-intervention measurement was completed in June 2007.

Outcome evaluation measures

A 30-item food and activity frequency questionnaire measured behavioral outcomes. The questionnaire, the ‘EatWalk Survey’, was a modified version of the validated Block food frequency instrument for children [23]. An instrument, ‘Tell Me About You’, measured theory constructs: outcome expectations, intention to change, perceived barriers, self-efficacy and autonomous motivation (competence and autonomy). Validity, reliability and data collection of the questionnaires are described in detail elsewhere [21].

Process evaluation measures

‘Faithfulness to the Curriculum’ was measured with a classroom observation form and trained classroom observers completed the forms. An initial score of 5 was assigned for the scale and if anything was altered, omitted, inserted or replaced, then one point for each alteration was deducted from the total score. Therefore, the possible range for this scale was from 1 to 5. ‘Lesson Completion’ was measured with a form that was specific to each lesson with a checklist listing each activity within the lesson. Each activity completed was checked and the lesson was scored on a 5-point scale: 1 indicated that none of the lesson was completed and 5 indicated that the entire lesson was completed. The implementation coordinator completed the form with the teacher during their weekly meeting.

‘Student Engagement’ and ‘Classroom Management’ were both measured on the classroom observation form completed by trained research staff. ‘Student Engagement’ scale was a quantitative 4-point scale. The response options were: 1 = uninterested (overall, <1/3 of the participants involved); 2 = few/some involved (overall, between 1/3 and 2/3 of the participants involved); 3 = most involved (overall, >2/3 of the participants involved, but not all); and 4 = all involved (during the entire class period, all of the participant actively involved). ‘Classroom Management’ used a 3-point scale. Response options were: 1 = major problems (where extensive disciplinary issues throughout the class period hindered delivery of the lesson); 2 = minor problems (when the class was disturbed by student(s)’ behavioral problems, but the teacher was able to handle the problems and keep doing the lesson); and 3 = no problems (when the lesson was completed without any ‘Classroom Management’ issues).

‘Student Satisfaction’ with the curriculum was measured using one question, asked once post intervention, “Did you like ‘Choice, Control and Change’ curriculum?” with response options: not at all; a little; somewhat; and mostly.

Summary of the ‘Choice, Control and Change’ program results

Outcome evaluation

The intervention resulted in significant changes in targeted EBRBs; in particular, students who participated in the curriculum reported fewer sweetened beverages and processed packaged snacks, smaller sizes at fast food restaurants, decreased leisure screen time and increased physical activity compared with control students [21]. Students also showed significantly positive results in outcome expectations, self-efficacy, goal intentions, competence and autonomy.

Process evaluation

Overall, the ‘Choice, Control and Change’ curriculum was well implemented by teachers and well received by students [22]. Yet, there were some variations in process components across classes and schools across the 20 classes. The range of ‘Faithfulness to the Curriculum’ scores was 62–93%, with a mean of 76%. For ‘Lesson Completion’, the range was 60–93% and the mean was 70%. The range of ‘Student Engagement’ was 49–100%, with a mean of 72% and the ‘Classroom Management’ ranged from 33 to 100%, with a mean of 67%. For ‘Student Satisfaction’, the mean score was 2.9 (± 0.9) on the 4-point scale.

Statistical analysis

Process components in the analysis included ‘Teacher Implementation’ and ‘Student Reception’ (Fig. 1). ‘Teacher Implementation’ was a combined score of ‘Faithfulness to the Curriculum’ and ‘Lesson Completion’. Two measures were highly correlated (r = 0.81; P < 0.001) and worked well as a scale (Cronbach’s alpha = 0.89). The mean ‘Teacher Implementation’ was 74% (range 61–91%). Similarly, ‘Student Reception’ was a combined score of ‘Student Engagement’ and ‘Classroom Management’, which were highly correlated (r = 0.93; P < 0.001), and worked well as a scale (Cronbach’s alpha = 0.94). The average of ‘Student Reception’ was 69% (range 41–100%). Determined by previous studies, levels of these process components were divided into three categories as: low (<33%), medium (33–67%) and high (>67%) [22]. The data showed that there was no low process level group. Therefore, analysis of covariance (ANCOVA) was used to examine how student outcomes differ among three study groups (high, medium process level and control groups). Gender and baseline scores were included as covariates, and Bonferroni correction was applied to adjust for multiple comparisons.

Because process components were measured at the classroom level while student outcomes were measured at the individual level, a hierarchical linear model (HLM) was used to investigate any linear relationship between the process data and the student outcome data. Variance components and intra-class correlation coefficient were investigated and the results showed that there were significant outcome variances (P < 0.05–0.001) at the class level. In addition, the unit of curriculum implementation and observation was the classroom, and qualitative data also confirmed that there were implementation differences across intervention classes [22]; therefore, the classroom level process variables were used as random effects in the HLM models. Gender and baseline scores were group-centered and controlled as a fixed effect and a covariate, respectively. Dependent variables included all student behavioral and psychosocial outcomes. HLM (version 6.0 for Windows, Scientific Software International, Inc., Lincolnwood, IL, 2004) statistical software was used for all analyses.

Finally, a series of multiple regression analysis was performed to investigate the relationship between ‘Student Satisfaction’ and the primary outcomes (behavioral and psychosocial outcomes). Baseline data were included as a covariate and Bonferroni correction procedure was applied to adjust for multiple statistical tests. Prior to any adjustment, all data analyses were tested at 0.05 significance level.

Results

‘Teacher Implementation’ influence on student outcomes

Table II illustrates student outcome results based on the ‘Teacher Implementation’ levels. Students from the high ‘Teacher Implementation’ classes significantly consumed fewer sweetened beverages at meals (P = 0.001) and with snacks (P = 0.003), packaged snacks (P = 0.016) and fast food value/combo meals (P = 0.047), and smaller sizes of sweetened beverages at meals (P < 0.001) and with snacks (P = 0.012) and fast food (P = 0.001), compared with control students. There was no significant difference in any eating behavior outcomes between medium ‘Teacher Implementation’ classes and control group. Both medium and high ‘Teacher Implementation’ groups showed significant improvements in physical activity and screen time behavior outcomes compared with control group (P < 0.05).

Table II.

Behavioral outcomes by the ‘Teacher Implementation’ in the ‘Choice, Control and Change’ project

Scale	Adjusted post mean (95% CI)			P value for Omnibus test
	Control (n = 350)	Medium teacher implementation (n = 149)	High teacher implementation (n = 251)
Food choices
Fruit at meals
Days previous week (0–7)	3.5 (3.2, 3.7)	3.6 (3.2, 4.1)	3.7 (3.4, 3.9)	0.449
Pieces per day (0–4)c	2.0 (1.9, 2.1)	1.9 (1.7, 2.1)	2.0 (1.9, 2.2)	0.630
Fruit for snacks
Days previous week (0–7)	3.7 (3.4, 4.0)	3.4 (2.9, 3.9)	3.3 (3.0, 3.6)	0.110
Pieces per day (0–4)c	2.3 (2.1, 2.4)	2.1 (1.8, 2.4)	2.0 (1.8, 2.4)	0.057
Vegetables at meals
Days previous week (0–7)	2.5 (2.2, 2.7)	2.6 (2.2, 3.0)	2.6 (2.4, 2.9)	0.519
Cups per day (0–4)c	1.4 (1.3, 1.5)	1.4 (1.2, 1.6)	1.4 (1.3, 1.5)	0.891
Vegetables for snacks
Days previous week (0–7)	1.7 (1.5, 1.9)	1.7 (1.3, 2.2)	1.8 (1.5, 2.0)	0.917
Cups per day (0–4)c	1.0 (.9, 1.1)	1.1 (.9, 1.4)	1.1 (1.0, 1.3)	0.389
Water at meals
Days previous week (0–7)	4.2 (4.0, 4.5)	4.0 (3.6, 4.4)	4.3 (4.0, 4.5)	0.583
8-oz glasses per day (0–4)d	1.9 (1.8, 2.0)	1.9 (1.7, 2.1)	2.1 (1.9, 2.2)	0.172
Water with snacks and in between
How many days (0–7)	4.0 (3.8, 4.3)	3.6 (3.1, 4.0)	3.8 (3.6, 4.1)	0.222
8-oz glasses per day (0–4)d	1.8 (1.7, 2.0)	1.8 (1.5, 2.0)	1.9 (1.7, 2.0)	0.612
Sweetened beverages (SB) at meals
How many days (0–7)	3.6 (3.4, 3.8)a	3.1 (2.7, 3.5)a,b	3.0 (2.8, 3.2)b**	0.001
Size (0–4)e	1.7 (1.6, 1.8)a	1.5 (1.3, 1.7)a,b	1.4 (1.3, 1.5)b***	<0.001
SB with snacks
How many days (0–7)	3.9 (3.6, 4.1)a	3.4 (3.0, 3.9)a,b	3.2 (3.0, 3.5)b**	0.003
Size (0–4)e	1.8 (1.7, 1.9)a	1.7 (1.5, 1.9)a,b	1.6 (1.4, 1.7)b**	0.012
Packaged snacks
How many days (0–7)	3.5 (3.3, 3.7)a	3.1 (2.7, 3.4)a,b	3.1 (2.9, 3.3)b*	0.016
Per day (0–4)d	1.7 (1.6, 1.7)	1.5 (1.4, 1.7)	1.6 (1.5, 1.7)	0.393
Size (1–3)f	1.6 (1.6, 1.7)	1.5 (1.4, 1.6)	1.5 (1.5, 1.6)	0.171
Fast food
How many days (0–7)	1.7 (1.5, 1.9)	1.9 (1.6, 2.2)	1.5 (1.5, 1.8)	0.321
Size (1–4)g	2.0 (1.9, 2.1)a	1.9 (1.8, 2.1)a,b	1.8 (1.7, 1.9)b**	0.001
Value/combo meal (0–3)h	1.3 (1.2, 1.4)a	1.2 (1.0, 1.3)a,b	1.1 (1.0, 1.2)b*	0.047
Healthier option (0–3)h	1.4 (1.3, 1.5)	1.3 (1.1, 1.4)	1.5 (1.4, 1.6)	0.057
Physical activity
Purposely walking
How many days (0–7)	2.8 (2.6, 3.0)a	3.5 (3.1, 3.9)b**	3.3 (3.1, 3.5)b**	<0.001
Speed (0–3)i	1.5 (1.4, 1.6)	1.6 (1.5, 1.8)	1.6 (1.5, 1.7)	0.051
Purposeful stair climbing
How many days (0–7)	2.2 (2.0, 2.5)a	3.2 (2.7, 3.7)b**	3.0 (2.7, 3.3)b**	<0.001
How many flights (0–4)j	1.5 (1.4, 1.7)a	1.9 (1.7, 2.2)b*	1.9 (1.7, 2.0)b**	0.003
Screen time
How many days (0–7)	5.5 (5.3, 5.6)a	5.0 (4.7, 5.3)b*	4.9 (4.7, 5.1)b**	<0.001

Response options: ^c0 = 0, 1 = 1/2, 2 = 1, 3 = 2, 4 = >2. ^d0 = 0, 1 = 1, 2 = 2, 3 = 3, 4 = 4+. ^e0 = 0, 1 = <12 oz, 2 = 12 oz can, 3 = 20 oz bottle, 4 = >20 oz. ^f1 = small, 2 = medium, 3 = large. ^g1 = small, 2 = medium, 3 = large, 4 = x-large. ^h0 = never, 1 = rarely, 2 = sometimes, 3 = always. ⁱ0 = didn’t do, 1 = slow, 2 = medium, 3 = fast. ^j0 = 0, 1 = 1, 2 = 2–3, 3 = 4–5, 4 = 6+. ‘Teacher Implementation’ is a combination of ‘Faithfulness to the Curriculum’ and ‘Lesson Completion’. Bold text indicates P < 0.05. The Bonferroni adjustment was made for multiple comparisons. Letters (a, b) indicate significant differences among groups by post-hoc comparisons. *P < 0.05, **P < 0.01, ***P < 0.001.

For psychosocial variable outcomes, students in high ‘Teacher Implementation’ classes showed significantly higher scores for most variables, compared with control group (P < 0.05), while students in medium ‘Teacher Implementation’ classes showed significant improvement only in two variables: self-efficacy on drinking fewer sweetened beverages (P = 0.001) and doing more physical activity (P < 0.001; Table III).

Table III.

Psychosocial outcomes by the ‘Teacher Implementation’ in the ‘Choice, Control and Change’ project

Scale	Adjusted post mean (95% CI)			P value for Omnibus test
	Control (n = 337)	Medium ‘Teacher Implementation’ (n = 100)	High ‘Teacher Implementation’ (n = 266)
Intention to change (1–5c)
Total for all behaviors	3.2 (3.1, 3.3)a	3.4 (3.3, 3.6)a,b	3.5 (3.4, 3.6)b***	<0.001
Drinking less soda and other sweetened beverages	2.7 (2.6, 2.8)a	2.8 (2.6, 3.1)a,b	3.0 (2.9, 3.2)b**	0.004
Eat less frequently at fast-food restaurants	2.7 (2.6, 2.9)a	3.0 (2.8, 3.3)a,b	3.1 (2.9, 3.2)b**	0.003
Eat fewer packaged snacks	2.8 (2.6, 2.9)a	3.0 (2.8, 3.2)a,b	3.1, (2.9, 3.2)b**	0.009
Drink more water	3.8 (3.7, 3.9)	4.0 (3.8, 4.2)	4.0 (3.8, 4.1)	0.130
Eat more fruit and vegetables	3.4 (3.2, 3.5)a	3.5 (3.2, 3.7)a,b	3.6 (3.5, 3.8)b*	0.048
Do more physical activity	3.6 (3.5, 3.7)	3.8 (3.6, 4.0)	3.8 (3.7, 4.0)	0.042
Walk more	3.7 (3.5, 3.8)	3.9 (3.7, 4.2)	3.9 (3.7, 4.0)	0.042
Outcome expectations (1–5d)
Drinking lots of sweetened beverages	3.3 (3.2, 3.3)a	3.4 (3.3, 3.5)a,b	3.5 (3.5, 3.6)b***	<0.001
Eating frequently at the fast food restaurant	3.5 (3.4, 3.5)a	3.6 (3.4, 3.7)a,b	3.7 (3.7, 3.8)b***	<0.001
Eating lots of packaged snacks	3.4 (3.4, 3.5)a	3.5 (3.4, 3.7)a,b	3.7 (3.6, 3.8)b***	<0.001
Drinking plenty of water	3.8 (3.7, 3.8)a	3.9 (3.8, 4.1)a,b	3.9 (3.8, 4.0)b**	0.004
Eating lots of fruit and vegetables	3.8 (3.7, 3.9)a	4.0 (3.8, 4.1)a,b	4.0 (3.9, 4.1)b**	0.002
Walking	3.9 (3.8, 4.0)a	4.1 (3.9, 4.2)a,b	4.1 (4.0, 4.1)b**	0.002
Perceived barriers (1–5d)
Eating healthfully	3.3 (3.2, 3.4)	3.3 (3.2, 3.5)	3.4 (3.3, 3.5)	0.482
Being physically active	3.6 (3.5, 3.7)	3.7 (3.5, 3.9)	3.8 (3.7, 3.9)	0.111
Self-efficacy (1–4e)
Drinking fewer sweetened beverages	2.5 (2.4, 2.6)a	2.8 (2.6, 2.9)b*	2.7 (2.6, 2.8)b**	0.001
Eating less at the fast food restaurants	2.6 (2.5, 2.7)	2.7 (2.5, 2.9)	2.7 (2.6, 2.8)	0.076
Eating fewer packaged snacks	2.6 (2.5, 2.7)	2.7 (2.6, 2.9)	2.7 (2.6, 2.8)	0.097
Drinking lots of water	2.8 (2.7, 2.9)a	2.8 (2.7, 3.0)a,b	3.0 (2.9, 3.1)b*	0.029
Eating fruit and vegetables	2.6 (2.5, 2.7)	2.7 (2.6, 2.9)	2.7 (2.6, 2.8)	0.445
Walking and taking stairs	2.6 (2.5, 2.7)a	2.9 (2.7, 3.0)b**	2.9 (2.8, 3.0)b**	<0.001
Autonomous motivation (1–4e)
Eating
Competence	2.8 (2.7–2.8)a	3.0 (2.8–3.1)a,b	2.9 (2.9–3.0)b*	0.005
Autonomy	2.8 (2.7–2.8)a	2.9 (2.8–3.1)a,b	3.0 (2.9–3.0)b**	0.002
Physical activity
Competence	3.0 (2.9–3.0)a	3.1 (3.0–3.3)a,b	3.1 (3.0–3.2)b*	0.018
Autonomy	3.0 (2.9–3.0)	3.2 (3.0–3.3)	3.1 (3.0–3.2)	0.016

Response options: ^c1 = won’t do it within next 6 months, 2 = will try within the next 6 months, 3 = plan to do it in a month or so, 4 = currently doing it for past 1–6 months, 5 = have been doing it for over past 6 months. ^d1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, 5 = strongly agree. ^e1 = not sure, 2 = a little sure, 3 = somewhat sure, 4 = very sure. Higher scores indicate more desirable attitudes. ‘Teacher Implementation’ is a combination of ‘Faithfulness to the Curriculum’ and ‘Lesson Completion’. Bold text indicates P < 0.05. The Bonferroni adjustment was made for multiple comparisons. Letters (a, b) indicate significant differences among groups by post-hoc comparisons. *P < 0.05, **P < 0.01, ***P < 0.001.

The results from the HLM analyses showed that the ‘Teacher Implementation’ level predicted sweetened beverage behavior change (β = −0.04; P = 0.028) as well as intention to drink fewer sweetened beverages (β = 0.02; P = 0.038). This indicates that when ‘Teacher Implementation’ increased 1%, students reported decreases in sweetened beverages by 0.04 points on a 9-point scale and increases intention to change by 0.02 points on a 5-point scale. Similarly, there was a trend in decreasing eating at fast food restaurant behavior scores as the ‘Teacher Implementation’ increased, but it was only marginally statistically significant at the cut off significance level (β = −0.04; P = 0.052). ‘Teacher Implementation’ was not linearly associated with other behaviors or related psychosocial outcomes.

Additional HLM model showed that ‘Teacher Implementation’ also significantly predicted ‘Student Satisfaction’ rates (β = 0.03; P = 0.003; data not shown).

‘Student Reception’ influence on student outcomes

Table IV illustrates student outcome results based on the ‘Student Reception’ levels. When the ‘Student Reception’ level was divided into three groups (high, medium and control), the ANCOVA results showed that only high ‘Student Reception’ group showed significant improvements on sweetened beverage frequency at meals (P = 0.001) and with snacks (P = 0.003), packaged snack frequency (P = 0.016), and smaller sizes of sweetened beverages at meals (P < 0.001) with snacks (P = 0.012) and fast food (P = 0.001), compared with control group. For activity outcomes, both medium and high ‘Student Reception’ groups showed significant improvements in frequencies of purposely walking, stair climbing and reducing screen time, compared with control group (P < 0.01).

Table IV.

Behavioral outcomes by the ‘Student Reception’ level in the ‘Choice, Control and Change’ project

Scale	Adjusted post mean (95% CI)			P value for Omnibus test
	Control (n = 350)	Medium ‘Student Reception’ (n = 149)	High ‘Student Reception’ (n = 251)
Food choices
Fruit at meals
Days previous week (0–7)	3.5 (3.2, 3.7)	3.6 (3.3, 3.9)	3.7 (3.4, 4.0)	0.399
Pieces per day (0–4)c	2.0 (1.9, 2.1)	2.0 (1.9, 2.2)	2.0 (1.8, 2.1)	0.911
Fruit for snacks
Days previous week (0–7)	3.7 (3.4, 4.0)	3.3 (2.9, 3.7)	3.3 (3.0, 3.7)	0.113
Pieces per day (0–4)c	2.3 (2.1, 2.4)	2.1 (1.8, 2.3)	2.0 (1.8, 2.2)	0.065
Vegetables at meals
Days previous week (0–7)	2.5 (2.2, 2.7)	2.5 (2.2, 2.8)	2.7 (2.5, 3.0)	0.244
Cups per day (0–4)c	1.4 (1.3, 1.5)	1.4 (1.2, 1.6)	1.4 (1.3, 1.5)	0.891
Vegetables for snacks
Days previous week (0–7)	1.7 (1.5, 1.9)	1.7 (1.3, 2.2)	1.8 (1.5, 2.0)	0.917
Cups per day (0–4)c	1.0 (.9, 1.1)	1.1 (.9, 1.4)	1.1 (1.0, 1.3)	0.389
Water at meals
Days previous week (0–7)	4.2 (4.0, 4.5)	4.0 (3.6, 4.4)	4.3 (4.0, 4.5)	0.583
8-oz glasses per day (0–4)d	1.9 (1.8, 2.0)	1.9 (1.7, 2.1)	2.1 (1.9, 2.2)	0.172
Water with snacks and in between
Days previous week (0–7)	4.0 (3.8, 4.3)	3.6 (3.1, 4.0)	3.8 (3.6, 4.1)	0.222
8-oz glasses per day (0–4)d	1.8 (1.7, 2.0)	1.8 (1.5, 2.0)	1.9 (1.7, 2.0)	0.612
Sweetened beverages (SB) at meals
How many days (0–7)	3.6 (3.4, 3.8)a	3.1 (2.7, 3.5)a,b	3.0 (2.8, 3.2)b**	0.001
Size (0–4)e	1.7 (1.6, 1.8)a	1.5 (1.3, 1.7)a,b	1.4 (1.3, 1.5)b***	<0.001
SB with snacks
How many days (0–7)	3.9 (3.6, 4.1)a	3.4 (3.0, 3.9)a,b	3.2 (3.0, 3.5)b**	0.003
Size (0–4)e	1.8 (1.7, 1.9)a	1.7 (1.5, 1.9)a,b	1.6 (1.4, 1.7)b**	0.012
Packaged snacks
How many days (0–7)	3.5 (3.3, 3.7)a	3.1 (2.7, 3.4)a,b	3.1 (2.9, 3.3)b*	0.016
Per day (0–4)d	1.7 (1.6, 1.7)	1.5 (1.4, 1.7)	1.6 (1.5, 1.7)	0.393
Size (1–3)f	1.6 (1.6, 1.7)	1.5 (1.4, 1.6)	1.5 (1.5, 1.6)	0.171
Fast food
How many days (0–7)	1.7 (1.5, 1.9)	1.9 (1.6, 2.2)	1.5 (1.5, 1.8)	0.321
Size (1–4)g	2.0 (1.9, 2.1)a	1.9 (1.8, 2.1)a,b	1.8 (1.7, 1.9)b**	0.001
Value/combo meal (0–3)h	1.3 (1.2, 1.4)	1.2 (1.0, 1.3)	1.1 (1.0, 1.2)	0.047
Healthier option (0–3)h	1.4 (1.3, 1.5)	1.3 (1.1, 1.4)	1.5 (1.4, 1.6)	0.057
Physical activity
Purposely walking
How many days (0–7)	2.8 (2.6, 3.0)a	3.5 (3.0, 3.6)b**	3.4 (3.1, 3.6)b**	<0.001
Speed (0–3)i	1.5 (1.4, 1.6)	1.6 (1.5, 1.7)	1.6 (1.5, 1.7)	0.056
Purposeful stair climbing
How many days (0–7)	2.2 (2.0, 2.5)a	3.0 (2.6, 3.4)b**	3.0 (2.7, 3.4)b**	<0.001
How many flights (0–4)j	1.5 (1.4, 1.7)a	1.8 (1.6, 2.1)a,b	1.9 (1.7, 2.1)b**	0.003
Screen time
How many days (0–7)	5.4 (5.3, 5.6)a	4.7 (4.5, 5.0)b***	5.0 (4.8, 5.2)b**	<0.001

Response options: ^c0 = 0, 1 = 1/2, 2 = 1, 3 = 2, 4 = >2; ^d0 = 0, 1 = 1, 2 = 2, 3 = 3, 4 = 4+; ^e0 = 0, 1 = <12 oz, 2 = 12 oz can, 3 = 20-oz bottle, 4 = >20 oz; ^f1 = small, 2 = medium, 3 = large; ^g1 = small, 2 = medium, 3 = large, 4 = x-large; ^h0 = never, 1 = rarely, 2 = sometimes, 3 = always; ⁱ0 = didn’t do, 1 = slow, 2 = medium, 3 = fast; ^j0 = 0, 1 = 1, 2 = 2–3, 3 = 4–5, 4 = 6+. ‘Student Reception’ is a combination of ‘Student Engagement’ and ‘Classroom Management’. Bold text indicates P < 0.05. The Bonferroni adjustment was made for multiple comparisons. Letters (a, b) indicate significant differences among groups by post-hoc comparisons. *P < 0.05, **P < 0.01, ***P < 0.001.

Several psychosocial outcomes significantly improved in the medium ‘Student Reception’ group, compared with control (Table V). Those outcomes were overall intention to change behaviors (P < 0.001), outcome expectations in all six EBRB scales (P < 0.01), self-efficacy for walking and stair climbing, and eating and physical activity autonomy (P < 0.05). Some psychosocial outcomes only improved in high ‘Student Reception’ group, compared with control, including intention to change fast food, packaged snacks and water intake behaviors (P < 0.05), self-efficacy to drink fewer sweetened beverages (P < 0.01), and eating and physical activity competence (P < 0.05).

Table V.

Psychosocial outcomes by the ‘Student Reception’ in the ‘Choice, Control and Change’ project

Scale (score range and no. of items per scale)	Adjusted post mean (95% CI)			P value for Omnibus test
	Control (n = 337)	Medium ‘Student Reception’ (n = 100)	High ‘Student Reception’ (n = 266)
Intention to change (1–5c)
Total for all behaviors	3.2 (3.1–3.3)a	3.5 (3.4–3.6)b**	3.5 (3.4–3.6)b**	<0.001
Drinking less soda and other sweetened beverages	2.7 (2.6–2.8)a	3.1 (2.9–3.3)b**	2.9 (2.7–3.1)a,b	0.003
Eat less frequently at fast-food restaurants	2.7 (2.6–2.9)a	3.0 (2.8–3.2)a,b	3.1 (2.9–3.3)b**	0.003
Eat fewer packaged snacks	2.8 (2.6–2.9)a	3.0 (2.8–3.2)a,b	3.1 (2.9–3.2)b*	0.009
Drink more water	3.8 (3.7–3.9)a	3.9 (3.7–4.1)a,b	4.1 (3.9–4.2)b*	0.049
Eat more fruit and vegetables	3.4 (3.2–3.5)	3.6 (3.4–3.8)	3.6 (3.4–3.7)	0.085
Do more physical activity	3.6 (3.5–3.7)	3.9 (3.7–4.0)	3.8 (3.6–4.0)	0.036
Walk more	3.7 (3.5–3.8)	3.9 (3.7–4.1)	3.9 (3.7–4.0)	0.044
Outcome Expectations (1–5d)
Drinking lots of sweetened beverages	3.3 (3.2–3.3)a	3.5 (3.4–3.5)b**	3.5 (3.4–3.6)b***	<0.001
Eating frequently at the fast food restaurant	3.5 (3.4–3.5)a	3.7 (3.6–3.8)b**	3.7 (3.6–3.8)b**	<0.001
Eating lots of packaged snacks	3.4 (3.4–3.5)a	3.7 (3.6–3.8)b***	3.7 (3.6–3.7)b***	<0.001
Drinking plenty of water	3.8 (3.7–3.8)a	3.9 (3.8–4.0)b*	3.9 (3.8–4.0)b*	0.004
Eating lots of fruit and vegetables	3.8 (3.7–3.9)a	4.0 (3.9–4.1)b*	4.0 (3.9–4.1)b**	0.002
Walking	3.9 (3.8–4.0)	4.0 (3.9–4.1)b*	4.1 (4.0–4.2)b**	0.002
Perceived Barriers (1–5d)
Eating healthfully	3.3 (3.2–3.4)	3.4 (3.2–3.5)	3.4 (3.3–3.5)	0.642
Being physically active	3.6 (3.5–3.7)	3.8 (3.7–4.0)	3.7 (3.6–3.9)	0.098
Self-efficacy (1–4e)
Drinking fewer sweetened beverages	2.5 (2.4–2.6)a	2.7 (2.6–2.8)a,b	2.8 (2.6–2.9)b**	0.001
Eating less at the fast food restaurants	2.6 (2.5–2.7)	2.7 (2.6–2.8)	2.7 (2.6–2.8)	0.073
Eating fewer packaged snacks	2.6 (2.5–2.7)	2.7 (2.6–2.8)	2.8 (2.7–2.9)	0.072
Drinking lots of water	2.8 (2.7–2.9)	2.9 (2.8–3.1)	3.0 (2.9–3.1)	0.109
Eating fruit and vegetables	2.6 (2.5–2.7)	2.7 (2.6–2.9)	2.7 (2.6–2.9)	0.447
Walking and taking stairs	2.6 (2.5–2.7)a	2.9 (2.7–3.0)b**	2.9 (2.8–3.0)b**	<0.001
Autonomous motivation (1–4e)
Eating
Competence	2.8 (2.7–2.8)a	2.9 (2.8–3.0)a,b	3.0 (2.9–3.1)b**	0.003
Autonomy	2.8 (2.7–2.8)a	2.9 (2.8–3.0)b*	3.0 (2.9–3.1)b**	0.002
Physical activity
Competence	3.0 (2.9–3.0)a	3.1 (3.0–3.2)a,b	3.1 (3.0–3.3)b*	0.016
Autonomy	3.0 (2.9–3.0)a	3.1 (3.0–3.3)b*	3.1 (3.0–3.2)a,b	0.016

Response options: ^c1 = won’t do it within next 6 months, 2 = will try within the next 6 months, 3 = plan to do it in a month or so, 4 = currently doing it for past 1–6 months, 5 = have been doing it for over past 6 months. ^d1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, 5 = strongly agree. ^e1 = not sure, 2 = a little sure, 3 = somewhat sure, 4 = very sure. Higher scores indicate more desirable attitudes. ‘Student Reception’ is a combination of ‘Student Engagement’ and ‘Classroom Management’. Bold text indicates P < 0.05. The Bonferroni adjustment was made for multiple comparisons. Letters (a, b) indicate significant differences among groups by post-hoc comparisons. *P < 0.05, **P < 0.01, ***P < 0.001.

The results from the HLM analyses showed that the level of ‘Student Reception’ significantly predicted the sweetened beverage behavioral outcome, indicating that when ‘Student Reception’ increased by 1%, the average sweetened beverages behavior scores decreased by 0.02 points on a 9-point scale (β = −0.02; P < 0.05). ‘Student Reception’ was not linearly associated with other behaviors or related psychosocial outcomes.

Additional HLM model showed that Student Reception was also significantly associated with ‘Student Satisfaction’ rates (β = 0.01; P = 0.001; data not shown).

‘Student Satisfaction’ influence on student outcomes

Results from a series of multiple regression analysis showed that ‘Student Satisfaction’ was a significant predictor for student outcomes (Table VI). Higher ‘Student Satisfaction’ rate was strongly associated with desirable student outcomes: improving outcome expectation and self-efficacy for eating more fruit and vegetables (P < 0.001), outcome expectation for drinking more water (P = 0.001), self-efficacy for reducing sweetened beverage intake (P = 0.001), improving autonomous motivation (competence and autonomy) on healthy eating (P < 0.001), and increasing physical activity (P = 0.001) and improving intention to walk more (P < 0.001), intention to increase overall physical activity (P = 0.002) and outcome expectation for physical activity (P < 0.001). There was no significant association between ‘Student Satisfaction’ and packaged snack behavior or related psychosocial variables (data not shown).

Table VI.

Associations between ‘Student Satisfaction’ and student outcomes of the ‘Choice, Control and Change project’

Dependent variablesa		βb	SE	t-ratio	P value
Fruit and vegetables-related variables
Behavior	Eating fruit and vegetablesc	0.24	0.11	2.18	0.031
Mediating variables	Intention to changed	0.19	0.08	2.45	0.015
	Outcome expectatione	0.19	0.04	8.46	<0.001***
	Self-efficacyf	0.23	0.05	4.34	<0.001***
Water related variables
Behavior	Drinking waterc	0.02	0.16	0.14	0.890
Mediating variables	Intention to changed	0.20	0.08	2.60	0.01
	Outcome expectatione	0.14	0.04	3.40	0.001**
	Self-efficacyf	0.07	0.05	1.45	0.147
Sweetened beverage-related variables
Behavior	Drinking sweetened beveragec	−0.26	0.13	−2.03	0.044
Mediating variables	Intention to changed	0.22	0.08	2.69	0.008
	Outcome expectatione	0.09	0.04	2.32	0.021
	Self-efficacyf	0.16	0.05	3.25	0.001**
Fast food restaurants-related variables
Behavior	Eating at fast food restaurantsc	−0.28	0.10	−2.81	0.005
Mediating variables	Intention to changed	0.25	0.09	2.85	0.005
	Outcome expectatione	0.11	0.04	2.46	0.014
	Self-efficacyf	0.15	0.05	2.78	0.006
Overall eating-related mediating variables
Perceived barrierse		0.09	0.05	1.72	0.087
Autonomous motivation on eatingf		0.17	0.05	3.82	<0.001***
Competence on eatingf		0.18	0.05	3.63	<0.001***
Autonomy on eatingf		0.17	0.05	3.69	<0.001***
Physical activity-related variables
Behavior	Physical activity behaviorc	0.42	0.13	3.37	0.001**
Mediating variables	Intention to walk mored	0.27	0.08	3.54	<0.001***
	Intention to increase overall physical activityd	0.23	0.07	3.19	0.002**
	Outcome expectatione	0.15	0.04	3.52	<0.001***
	Perceived barriers on physical activitye	0.05	0.06	0.73	0.465
	Self-efficacy on walkingf	0.11	0.5	2.35	0.020
	Autonomous motivation on physical activityf	0.08	0.04	1.88	0.061
	Competence on physical activityf	0.06	0.05	1.19	0.237
	Autonomy on physical activityf	0.11	0.05	2.39	0.017

^aDependent variables are post-test scores; ^b‘Student Satisfaction’ was included as a predictor, and gender and pre-test scores were included as covariates in the model. ^cScale: 1–9 point system: 1 = lowest consumption to 9 = highest consumption. ^dIntention to change response options are based on stage of changes: 1 = won’t do it within next 6 months, 2 = will try within the next 6 months, 3 = plan to do it in a month or so, 3 = currently doing it for past 1–6 months, 4 = have been doing it for over past 6 months. ^eResponse options: 1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, 5 = strongly agree. ^fResponse options: 1 = not sure, 2 = a little sure, 3 = somewhat sure, 4 = very sure.

Bold text indicates P < 0.05. Adjustment for multiple comparisons were manually done with Bonferroni method (adjusted significance level α = 0.002). **P ≤ 0.002; ***P < 0.001.

Discussion

Group comparisons and linear relationship investigations in this study confirmed that implementation process influenced study outcomes in a field-based nutrition intervention study. Our findings suggest that, compared with the control group, only a high ‘Teacher Implementation’ level positively impacted students’ sweetened beverages, packaged snacks and fast food behaviors in a middle school obesity prevention curriculum, ‘Choice, Control and Change’. For physical activity behavior, both medium and high ‘Teacher Implementation’ groups showed significant improvement in purposely walking and stair climbing and in reducing screen time, compared with the control group. There were similar findings in terms of theory-based psychosocial mediating variables: more psychosocial indicators improved when the ‘Teacher Implementation’ level was high. In addition to the level of teachers’ ability to faithfully implement an adequate dose of lessons, how well students were engaged in the curriculum activities and how much behavioral issues were present and managed in classroom (‘Student Reception’) influenced students’ EBRB and psychosocial outcomes in ‘Choice, Control and Change’.

In summary, the sequence in the systematic model described in Figure 1 captures well the results of this study. Higher ‘Teacher Implementation’ and ‘Student Reception of Choice, Control and Change’ means better student EBRB and psychosocial outcomes. ‘Teacher Implementation’ and ‘Student Reception’ components were significantly associated with ‘Student Satisfaction’, which were also strongly associated with student outcomes. These relationships indicate that how teachers implemented and how students received the curriculum in class affected ‘Student Satisfaction’, which in turn influenced the students’ psychosocial and behavioral variable changes. Additionally, our previous study indicated that ‘Teachers’ Curriculum Evaluation’ and ‘Teacher Satisfaction with Teaching the Curriculum’ were also significantly associated with ‘Student Satisfaction’ [22]. Therefore, ensuring ‘Teacher Implementation’ and ‘Student Reception’ in class and supporting teachers throughout the intervention period (e.g. troubleshooting any barriers to implement the curriculum) are all important to maintain both teacher and student satisfaction levels high, which then are likely to improve students’ psychosocial and behavioral outcomes.

Because process evaluation has become recognized as an important part of field-based intervention studies, more studies have described various implementation process components [9, 11, 13–16]. Yet, few studies used implementation data in the analysis of the primary study outcomes [10, 18–20]. Doing so may not only prevent a ‘black box’ evaluation of intervention studies but also contribute to a deeper understanding of the implementation process and its sequential relationship to outcome variables.

Among the few, CATCH and LEAP studies comprehensively measured process components and linked them to outcomes. The results of both studies showed that implementation influenced study outcomes. Slightly different from our findings, CATCH curriculum evaluation showed that when teachers modified the curriculum content, students changed their self-efficacy and knowledge in more desirable directions, whereas this study results showed that there was a positive association between ‘Teacher Implementation’, including ‘Faithfulness to the Curriculum’, and student outcomes. Further research is warranted to investigate whether modifying the curriculum makes teachers more confident teaching the curriculum, therefore positively influencing student outcomes; or whether no matter how teachers feel, being faithful to the curriculum has greater impact on student outcomes.

More similar to the results of the current study, LEAP, a physical activity intervention study for high school girls, showed that girls in the high-implementation schools had a higher rate of participation in vigorous physical activity compared with those in control schools. The study also reported that there was a significant linear dose-response in proportion to participation in vigorous physical activity. Similarly, this study had a significant linear dose-response with sweetened beverage outcomes; a higher implementation of ‘Choice, Control and Change’ influenced students to drink fewer sweetened beverages.

A strength of this study is that the classroom process components were objectively assessed by observers with adequate inter-rater reliability [22] using a systematic conceptual model. This conceptual model and analysis approach provided us a better understanding about the intervention pathway.

A limitation of the study is that there were a small number of schools, which caused the study to utilize a small sample size through which to examine school context variance components on student outcomes. Using self-reported data for student outcomes was also a limitation. In addition, even though the study findings showed that the degree of ‘Student Satisfaction’ was the strongest predictor of student outcomes, indicating that intervention efforts should focus on maximizing ‘Student Satisfaction’, it is still not clear why certain behaviors in the program changed more than other behaviors. It is likely that unique motivational factors for each behavioral change may play an important role when the participants apply their knowledge and skills to their lives outside the classroom. It might be also helpful to use qualitative methods such as student interviews to investigate more in depth relationships among these variables or unknown factors.

Because field-based nutrition and physical activity interventions are complex and each study may have its own challenges, systematically documenting the implementation process and examining its influences on outcomes are keys to improving school-based interventions. This study provides an example of how a systematic process evaluation model can be developed and applied to investigate the link between process and outcome evaluations. ‘Teacher Implementation’, ‘Student Reception’ and ‘Student Satisfaction’ or equivalent process components need to be ensured during the intervention period, to maximize the effectiveness of the study. Continued investigation of the link between process evaluation data and outcome data is needed to understand further how nutrition and physical activity-related interventions work and to maximize their effectiveness and benefits. Documenting consistent implementation process components across studies and providing clearly articulated conceptual maps linking these process components to outcomes will catalyse the growth of this research area and will contribute to determining how an intervention works.

Funding

The research was supported by R25 RR20412, Science Education Partnership Award, National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH).

Conflict of interest statement

None declared.