RE-AIM analysis of a randomized school-based nutrition intervention among fourth-grade classrooms in California

Abstract

Childhood overweight and obesity are major health problems. School-based programs enable intervening with large groups of children, but program overall health impact is rarely completely assessed. A RE-AIM (Reach, Efficacy, Adoption, Implementation, Maintenance) analysis tested the overall public health impact of the fourth-grade “Nutrition Pathfinders” school-based nutrition-education program. A randomized controlled trial in 47 fourth-grade California classrooms (1713 students) tested program efficacy, and a secondary analysis of archival data tested program dissemination. Desired effects were seen in child nutrition knowledge, attitudes, consumption of low-nutrient high-density foods, sugar-sweetened beverages, proteins, grains, and parent willingness to serve new foods. The program was disseminated to ∼25 % of public school fourth-grade classrooms in California and cost about $1.00 per student to implement. The Nutrition Pathfinders program demonstrates potential for moderate to high public health impact due to its wide dissemination, effectiveness in altering attitudes and behaviors, and its relatively inexpensive cost of implementation.

Weight-related diseases are among the most costly health issues facing societies today [1, 2]. Childhood overweight and obesity are related to numerous health problems later in life [3–5], and obese children are more likely to be obese in adulthood [6]. Although the rise in childhood obesity rates has plateaued recently, 17 % of children ages 2–19 years are obese and another 15 % are overweight [7]. A common method for preventing and treating unhealthy weight gain in children is through school-based nutrition and physical activity interventions [8].

Schools have been recognized as an important context for providing nutrition education because of the relative ease and cost-effectiveness of integrating nutrition education into pre-existing curriculum [9–13]. Recent meta-analyses have found that school-based interventions can effectively alter nutritional intake, as well as biological and anthropometric measurements of overweight and obesity [14, 15]. A recent review reported the most common components of school-based interventions were classrooms activities, parental involvement, and food provisions provided by the school, and that interventions containing at least two of these features were typically effective at altering target outcomes (10 out of 12 studies) [16]. However, some evidence also suggests that school-based interventions are more effective at treating obesity compared to preventing it [17]. Furthermore, studies suggest that school-based interventions can have broader influences on home environments when they include components aimed to change parents’ knowledge, attitudes, and practices regarding children’s nutrition [18, 19]. Changing parent behavior may benefit children’s dietary intake and obesity risk, as a majority of children’s meals are eaten or prepared at home [20], although school-provided breakfast is becoming more common in low-income school districts in California [21].

Another benefit of school-based interventions is the opportunity to disseminate information to a large number of children [9, 10]. Dissemination is often overlooked when evaluating the overall public health impact of interventions [22, 23]. For instance, a health program exhibiting a large effect on the target outcome may have a small public health impact if it is too costly to implement or fails to reach enough people. On the other hand, even modest effect sizes can be meaningful if the intervention reaches a large number of people [22, 23]. This point is particularly important for effects that are compounded over time, such as daily food intake [24].

The RE-AIM model is useful for assessing the overall health impact of interventions [25–27]. RE-AIM is an acronym for five evaluation dimensions: reach (the number or proportion of the target population that participated), efficacy (the ability to change desired outcomes), adoption (the number or proportion of desired contexts involved), implementation (how well the intervention was delivered as designed), and maintenance (how well program effects are maintained and the continued availability of the program). Viable interventions should reach a large audience across numerous settings, be relatively easy and cost-effective to implement, and should be sustainable over time.

Only a few school-based nutrition education interventions have been evaluated using the RE-AIM framework to date [28, 29], one of which was the “Shaping Up My Choices” program [30]. The Shaping Up My Choices program was a school-based nutrition intervention delivered by teachers in third-grade classrooms in California. The program had positive effects on nutrition knowledge, self-efficacy, outcome expectations, and some dietary intake variables. The program also had impressive reach, adoption, and implementation. However, less support was found for the maintenance of the program because some of the effects failed to last through a 3-month follow-up period. The present study uses the RE-AIM framework to evaluate the Nutrition Pathfinders program, a more developmentally advanced version of the topics presented in the Shaping Up My Choices program, designed for fourth-grade students.

MATERIALS AND METHODS

Intervention

The Nutrition Pathfinders program was developed by Dairy Council of California to promote healthy eating behaviors and attitudes in children (see http://www.HealthyEating.org/NP4). The Nutrition Pathfinders program consists of seven lessons delivered by classroom teachers over a 4- to 10-week time period. The program includes a teacher’s guide with instructions and lesson plans, a student’s workbook, and family homework for particular lessons. The curriculum was based on an integration of the health belief model and social cognitive theory [31–34]. The Nutrition Pathfinders program is similar to the Shaping Up My Choices program, but includes more critical thinking components to match the increased age of participants. The program utilizes interactive activities, simulations, and reflections to teach students: (a) the five food groups and main nutrients; (b) the importance of eating balanced breakfast, lunch, dinner, and snacks; (c) how to read food labels, with a focus on understanding fat content; (d) how to estimate appropriate serving sizes using hand symbols; (e) understanding the recommended number of serving sizes per day for each food group; (f) how to be physically active for 60 min each day, and how physical activity relates to food intake; and (g) how to analyze food records and set goals for themselves to improve. In order to improve compliance, typical standards for compliance were expressed to teachers and teachers received a $25.00 bonus for returning at least 80 % of the surveys. The curriculum aligns with California’s Common Core Content Standards and the California and National Health Education Standards [35, 36]. Dairy Council of California provides the Nutrition Pathfinders materials free of charge to schools within the state of California and makes them available for purchase by educators in other states.

Design

The efficacy of the program was tested through a randomized controlled trial with pre-, post-, and follow-up assessments. Primary outcomes tested were changes in children’s nutrition knowledge, attitudes (nutrition self-efficacy and outcome expectations), and dietary intake, as well as parents’ attitudes and behavior. Post-surveys were completed immediately after the conclusion of the program (i.e., 10 weeks after pre-surveys), and follow-up surveys were completed 3 months later. Program implementation was assessed with in-person classroom observations in the intervention group classrooms and teacher and parent surveys. Variables assessed were the following: the percentage of teachers completing the lessons, the degree to which teachers followed the Teacher’s Guide, the extent to which students were cooperative and participated during the lessons, and the percentage of students completing the assigned homework (and whether they completed it with a parent). Program reach, adoption, and maintenance were assessed by a secondary analysis of the Dairy Council of California, with information on all fourth-grade classrooms that order the program each year. Variables assessed included the following: the number and percent of fourth-grade students and classrooms from public schools in California that received the materials, the percent of teachers that ordered materials for the first time, and the percent of teachers that re-ordered materials the following year.

Sample

The present study included two samples: a dissemination sample (to evaluate reach, adoption, and maintenance) and a program evaluation sample (to assess efficacy and implementation). The dissemination sample consisted of all 4821 fourth-grade classrooms (152,065 students) that ordered the Nutrition Pathfinders materials for the 2011–2012 school year. The program evaluation sample of 47 classrooms was recruited from schools listed in the Dairy Council of California’s database as eligible to teach nutrition. Post hoc power analyses suggested that this number was chosen due to practical constraints, including budget and feasibility. Post hoc power analyses suggested that this number of classrooms had ∼80 % power (α = 0.05, two-tail) to detect small to medium effects (d = 0.23–0.28). Teachers were recruited primarily through email, in-person, and telephone conversations with members of Dairy Council of California. The majority of recruitment took place during the spring of the previous year (∼70 %), and some additional classrooms were recruited at the end of summer to replace classrooms that dropped out of the study prior to starting. Classrooms were eligible to take part in the study if they (a) did not teach a combination classroom (i.e., fourth-grade students only) and (b) they did not teach any other nutrition information apart from the Nutrition Pathfinders program. School locations were spread across the state of California. Classrooms were randomly assigned via coin-flip to either the intervention (n = 27 classrooms) or control groups (n = 20 classrooms). The participant flow of the evaluation sample is shown in Fig. 1. Twenty-two schools had multiple classrooms participating, 19 of which included at least one intervention and one control classroom to make the groups relatively similar. The study was conducted in accordance with the guidelines specified in the Declaration of Helsinki, and all procedures involving human subjects were approved by the ethics committee of the Institutional Review Board at Independent Review Consulting Inc. Written informed assent was obtained from all minors. A passive parental consent procedure was used. The minor was approached for assent if the parent did not decline consent.

Fig 1.

Flow diagram of classrooms and students through the recruitment, screening, and assessment phases of the evaluation study

The evaluation sample had a higher proportion of Asian (15 vs 9 %) and White students (32 vs 27 %), a lower proportion of Hispanic students (38 vs 51 %), and about the same proportion of African American students (8 vs 7 %) as compared with fourth-grade students attending public elementary schools across the state. Twenty-one percent of the students in the convenience sample were English learner students (i.e., not yet proficient in English), compared to 17 % of students throughout the state.

School-level socioeconomic status (SES) was defined by the percentage of students who were enrolled in free-or-reduced-price lunch programs. The school-level cutoff points for low, mid, and high SES were more than 75 % (low SES), between 75–25 % (mid SES), and below 25 % (high SES) of students enrolled in free-or-reduced-price lunch programs. The low SES category had 447 students, mid SES had 717 students, and high SES had 549 students. The evaluation sample had slightly fewer students enrolled in free-or-reduced-price lunch programs compared to the state (47 vs 56 %), indicating the sample had a slightly higher level of SES. Data on SES for the dissemination sample are unavailable, but this sample was sufficiently large that it is expected to be similar to that of the state of California.

Procedure and measures

Student survey

The student survey was administered at pre-, post-, and follow-up time points. In addition to demographic questions (gender and age), the survey measured student self-reported dietary intake, attitudes towards nutrition (self-efficacy and outcome expectations), and nutrition knowledge. A modified version of the elementary-level School Physical Activity and Nutrition Questionnaire (SPAN) was used to measure dietary intake. The SPAN questionnaire contains 22 items measuring a variety of foods and has been previously validated in fourth-grade classrooms [37, 38]. Students reported the number of each food item they consumed on the previous day as either “none,” “1 time,” “2 times,” or “3 or more times.” The present report utilizes summed composite scores for low-nutrient high-density foods (french fries/snack chips, punch/sports drinks, regular/diet soda, baked desserts, and candy), sugar-sweetened beverages (regular/diet soda and punch/sports drinks), proteins (meats, eggs, peanut butter, fish, and beans), dairy (milk, cheese, yogurt, and milk on cereal), grains (cereal, pasta, white bread, and dark bread), vegetables, and fruits (fruits and 100 % fruit juice). White bread and dark bread, part of the grains composite score, were also analyzed individually to test for differences across groups in changes on these subscales. Additionally, a previously validated latent variable referred to as healthy foods that combines the milk, fruit, 100 % fruit juice, and vegetables subscales was used [39].

Nutrition self-efficacy was measured with four items designed to map onto lessons taught by the program (e.g., I can eat breakfast every day, even if I am in a hurry). There was a four-point response scale with options “I can,” “Maybe I can,” “Maybe I can’t,” and “I can’t.” Nutrition outcome expectations were measured with six items also designed to map onto lessons taught by the intervention (e.g., I think that skipping a meal will make me feel tired). There was a four-point response scale with options “Yes,” “Maybe Yes,” “Maybe No,” and “No.” All items for self-efficacy and outcome expectations were coded so that high values correspond to preferred outcomes (i.e., high values correspond to higher self-efficacy and more positive outcome expectations). Mean values were calculated for both scales. Pilot tests in third-grade students (n = 57) indicated the two scales had adequate 7-day test-retest reliability with an r = 0.75 for outcome expectations and r = 0.57 for self-efficacy. The average Cronbach’s alphas for the self-efficacy and outcome expectation scales in the present study were 0.591 and 0.557, respectively. Although validity of the scales were not formally tested in the present evaluation, previous studies have shown evidence for their predictive validity by evaluating correlations of self-efficacy and outcome expectations with measures of nutrition knowledge and food intake in third graders. Both scales were positively correlated with nutrition knowledge and healthy food variables (e.g., fruits and vegetables) and negatively correlated with unhealthy food variables (e.g., junk foods and soda) [40].

Finally, the student survey measured six forms of nutrition knowledge, including the following: food group knowledge (6 items), main nutrient knowledge (3 items), main nutrient function knowledge (3 items), knowledge of balanced breakfast choices (2 items), healthy snack choices (2 items), and knowledge of balanced dinner choices (1 item). The 17 nutrition knowledge questions were designed by the research staff to map onto intervention lessons. The percent of correct responses for each group of questions was calculated and used in analyses.

Parent survey

The parent survey was developed by the researchers in order to map onto components of the intervention. It contained approximately 40 questions and was administered at pre and post only. Parents had the option of completing Spanish or English versions of the survey. Parent-reported perceptions of child interest in the main food groups was measured with five items, one item each for milk, grains, fruits, vegetables, and proteins, with answer options “Not interested,” “Somewhat Interested,” and “Very Interested.” Parents’ interest in the main food groups was measured with the same five items. The frequency that parents prepare balanced dinners was measured by a single item with response options “Every day,” “5–6 times each week,” “3–4 times each week,” and “2 or fewer times each week.” How often the child asks for foods was measured with six items, one item each for milk, vegetables, healthier food for snacks, breakfast every morning, healthier food for meals, and low-nutrient high-density foods, with answer options “Never,” “Not often,” “Sometimes,” and “Most of the time.” Whether or not the child has asked for any new foods in the last 2 months was measured by a single item with response options “Yes” and “No.” Parent reports of child behaviors related to nutrition were measured with eight items (e.g., talking about food, asking to buy foods belonging to certain food groups, trying different foods, talking about serving sizes, noticing if a meal is balanced, talking about food labels, choosing healthy foods over low-nutrient high-density foods, and cooking at home) with response options “Never,” “Not often,” “Sometimes,” and “Most of the time.” Lastly, whether parents had been serving different amounts of foods recently was measured by six items, one item each for milk, vegetables, fruits, grains, proteins, and low-nutrient high-density foods, with response options “Less,” “Same Amount,” and “More.” All items were coded so that higher scores correspond to more desired outcomes (e.g., greater serving of healthy food groups and lower serving of unhealthy foods). Scores were summed for each construct, and difference scores were calculated and used in analyses for all constructs except for parents serving different amounts of foods because it was only measured at post-intervention. Average Cronbach’s alphas for each construct ranged from 0.492–0.752. Finally, demographic questions assessed the parent’s relationship to the child (mother, father, grandparent, older sibling, or other) and their child’s age and gender. Anyone who completed the parent survey is referred to as parents of children for simplification purposes, even though some participants held different relationships to the child (e.g., grandparents, older siblings).

Teacher survey

The teacher survey was completed by teachers in the intervention group at the conclusion of the program. Teachers reported whether they completed each lesson, whether they completed each lesson in one or more sessions, whether they added material of their own to the lessons, and whether they would change the lesson the next time they taught it using a “yes” or “no” response scale. Teachers reported the approximate length of each lesson and the length of preparation required for each lesson in minutes. The teacher survey also measured the ease of use of the lessons (using a four-point scale with anchors “Very difficult” and “Very easy”), how much of the material was presented by the teachers (using a six-point scale with anchors “None or almost none of the material” and “All of the material”), whether the teacher’s objectives were met by the lessons (using a five-point scale with anchors “Not at all” and “Very much”), how cooperative their students were during the lessons (using a four-point scale with anchors “Very uncooperative and unruly” and “Very cooperative and well-behaved”), how well their students participated when required (using a four-point scale with anchors “Less than 25 % of students participating” and “76 % to 100 % of students participating”), and the overall interest of their students (using a five-point scale with anchors “Not at all” and “Very much”). Finally, the survey also measured various qualitative aspects of the program, including the degree the teacher’s students and their parents liked or disliked the family homework assignments, and teacher satisfaction with the teacher guide, student workbook, family homework, and master document program components.

Classroom observations

Trained research staff observed a subset of the program lessons during October and November of 2011. Observers received formal training on how to use the classroom observation form, including practice rating a videotaped sample lesson. The classroom observation instrument was based on items used to evaluate the Midwestern Prevention Project (MPP), a school-based drug abuse prevention program [41]. Each lesson was divided into five parts: anticipatory set, vocabulary, step-by-step, check for understanding, and guided practice. Observers rated teachers on whether they presented each activity and whether teachers changed or added any material to the lessons using a “Yes” or “No” response scale. Finally, observers also rated student engagement (using a three-point scale with answer options “Few/none participated,” “Some participated,” and “Many participated”).

Data analysis

An “intention to treat” analysis was conducted. Descriptive statistics were computed in IBM SPSS Statistics version 19 (IBM Corp, Armonk, NY). Multilevel modelling, controlling for students clustered within classrooms, was used to evaluate group differences in changes of knowledge, attitudes, and dietary intake, and for changes in parent survey constructs. Multilevel models were tested in R version 2.15.1 (open source software) using the nlme package (Pinheiro, Bates, DebRoy, Sarkar, & R Core Team, 2014) with maximum likelihood estimation. The current paper tested for intervention versus control group differences in change scores (from pre to post and pre to follow-up) for each of the student survey constructs and for pre to post changes in parent survey constructs (except for parents serving different amounts of foods, which was only measured at post). Because the outcomes were change scores, analyses controlled for the pre-survey scores for each dependent variable. Additionally, student gender and school-level SES were included as covariates in all analyses. Listwise deletion for missing data was necessary to compute change scores.

RESULTS

Program efficacy

Student survey

Student surveys were collected from 1713 children (n = 971 intervention). There were 1221 pre-surveys completed (n = 696 intervention), 1237 post-surveys completed (n = 723 intervention), and 1248 follow-up surveys completed (n = 714 intervention). Pre- and post-surveys were successfully matched for 921 students (n = 543 intervention), 931 had their pre- and follow-up surveys matched (n = 532 intervention), and 804 had all three surveys successfully matched (n = 478 intervention). The sample was 52 % boys and had an average age of 9.00 years (sd = 0.43).

Analysis of missing data

Dummy-coded variables for missingness were created for each time point by coding students missing surveys as 1 and students completing surveys as 0. Chi-square tests and logistic regression were used to predict which students were missing at pre, post, and follow-up (p < .05). High SES students were less likely to be missing at all three time points, while mid SES students were more likely to be missing at all three time points, and low SES students were more likely to be missing at post only. Additionally, the control group was more likely to be missing at post. Finally, students scoring higher on balanced breakfast knowledge and reporting that they consumed more low-nutrient high-density foods at pre were more likely to be missing at post, while no dependent variables at pre significantly predicted which students were missing at follow-up.

Group differences prior to intervention

Independent samples t tests and chi-square tests of independence were used to test for differences in groups at pre (p < .05). There were no differences in gender or age of the two groups; however, the intervention group had significantly more students in the high SES category and fewer students in the mid SES category. Additionally, the intervention group reported consuming significantly less protein and scored significantly higher on food group knowledge at pre-intervention. There were no pre-survey differences in either nutrition self-efficacy or outcome expectations. Because the groups differed in SES, and because SES significantly predicted missingness at all three time points, SES was included as a covariate in all subsequent analyses.

Group differences in intervention outcomes

Table 1 shows pre- to post-survey changes for each group of students on knowledge, attitudes, and dietary intake. The intervention group showed significantly greater increases in all knowledge variables at the p < .001 level, except for knowledge of balanced breakfast choices, which the intervention group exhibited a marginally significant increase compared to the control group (p < .10). The intervention group also had significantly larger increases in both nutrition self-efficacy and nutrition outcome expectations at the p < .05 and p < .01 level, respectively. Finally, the intervention group significantly reduced consumption of low-nutrient high-density foods and sugar-sweetened beverages (p < .05) compared to the control group, while simultaneously marginally increasing consumption of proteins and white bread (p < .10). Gender by treatment-group interactions were also explored for the dietary consumption variables. Figure 2 shows a significant interaction for proteins (p < .05), in which girls in the intervention group significantly increased their consumption compared to girls in the control group (p < .001), whereas this effect was not seen in boys (p = .849).

Table 1.

Differences by study group for pre- to post-survey changes in nutrition knowledge, self-efficacy, outcome expectancies, and dietary intake observed in the “Nutrition Pathfinders” program evaluation among fourth-grade children in California (n = 921)

	Intervention				Control				Group comparison for mean change (b) a
	Pre mean	Pre SD	Change mean	Change SD	Pre mean	Pre SD	Change mean	Change SD
Nutrition knowledgeb
Food groups	0.52	0.19	0.18	0.22	0.49	0.20	0.05	0.21	.157***
Main nutrients	0.34	0.30	0.24	0.43	0.33	0.30	−0.03	0.37	.287***
Nutrient functions	0.22	0.26	0.39	0.41	0.20	0.24	0.05	0.33	.253***
Breakfast choices	0.27	0.31	0.06	0.41	0.25	0.31	0.01	0.38	.057(*)
Snack choices	0.72	0.33	0.14	0.31	0.69	0.32	0.02	0.31	.130***
Dinner choices	0.17	0.38	0.21	0.58	0.14	0.35	−0.03	0.41	.256***
Attitudesc
Self-efficacy	3.17	0.70	0.09	0.59	3.16	0.67	−0.04	0.59	.161*
Outcome expectations	3.07	0.60	0.18	0.52	3.05	0.56	0.01	0.51	.203**
Dietary intaked
Low-nutrient high-densitye	3.48	2.95	−0.40	2.76	3.78	3.45	−0.08	2.63	−.338*
Sugar-sweetened beveragesf	1.35	1.43	−0.31	1.36	1.45	1.57	−0.13	1.35	−.164*
Proteing	2.59	1.87	0.28	2.14	2.83	2.17	−0.15	2.07	.299(*)
Dairyh	4.44	1.91	0.07	2.33	4.31	1.94	0.02	1.97	.028
Grainsi	2.75	1.81	0.15	2.13	2.69	1.93	−0.10	2.11	.238
White Bread	0.93	0.88	0.15	1.14	0.88	0.92	0.01	1.12	.130(*)
Dark bread	0.57	0.78	0.05	0.98	0.56	0.75	0.02	0.99	.014
Vegetables	1.20	1.05	0.01	1.12	1.53	1.07	−0.01	1.08	.046
Fruit/fruit juice	2.36	1.66	−0.01	1.60	2.53	1.71	−0.20	1.67	.154
Healthy foodsj									.137

All values coded so that higher values are more preferred outcomes except for low-nutrient high-density foods and sugar-sweetened beverages

^aValues represent beta coefficients for multilevel models taking into account clustering of students within classrooms and controlling for gender, pre-survey scores, and school-level socioeconomic status

^bValues represent percentage of correct answers for each category

^cValues based on a four-point scale

^dValues represent average daily food intake for each type of food

^eLow-nutrient high-density foods include French fries/snack chips, punch/sports drinks, regular/diet soda, baked desserts, and candy subscales

^fSugar-sweetened beverages include regular/diet soda and punch/sports drinks subscales

^gProteins food group includes meats, eggs, peanut butter, fish, and beans subscales

^hDairy food group includes milk, cheese, yogurt, and milk on cereal subscales

ⁱGrains food group includes cereal, pasta, white bread, and dark bread subscales

^jHealthy foods is a latent variable calculated from the milk, vegetables, fruits, and 100 % fruit juice subscales

***p < .001, **p < .01, *p < .05, ^(*) p < .10

Fig 2.

Average daily intakes of proteins (meat, eggs, peanut butter, fish, and beans) reported at pre- and post-survey by study group and gender in the evaluation of the “Nutrition Pathfinders” program among fourth-grade children in California. Intervention group boys (gray broken line); intervention group girls (black broken line); control group boys (gray solid line); control group girls (black solid line)

Table 2 shows pre- to follow-up-survey changes for each group of students on knowledge, attitudes, and dietary intake. The significant increases in knowledge observed in the intervention group at post for food group knowledge, main nutrients knowledge, nutrient functions knowledge, and healthy snack choices remained significant at follow-up (p values <.001), while balanced dinner choices knowledge remained significant at the (p < .01) level. There were no significant changes observed between pre- and follow-up-surveys on either nutrition self-efficacy or nutrition outcome expectations (p > .05). The intervention group significantly increased their consumption of proteins, grains, and white bread (p < .05), while exhibiting marginally significant increases in dark bread, the fruit and 100 % fruit juice composite, and the healthy foods latent variable compared to the control group (p < .01). Gender by treatment-group interactions were also explored for the dietary consumption variables, but no significant interactions were observed.

Table 2.

Differences by study group for pre- to follow-up-survey changes in nutrition knowledge, self-efficacy, outcome expectancies, and dietary intake observed in the “Nutrition Pathfinders” program evaluation among fourth-grade children in California (n = 931)

	Intervention				Control				Group comparison for mean change (b)a
	Pre mean	Pre SD	Change mean	Change SD	Pre mean	Pre SD	Change mean	Change SD
Nutrition knowledgeb
Food groups	0.52	0.19	0.14	0.41	0.49	0.20	0.04	0.20	.110***
Main nutrients	0.34	0.30	0.15	0.42	0.33	0.30	−0.02	0.36	.189***
Nutrient functions	0.22	0.26	0.15	0.37	0.20	0.24	0.06	0.35	.107***
Breakfast choices	0.27	0.31	0.07	0.41	0.25	0.31	0.07	0.43	.016
Snack choices	0.72	0.33	0.11	0.35	0.69	0.32	0.04	0.33	.084***
Dinner choices	0.17	0.38	0.09	0.54	0.14	0.35	0.01	0.40	.121**
Attitudesc
Self-efficacy	3.17	0.70	−0.04	0.70	3.16	0.67	−0.05	0.71	.022
Outcome expectations	3.07	0.60	0.16	0.55	3.05	0.56	0.11	0.55	.079
Dietary intaked
Low-nutrient high-densitye	3.48	2.95	−0.33	3.01	3.78	3.45	−0.25	3.07	−.245
Sugar-sweetened beveragesf	1.35	1.43	−0.21	1.52	1.45	1.57	−0.18	1.47	−.054
Proteing	2.59	1.87	−1.55	2.03	2.83	2.17	−0.25	2.39	.322*
Dairyh	4.44	1.91	−1.55	2.03	4.31	1.94	−1.32	2.01	−.112
Grainsi	2.75	1.81	0.07	2.16	2.69	1.93	−0.29	1.98	.401**
White bread	0.93	0.88	0.17	1.18	0.88	0.92	−0.05	1.14	.206**
Dark bread	0.57	0.78	0.05	1.03	0.56	0.75	−0.02	0.93	.091(*)
Vegetables	1.20	1.05	−0.04	1.21	1.53	1.07	−0.10	1.19	.101
Fruit/fruit juice	2.36	1.66	−0.05	1.74	2.53	1.71	−0.35	1.79	.243(*)
Healthy foodsj									.229(*)

All values coded so that higher values are more preferred outcomes except for low-nutrient high-density foods and sugar-sweetened beverages

^aValues represent beta coefficient for multilevel models taking into account clustering of students within classrooms and controlling for gender, pre-survey scores, and school-level socioeconomic status

^bValues represent percentage of correct answers for each category

^cValues based on a four-point scale

^dValues represent average daily food intake for each type of food

^eLow-nutrient high-density foods include French fries/snack chips, punch/sports drinks, regular/diet soda, baked desserts, and candy subscales

^fSugar-sweetened beverages include regular/diet soda and punch/sports drinks subscales

^gProteins food group includes meats, eggs, peanut butter, fish, and beans subscales

^hDairy food group includes milk, cheese, yogurt, and milk on cereal subscales

ⁱGrains food group includes cereal, pasta, white bread, and dark bread subscales

^jHealthy foods is a latent variable calculated from the milk, vegetables, fruits, and 100 % fruit juice subscales

***p < .001, **p < .01, *p < .05, ^(*) p < .10

Parent surveys

Parent surveys were collected from 1220 individuals (n = 732 intervention), with 920 collected at pre and 840 collected at post. Five-hundred forty surveys were successfully matched across time points (n = 342 intervention). Approximately 88 % of parents completed English versions of the survey. Six-hundred four surveys were completed by parents of boys and 616 by parents of girls. Additionally, 663 of the surveys were completed by mothers (79.8 %), 129 by fathers (15.5 %), 10 by grandparents (1.2 %), 10 by an older sibling (1.2 %), 17 by an “other” relationship to the child (2.0 %), and 10 surveys did not indicate the relationship of the survey taker to the child. Finally, 471 parent surveys qualified as high SES, 414 as mid SES, and 335 as low SES based on the percentage of children receiving free-or-reduced price lunches at each school.

Analysis of missing data

Dummy-coded variables for missingness were created for each time point by coding parents missing surveys as 1 and parents completing surveys as 0. Chi-square tests and logistic regression were used to predict which parents were missing pre- and post-surveys (p < .05). Child gender and group membership did not significantly predict missingness at either time point. School SES significantly predicted missingness, with parents of children at schools in the high SES category less likely to be missing at pre, while parents of children at schools in the mid SES category were more likely to be missing at pre, and parents of children at schools in the low SES category more likely to be missing at post. Finally, parents reporting higher scores on the child behavior composite score (e.g., how often children talk about food, how often children read food lables) were more likely to be missing at post.

Group differences prior to intervention

Independent samples t tests and chi-square tests of independence were used to test for differences in groups on the parent pre-survey (p < .05). There were no differences between the intervention and control groups in any of the composite scores at pre. However, the intervention group had a significantly higher proportion of parents reporting that their children were boys, and the intervention group also had a significantly higher proportion of children from schools in the high SES category (similar to results of the student surveys). Therefore, child gender and SES were included as covariates in all analyses.

Group differences in intervention outcomes

Table 3 shows pre- to post-survey changes for each group on the parent survey composite scores, and post-survey group differences for the parent serving new foods score because this construct was measured at post only. Parents in the intervention group reported significantly larger increases in their child’s interest in foods (p < .01), that their children were significantly more likely to ask to try new foods (p < .01), and that their children showed significantly larger increases in their nutrition related behaviors (p < .001). Additionally, the intervention group was significantly more likely to report serving more foods from the various food categories at post than they had been serving at the start of the program compared to the control group (this was a one-time retrospective estimate, not a change score; p < .01).

Table 3.

Differences by study group for pre- to follow-up-survey changes in parent-reported attitudes and behaviors observed in the “Nutrition Pathfinders” program evaluation among fourth-grade children in California (n = 342)

	Intervention				Control				Group comparison for mean change (b)a
	Pre mean	Pre SD	Change mean	Change SD	Pre mean	Pre SD	Change mean	Change SD
Child interest foodsb	12.39	1.70	0.12	1.78	12.54	1.67	−0.44	1.61	0.440**
Child asks for foodsc	12.24	3.00	−0.19	3.03	12.26	2.91	−0.01	2.43	−0.182
Child behaviord	19.22	4.19	3.92	7.00	19.13	4.30	2.38	7.25	1.955***
Child asks for new foodse	0.34	0.47	0.10	0.60	0.39	0.49	−0.07	0.57	0.139**
Parent interest foodsf	12.95	2.18	−0.54	2.69	13.18	1.88	−0.73	2.21	0.070
Frequency balanced dinnerg	2.22	0.92	0.00	0.89	2.22	0.93	−0.01	0.97	−0.002
	Post meanh		Post SD		Post mean		Post SD
Food parents servei	13.39		1.68		13.02		1.45		0.397**

All values coded so that higher values are more preferred outcomes

^aValues represent beta coefficient for multilevel models taking into account clustering of students within classrooms and controlling for child gender, pre-survey scores, and school-level socioeconomic status

^bChild interest foods score is a summed composite of child interest in each of the following food groups: milk, grains, fruits, vegetables, and proteins

^cChild asks for foods score is a summed composite of the frequency the child asks for foods in each of the following food groups: milk, grains, fruits, vegetables, and proteins

^dChild behavior score is a summed composite of the frequency the child performs behaviors taught by the program, including talking about foods, asking to buy foods belonging to certain food groups, trying different foods, talking about serving sizes, noticing if a meal is balanced, talking about food labels, choosing healthy foods over low-nutrient high-density foods, and cooking at home

^eChild asks for new foods score was a single item measuring whether the child asked for new foods in the last 2 months with “Yes” or “No” response option

^fParent interest foods score is a summed composite of parent interest in each of the following food groups: milk, grains, fruits, vegetables, and proteins

^hFrequency of balanced dinner score is a single item representing the frequency the parents make balanced dinners per week

ⁱThe following variables were only measured at post-survey, so there are no pre-survey scores or change scores

^jFood parents serve score is a summed composite of the parent’s likelihood they served new foods from each of the following food groups: milk, vegetables, fruits, grains, proteins, and low-nutrient high-density foods

***p < .001, **p < .01, *p < 

Program implementation

Twenty-four of the 27 teachers in the intervention group completed the teacher survey at the conclusion of the program (18 = female). The teachers had a median age of 43.48 years and a median of 12 years of teaching experience. The racial/ethnic composition of the teachers was 7 % (2 out of 30) African American/Black, 3 % (1 out of 30) Asian, 80 % (24 out of 30) Caucasian/White, 3 % (1 out of 30) Hispanic/Latino, and 7 % (2 out of 30) other. Two of the seven lessons were completed by all teachers, one was completed by 26 out of 27 teachers, two were completed by 25 out of 27 teachers, two lessons were completed by 24 out of 27 teachers, and 24 out of 27 teachers (88.9 %) reported completing all seven lessons. Teachers were able to finish the lessons in a single session 85 % of the time. If completed in one session, lessons averaged between 54 and 58 min in length. Seventeen out of 24 teachers reported presenting most of the material written in the teacher’s guide, and all teachers reported presenting at least half of it. Additionally, 18 out of 24 teachers reported adding none or between 1–24 % of their own material, and only one teacher reported adding more than 50 %. Teachers also reported that their students were either “Very” (15 out of 24) or “Somewhat cooperative and well-behaved” (9 out of 24) during the lessons. Twenty-three out of 24 teachers reported assigning some or all of the family homework. Parents reported that 83.8 % of their children completed the homework, while 6.8 % reported their child did not complete the homework and 9.4 % were unsure whether their child completed it. Approximately 58.8 % of the students that completed the homework did so by working closely with a parent, grandparent, or other adult caretaker, while 38.2 % completed it on their own, and 3 % completed it working with a sibling.

Classroom observers rated the fidelity of teacher implementation for the five components of each lesson (anticipatory set, vocabulary, step-by-step, check for understanding, and guided practice). A total of 14 observers completed 92 individual observations. Every classroom participating in the intervention group was observed 1–3 times, and each of the seven lessons was observed 7–11 times. There were 31 observations conducted by multiple observers completing separate observation forms in order to calculate inter-rater reliability. Overall, there was 92 % agreement between raters on whether the teacher presented lesson components, with percent agreement ranging from 82–96 % for individual lesson components. Classroom observers reported a high percent of teachers following the teacher guide, with teachers completing 80 % of the materials. However, teachers were more likely to follow the teacher guide more closely at the beginning of lessons and deviate from it at the end, with 87 % of teachers following the guide for the first three components of the lesson and 68 % of teachers following the guide for the last two components. Lastly, observers reported that “Many students participated” during 79.8 % of the lessons, while they reported that “Few/no students participated” only 2 % of the time.

Program dissemination

Reach

The program reached a total of 152,065 fourth-grade students attending public schools in California during the 2011-2012 School year. This represents approximately 33 % of all fourth-grade students attending public schools in the state that year (462,403 total students).

Adoption

A total of 4821 fourth-grade classrooms in California ordered the Nutrition Pathfinders program for the 2011–2012 school year. This represents approximately 26 % of all public elementary school classrooms in the state (about 18,500 total). About 53 % of teachers that ordered the materials were “new adopters.”

Maintenance

At the completion of the program, 15 out of 24 intervention teachers reported that they were planning to re-order the materials the following school year, and only one teacher indicated that they would not. Dairy Council of California records for orders showed that 41 % of all teachers re-ordered the materials for the following year (2012–2013).

DISCUSSION

The current study used the RE-AIM framework to evaluate the public health impact of a randomized controlled school-based nutrition intervention in 47 fourth-grade classrooms located throughout California. Using a pre-, post-, and 3-month follow-up-survey design, the evaluation found that the nutrition intervention significantly increased nutrition knowledge, nutrition self-efficacy, and nutrition outcome expectations immediately following the intervention. Additionally, the intervention group reported significant decreases in consumption of low-nutrient high-density foods and sugar sweetened beverages and marginally significant increases in consumption of proteins and white bread. However, the increase in proteins was seen in girls only.

Most of the gains in knowledge at post remained present at the 3-month follow-up. Other school-based interventions have seen similar improvements in nutrition knowledge [42]. In the current study, the gains seen in self-efficacy, outcome expectations, and consumption of low-nutrient high-density foods and sugar-sweetened beverages failed to persist to follow-up. However, the marginal increases in consumption of proteins and white bread observed at post became significant at follow-up, as well as consumption of grains. Additionally, the intervention group reported marginally greater consumption of fruits, the healthy foods latent variable, and dark bread. Changes in fruits, vegetables, and various forms of unhealthy foods have been commonly tested in school-based interventions, and positive effects have been found [43, 11]. Additionally, intervention group parents reported that their child was more interested in foods from the five food groups, that their child was more likely to ask for new foods, and that their child exhibited more healthy behaviors. Finally, intervention group parents reported that they had been serving new foods from the food groups more often in the time during the intervention. The changes in parent attitudes and behaviors are promising, as research has shown that parent behavior, namely parental support, may mediate intervention effects on child behavior [44]. Additionally, intervention effects observed in the home-environment resulting from parent interactions with students while completing the family homework assignments offers an exciting avenue for extending the public impact of school-based interventions to the broader community. Overall, these data offer evidence for the efficacy of the Nutrition Pathfinders program across multiple outcomes.

Reviews of the RE-AIM framework show that interventions often fail to report specific aspects of dissemination [45]. Although data on intervention reach are often reported, data describing adoption, implementation, and maintenance are less common [46]. Successful dissemination is a critical piece needed for any intervention to have substantial public health impact [46]. The present study was implemented with high fidelity, was widely disseminated, and shows the potential to be maintained over time. The vast majority of the teachers completed all seven of the program lessons, and classroom observers reported that teachers completed 80 % or more of the materials for most of the lessons observed. Dissemination data suggest that the program was ordered by about a quarter of all fourth-grade public school classrooms in California during the year of the study and reached about a third of fourth-grade students at public schools in California that year. Preliminary evidence assessing the maintenance of the program indicated that 41 % of teachers ordered the materials the following year. Secondary cost-analysis revealed that the program cost approximately $1.00 per student to implement.

Results from the current evaluation of the fourth-grade Nutrition Pathfinders curriculum in many ways resemble the results observed from the evaluation of the third-grade Shaping Up My Choices version of the program [30]. Similar to Shaping Up My Choices, the fourth-grade Nutrition Pathfinders program led to improvements in nutrition knowledge and attitudes and decreases in low-nutrient high-density foods and sugary drinks at post. Additionally, the Shaping Up My Choices led to a significant increase in the healthy foods latent variable at post while the Nutrition Pathfinders program led to marginal increases in the healthy foods latent variable at follow-up [39]. However, Nutrition Pathfinders appeared to have a favorable impact on a wider range of dietary consumption variables than Shaping Up My Choices, including proteins, grains, and white bread, as well as marginal impacts on dark breads and fruits. The larger number of food groups affected by the Nutrition Pathfinders intervention could reflect its more advanced critical thinking activities, such as goal-setting, which have been shown to improve self-regulatory ability in children [47–49], and/or children’s higher level of developmental receptivity for these skills.

Of note, group differences in self-efficacy and outcome expectations at post failed to persist through follow-up, while several nutrition variables did. This may call into question the effectiveness of the theoretical components of the study based on social cognitive theory and the health belief model [31–34]. It could mean that changes in nutrition intake at follow-up were due to a different component of the intervention, such as nutrition knowledge persisting through follow-up, the parental component of the intervention, or perhaps some other feature that was not measured. However, it may also be possible that gaps in nutrition education may be detrimental to students’ self-efficacy and outcome expectations at this age and that if education had continued then there might have been larger effects on nutrition outcomes. These important questions are more appropriately evaluated via tests of mediation to determine which features of the intervention mediated the effect of the intervention on outcome variables [50], which is outside the scope of the present manuscript.

Study limitations

There were a few limitations to the current study. Some of the composite scores (e.g., nutrition self-efficacy and outcome expectations) had fairly low Cronbach’s alphas. However, this low reliability would only limit the ability to detect change by introducing unwanted error, and the effects may perhaps be even larger than as they appear here. The validity of the scales has not been formally evaluated in this sample, although previous work offers support for the predictive validity of the scales in third grade students [40]. Another limitation is that the study used child-reported dietary intake measures, which may be susceptible to errors and biases [51, 52]. Additionally, the dietary intake measures only assessed 1 day of dietary intake, which may not be representative of typical consumption. There were also pre-survey differences in groups, most important of which was that the intervention group had more students with high SES. However, these differences were controlled for statistically. The study was randomized at the classroom level, which creates the possibility of contamination effects caused by multiple classrooms in the same school randomized to different study groups. The 3-month follow-up is also a relatively short time frame for evaluating long-term effects of the NP program.

CONCLUSIONS

Evaluating the Nutrition Pathfinders program using the RE-AIM framework suggests that it has potential for moderate to high public health impact based on its ability to induce changes in children and its wide dissemination. It was widely implemented, relatively inexpensive per student, easy for teachers to use, and the healthful dietary consumption effects generally remained to the 3-month follow-up. A vital component of practical interventions is the ability to reach a large proportion of its target population, which is done through cost-effectiveness and ease of implementation. School-based interventions have the opportunity to reach large audiences at a relatively cheap cost per participant, but they must also be easily implemented by teachers and effective enough to create changes in behavior. The RE-AIM framework outlines necessary characteristics that determine the overall public impact of interventions. Future research should apply the RE-AIM framework to assess the potential impact of interventions, as well as when planning new or updating current interventions.