Using the RE-AIM Framework to Evaluate the Statewide Dissemination of a School-Based Physical Activity and Nutrition Curriculum: “Exercise Your Options”

Abstract

Purpose

Examine the reach, efficacy, adoption, implementation, and maintenance of a physical activity and nutrition curriculum for middle-school students.

Design

Nonexperimental pilot evaluation of a statewide dissemination trial.

Setting

California middle schools during the 2006 to 2007 school year.

Subjects

Sixteen classes (N = 683 students and 16 teachers) sampled from the statewide pool who used the program.

Intervention

An eight-lesson nutrition and physical activity curriculum, “Exercise Your Options” (EYO), including a teacher guide, video clips, a student activity booklet, and ancillary materials was made available to teachers.

Measures

Program records, classroom observations, teacher surveys, and student presurveys and postsurveys (assessing physical activity, sedentary behaviors, and dietary intake).

Analysis

Descriptive statistics and multilevel random-coefficient modeling.

Results

The EYO program reached 234,442 middle-school students in California. During the program, total physical activity increased (p < .001), whereas watching TV/DVDs and playing electronic games/computer use decreased (p < .05). Intake of dairy products increased (p < .05), whereas consumption of sugars/sweets decreased (p < .001). Forty-two percent of eligible middle-school classrooms ordered the program materials. Eighty-six percent of sampled teachers implemented all of the lessons. Over the past 5 years, 51% of all middle-school students in California were exposed to the program.

Conclusions

The EYO program showed its potential for moderate to high public health impact among California middle-school students.

PURPOSE

Engaging in regular physical activity and maintaining a healthy diet can significantly reduce the risk of overweight among school-aged youth, a condition that affected approximately 17% of children and adolescents in the United States in 2003 to 2004.¹ In recent years school-based interventions have received growing research attention as a viable approach to preventing obesity. However, physical activity and nutrition programs delivered in the school setting have demonstrated only modest levels of efficacy.^3,4 These evaluations typically do not consider the overall public health impact of the interventions, which also takes into account the extent and ease with which they can be delivered to a large number of people.^5,6

To address this gap in the literature, the current study used the RE-AIM framework⁷ to evaluate the public-health impact of the “Exercise Your Options” (EYO) program, a school-based physical activity and nutrition curriculum delivered to middle-school students across California. The RE-AIM model describes the impact of an intervention in terms of five dimensions: Reach (i.e., proportion of the target population that participated), Efficacy (i.e., success rate), Adoption (i.e., proportion of target settings involved), Implementation (i.e., extent to which the program was delivered as intended), and Maintenance (i.e., extent to which the program was sustained over time).⁷

METHODS

Design

The present study served as a pilot evaluation of the statewide dissemination of the EYO program in California middle schools. Program reach, adoption, and maintenance were assessed through database monitoring of orders for program materials. Efficacy was examined through a nonexperimental design (one group with presurveys and postsurveys). Implementation was assessed through teacher surveys and classroom observations. Given the size and scope of the program dissemination, it was not feasible to recruit comparable control groups at the time of the study.

Sample

For evaluation of efficacy and implementation, a convenience sample of classrooms was drawn from the statewide pool of teachers who had indicated plans to offer the EYO program in their classrooms during the 2006 to 2007 school year. To be eligible, teachers had to confirm that they (1) had not yet taught nutrition during this school year, (2) taught mainstream seventh-grade science, health, or physical education classes, (3) were at traditional calendar schools, (4) were able to begin teaching nutrition in January and finish in February, and (5) would be able to use the DVD that is part of the program. Twenty of the 86 teachers contacted (23%) meet the criteria and agreed to participate in the evaluation. Teachers received $75.00 each to compensate for their participation. Teachers sent recruitment letters to parents/guardians of their students that provided parents with a phone number to call for questions regarding the study and explained that the form was to be returned if parents declined to have their children participate (i.e., active information and passive consent procedures). If the parent/guardian did not decline, the student was approached for assent. The Institutional Review Board at Independent Review Consulting, Inc., approved the study procedures.

Measures

Student Presurveys and Postsurveys

Physical activity levels, sedentary behaviors, and nutrition behaviors were assessed immediately before and after the 8-week program using items from the Youth Risk Behavior Surveillance System (YRBS).⁸ Physical activity, TV/DVD watching, and computer use/video game playing were assessed as follows: (1) “During the past 7 days, on how many days were you physically active for a total of at least 60 minutes per day? (Add up all the time you spend in any kind of physical activity that increases your heart rate and makes you breathe hard some of the time.),” (2) “Yesterday how many hours did you watch TV or video/DVD movies?,” and (3) “On an average school day, how many hours do you play video or computer games or use a computer (for something that is not school work)?” Modified YRBS items assessed consumption of dairy products (i.e., milk, yogurt, cheese, cottage cheese), fruits (including 100% fruit juices), vegetables, and sugars/sweets (i.e., sweet rolls, doughnuts, cookies, brownies, pies, cake, chocolate candy, soda, soft drinks, punch, Kool-Aid, sports drinks). Participants were asked to report how many times they had eaten various types of foods within each group on the previous day.

Teacher Surveys

Teacher surveys assessed the number of EYO lessons that were implemented in addition to other constructs not presented in the current study (e.g., satisfaction, perceptions of student interest).

Classroom Observations

Trained research staff observed teachers’ implementation of the EYO lessons (i.e., percent of lesson content conveyed, whether lessons were presented in recommended order). Four of the eight EYO lessons were observed over a 2-week period.

Intervention

EYO was a teacher-delivered physical activity and nutrition curriculum designed for California middle-school students (grades 6−8). The program consisted of an eight-lesson student activity booklet and supportive video clips, provided on DVD/CD-ROMs and through a Web site. A teacher's guide that contains lesson plans and instructions was provided. EYO lessons were designed to align with the U.S. Department of Agriculture's food guidance system and the California and National Education standards (middle school, grades 6−8) in language arts, math, and science. Materials promoted consumption of a balanced, nutrient-rich diet with a special emphasis on increasing the intake of low-fat and nonfat dairy products, whole grains, and fruits and vegetables and decreasing the intake of sugars and sweets. Participants learned about each food group, the quantity they should consume from each, and how they can trade sugars and sweets calories for nutrient-rich foods. The physical activity lesson asked students to complete activity charts and identify how they can substitute sedentary activities (e.g., watching TV, playing video games) with activities of greater intensity. The curriculum was designed to help students acquire skills to make healthy choices in all aspects of their lives (in both school and home settings). Program materials were free of charge to California teachers, who could order program materials online, by mail, or by phone. EYO was implemented in science, physical education, health, and other types of nutrition and wellness classes.

Analysis

Descriptive statistics were used to evaluate program reach, adoption, implementation, and maintenance. Program efficacy (i.e., the effect of the intervention on nutrition, physical activity, and sedentary behaviors) was analyzed using multilevel random coefficient modeling (HLM version 6.0, Scientific Software International, Lincolnwood, Illinois).⁹ Mean scores on the presurveys and postsurveys were compared to determine if there were significant changes in these outcomes over the course of the intervention. The multilevel random coefficient modeling procedure took into account that data were clustered within classrooms (N = 16) and violated the statistical assumption of independence. Because some variables were not normally distributed, robust standard errors were used. All analyses controlled for gender and grade in school.

RESULTS

Reach

Combined, the program reached 234,442 middle-school students in California during the 2006 to 2007 school year, which represents approximately 50% of the available students in any one grade (N = 490,223 students). (The EYO program is only offered to one grade per school.)

Efficacy

Of the 20 teachers who initially agreed to participate in the efficacy evaluation, 16 returned complete student-survey packets. A total of 695 students completed either presurveys or postsurveys (n = 567 for the presurvey and n = 566 students for the postsurvey). Due to absences on the survey days, some students did not complete both presurveys and postsurveys. Data from 12 students could not be used because of missing identification numbers. The remaining sample (n = 683) was composed of 53.5% boys and 46.5% girls. Ninety-four percent of students were in the seventh grade, 5% were in the eighth grade, and 1% was in the ninth grade. The mean age of the participants was 12.47 years (standard deviation = 0.60).

Tables 1 and 2 provide descriptive and difference statistics for physical activity/sedentary behaviors and food group intake reported on the presurveys and postsurveys. Between the presurveys and postsurveys, days per week of at least 60 minutes of physical activity increased (p < .001), whereas hours per day of TV/DVD watching (p = .024) and video game playing/nonschool-related computer use (p = .002) decreased (Table 1). There also was a statistically significant increase in the daily consumption of dairy products (p = .001). In contrast, the daily consumption of sweets/sugars decreased (p < .001). There was no change in the consumption of fruits or vegetables (Table 2).

Table 1.

Descriptive and Difference Statistics for Physical Activity and Sedentary Behaviors Reported on the Presurveys and Postsurveys^*

	Presurvey Mean (SD)	Postsurvey Mean (SD)	t(df)	p
Physical activity (days/week)	4.01 (2.15)	4.44 (2.07)	4.61 (15)	<0.001
Playing video games/computer use (hours/day)†	1.68 (1.54)	1.37 (1.35)	−3.84 (15)	0.002
Watching TV/DVDs (hours/day)	1.57 (1.07)	1.41 (1.03)	−2.52 (15)	0.024

^*t-tests were generated from the final estimation of fixed effects in multilevel modeling. Robust adjustments were made to the standard errors for the calculation of p. Models controlled for age and grade. Unadjusted means are presented. n = 630 students (physical activity), n = 625 students (playing video games/computer use), and n = 626 students (watching TV/DVDs).

^†Computer use was assessed for purposes other than school work.

SD indicates standard deviation.

Table 2.

Descriptive and Difference Statistics for the Daily Consumption of Dairy Products, Fruits, Vegetables, and Sugars/Sweets (in times per day) Reported on the Presurveys and Postsurveys^*

	Presurvey Mean (SD)	Postsurvey Mean (SD)	t(df)	p
Dairy products	2.84 (1.70)	3.14 (1.73)	2.87 (15)	0.001
Fruit	2.23 (1.56)	2.21 (2.12)	−0.40 (15)	0.698
Vegetables	1.15 (0.98)	1.17 (0.96)	−0.15 (15)	0.885
Sugars/sweets	4.17 (3.09)	3.44 (2.81)	7.00 (15)	<0.001

^*t-tests were generated from the final estimation of fixed effects in multilevel modeling. Robust adjustments were made to the standard errors for the calculation of p. Models controlled for age and grade. Unadjusted means are presented. n = 622 students (dairy), n = 617 students (fruit), n = 630 students (vegetables), and n = 601 students (sugars/sweets).

SD indicates standard deviation.

Adoption

Of the 3803 middle-school teachers (grades 6, 7, and 8) in California for whom data were available (i.e., approximately 95% of all middle-school teachers in California), 1606 teachers (42%) ordered the program materials for the 2006 to 2007 school year. The total number of orders was composed of 790 requests made by teachers who had not previously used the materials and 816 requests for resupply orders. Findings from the teacher interviews indicated that the teachers learned about the program through peer referrals (62%), educational conferences (31%), and exposure to program materials at school (7%).

Implementation

Teacher surveys (n = 14) and classroom observations (n = 8) were conducted among the 20 teachers who initially agreed to participate in the evaluation. On the teacher survey, 86% of the teachers reported implementing all eight of the EYO lessons. Classroom observation data showed that 75% of the lessons observed were implemented in the recommended order. Observers reported that all of the teachers conveyed 81% to 100% of the lesson content.

Maintenance

Program usage reports for the prior 5 school years (2002−2003 through 2006−2007) showed that 9340 teachers ordered EYO during this period, resulting in a total of 1,247,889 student workbook requests. Across the 5-year time span, an average of 1868 teachers and 249,578 students used the program each year (i.e., approximately 134 students per teacher per year). It was difficult to obtain comparable workbook reorder rates from teachers across consecutive years because the grade and subject areas for which the EYO program was available varied from year to year. On average, however, 52% of teachers reordered the program, whereas 48% ordered the program for the first time during the 5-year period.

DISCUSSION

Evaluating the overall impact of interventions to prevent childhood obesity requires a thorough consideration of the extent to which they are used, are effective, and are feasible in real-world environments. The current study used the RE-AIM framework⁷ to assess the statewide impact of the EYO program, a teacher-delivered nutrition and physical activity curriculum for middle-school students in California. Results showed that the program reached approximately 50% of applicable students and was adopted by 42% of the teachers to whom it was offered. The EYO program had a strong influence on physical activity and sedentary behaviors, whereas positive effects for dietary intake were observed for some but not all of the outcomes measured. The majority of teachers delivered all of the lessons and implemented individual lessons in the recommended order. Program use by teachers remained relatively constant during the prior 5 school years, with approximately 51% of students reached. Taken together, these findings indicate that the EYO program has the potential for a moderate to high public health impact among California middle-school students.

Despite the strengths of the current evaluation, including a large sample and a multimethod measurement approach, the nonexperimental nature of the design limits the ability to make causal conclusions about the program efficacy. Without an equivalent control group, the extent to which historical and external influences accounted for changes in physical activity and nutrition behavior is unknown. However, the RE-AIM framework can be used to evaluate studies with designs other than randomized, controlled trials.⁷ In the current study, efficacy was just one of a number of indicators used to describe overall program impact. Also, because of the self-report nature of the physical activity and dietary intake instruments, these assessments may be vulnerable to recall biases and social desirability. Lastly, only short-term results are presented for the program efficacy component. Long-term follow-up measures are needed to assess the maintenance of behavior changes at the individual level.

Overall, an assessment of the reach, efficacy, adoption, implementation, and maintenance of the EYO program showed its potential for moderate to high public health impact among California middle-school students. Given the paucity of research evaluating the large-scale dissemination of obesity prevention interventions, the RE-AIM assessment framework should be applied across a wider range of school-based programs and populations in future research.