Campus-Community-School Partnerships to Evaluate a Multicomponent Nutrition Intervention

In these times of tight budgets and decreased funding for public health, greater emphasis is being placed on designing and implementing evidence-based programs and policies. With this challenge comes a call for evaluating the efficacy of interventions and disseminating the results to stakeholders. The recent Institute of Medicine (IOM) report, Progress in Preventing Childhood Obesity: How Do We Measure Up?,¹ underscores the need to track and evaluate the efforts of schools, government agencies, and industry to reduce the burden of childhood overweight, so that successful initiatives can be replicated, adapted, refined, or amplified. At the same time, there is increasing interest in training public health professionals in the hands-on practice of public health, including program evaluation. However, the time required of faculty, students, and field preceptors can be barriers to meaningful experiences,² making it difficult to create and sustain these university-community partnerships.³

In schools, interventions are being driven by the adoption of national and state legislation that requires school nutrition and wellness policies. Sponsors of state and national policies seek evidence of their effectiveness, but resources for program evaluation are often limited. Furthermore, there may be under-recognition of the role evaluation plays in successful program implementation, planning, and sustainability. This article describes the evaluation of a multicomponent school nutrition intervention that successfully engaged public health students in meaningful public health practice, while providing key evaluation resources to the local school district, community organizations, and program funders.

BACKGROUND

In spring 2005, a partnership of local stakeholders working with Active Living by Design⁴ and led by Seattle advocacy organization Feet First was awarded funding to pilot a school-based nutrition intervention, the Eat Better, Feel Better program. The program, funded by the Robert Wood Johnson Foundation (RWJF) as part of Healthy Eating by Design,⁴ sought to increase access to healthy foods for children in low-income communities and schools. The goals of the program were to:

• Develop and maintain an effective partnership to promote healthy eating;

• Increase community awareness of the health and other benefits of healthy eating;

• Increase access to, and availability of, diverse healthy eating;

• Enhance policy and organizational supports for healthy eating; and

• Improve built environments, facilities, equipment, and provide other support for healthy eating.

The primary intervention activities were:

• Classroom nutrition education;

• Introduction of salad bar in lunchroom;

• Staff wellness programs;

• School assemblies that promoted enthusiasm for the project in the whole school and offered opportunities for students and staff to taste fruits and vegetables; and

• Family education and tasting nights.

During the summer of 2005, the Project Coordinator approached the Nutritional Sciences Program at the University of Washington (UW) for assistance in evaluating the cumulative impact of the Eat Better, Feel Better program, Food Sense Change⁵ (the nutrition education component within the program), and the U.S. Department of Agriculture (USDA) Fruit and Vegetable Pilot Program (FVPP)⁶ that were implemented concurrently in one elementary school.

RESOURCES AND PARTNERS

This evaluation project built on existing relationships and mobilized a variety of resources from the university, community, and schools to overcome traditional barriers to program evaluation. Most of the work to design, implement, analyze, and disseminate the evaluation results was conducted by students and faculty of the Nutritional Sciences Program (NSP) in the UW School of Public Health and Community Medicine. The NSP offers a practice-based course titled Public Health Nutrition that addresses multiple components of public health practice for nutrition, including theories and models for program planning and evaluation; cultural competency; leadership; management; and policy development, implementation, and change. The Eat Better, Feel Better program and FVPP evaluation provided students with an opportunity to develop and practice these competencies, while meeting an important community need.

During initial planning, it was determined that a coordinator acting as school liaison was essential for the project's success. A Master's of Public Health (MPH) student served as coordinator, conducting the preliminary literature review and gaining necessary approval from the UW Human Subjects Internal Review Board and the Seattle Public Schools research office. The student was also responsible for research design, developing data collection tools—including a draft student survey and a lunchroom observation tool—and for drafting interview questions. The graduate student coordinator also secured participation of a control school and met regularly with stakeholders for input and to maintain support and momentum for the project. This experience fulfilled the student's requirements for an MPH practicum. In addition to sponsoring the student evaluation team, the UW Center for Public Health Nutrition donated grocery store gift cards to thank parents and school staff who participated in interviews.

Seattle School District staff provided critical access to the schools, reviewed project materials, mailed study materials to parents, and distributed information to teachers and staff. Teachers assisted during the lunchroom observation days and distributed and collected the student surveys in the classroom. A culture of trust between the school district and the university, as well as supportive school principals, was essential to the project's success.

THE CLASS EVALUATION PROJECT

The objectives of the evaluation were: (1) to evaluate the effectiveness of the Eat Better, Feel Better program in promoting healthy eating behaviors among fifth-grade students and their families, and among school staff at the target elementary school, and (2) to evaluate attitudes about the program and its impact on school staff workload.

This project took place in three phases during a 10-week academic quarter. During phase one, students gathered background information, prepared logic models, and trained each other in the skills they would need to conduct lunchroom observations and interviews. In phase two, all of the Public Health Nutrition students (n = 16) were involved in lunchroom observations, which were conducted on three different days at both the intervention and the control school. All students also conducted parent and teacher interviews. In the final phase, students compiled and analyzed data, drafted a report and an executive summary, and presented their results to school and agency staff and other stakeholders. A videotape of the presentation and a copy of the final report were posted on the Web so that all interested community members could access materials.

RESULTS

Fifth-grade students in both the intervention and control schools were asked to complete a brief survey that asked about their fruit and vegetable knowledge and behaviors (Table). The results illustrate that students at the intervention school were more willing to choose vegetables served at lunch and reported a significantly greater intake of fruit and a greater intake of vegetables compared with students at the control school. Graduate students conducted semi-qualitative open-ended teacher, staff, and parent interviews (n = 30) at the intervention school. Key findings from these interviews are presented in the Figure.

Table.

Student survey results

^aThe Chi-Squared test or Fisher's exact test was performed.

^bQuestions derived from California Nutrition Network for Healthy, Active Families. Harvest of the month survey: fourth and fifth grade questionnaire. 2005 [cited 2006 Nov 8]. Available from: URL: http://www.harvestofthemonth.com/download/Impact-Evaluation.pdf

^cCategorized as <3 fruits/vegetables or ≥3 fruits/vegetables

Figure.

Key findings—parent/guardian and teacher/staff interviews

The graduate students developed a protocol for observing fifth graders' fruit and vegetable consumption during lunch. The protocol was based on limited existing methodological literature. Observations were completed at both the intervention and the control school. It was disappointing to all of the partners that there were no differences in fruit and vegetable intake during lunch between the schools. However, in the context of graduate student learning, this finding provided a rich opportunity to talk about the factors that were outside the control of the intervention and its evaluators, and the difficulties posed when evaluation results are not as expected.

CHALLENGES, LESSONS LEARNED, AND BENEFITS

School district staff, anxious to offer evidence of prudent fiscal investment and effectiveness to the Seattle School Board, and to funders such as RWJF and USDA, were supportive and engaged partners. However, institutional scheduling conflicts limited the opportunities for graduate students to work with schools and families. The 10-week Public Health Nutrition course schedule overlapped with the public school spring break, standardized academic testing, and end-of-year blackout dates, during which time university students were not able to be present in the intervention or control school.

Nevertheless, interviews and observations provided real-world experience for students to experience and grapple with the day-to-day challenges of program evaluation. Many graduate students wrote on their course evaluations that the opportunity to work on this project was a valuable part of their graduate student experience; others found it to be uncomfortable. As observed elsewhere, graduate students who perform best in classroom settings are not always those who perform best in practice settings,⁸ and the varying degrees of experience and confidence among students made additional classroom discussion and faculty mentoring essential.

The project represented a $50,000 intervention during an 18-month grant time period. A primary goal of the RWJF Healthy Eating by Design program was to rapidly deploy nutrition intervention pilot projects, so that RWJF could learn how to structure larger-scale and longer-term projects that include environmental changes as strategies to address childhood obesity.

While RWJF conducted national site evaluation visits to the 12 grantee sites, no quantitative data were generated. In part due to the successful evaluation partnership with UW, the Eat Better, Feel Better project was given an Innovative Project award and was featured by RWJF on its website in spring 2006.

The Eat Better, Feel Better evaluation project resulted in sustainable relationships, which are providing additional opportunities for collaboration. In 2007, the NSP is again partnering with Seattle Public Schools and community agencies to evaluate a combined school breakfast and walk-to-school intervention. Pilot results from FVPP evaluation are being used to support a proposal to increase the fruit and vegetable program in Washington State, and further evaluation of the salad bar venue as a way to increase fruit and vegetable intake is underway as part of an MPH student thesis project.

CONCLUSION

Schools of public health are often challenged to provide students with practical, real-world learning experiences. Creative partnerships that leverage resources from several sectors can be used to build a public health workforce equipped to develop, implement, and evaluate programs and policies, while providing a critical service to local public schools and agencies.