Process evaluation results from the HEALTHY nutrition intervention to modify the total school food environment

Abstract

The process evaluation of HEALTHY, a large multi-center trial to decrease type 2 diabetes mellitus in middle school children, monitored the implementation of the intervention to ascertain the extent that components were delivered and received as intended. The purpose of this article is to report the process evaluation findings concerning the extent to which the HEALTHY nutrition intervention was implemented during the HEALTHY trial.

Overall, the observed fidelity of implementing nutrition strategies improved from baseline to the end of the study. By the last semester, all but two nutrition process evaluation goals were met. The most challenging goal to implement was serving high fiber foods, including grain-based foods and legumes. The easiest goals to implement were lowering the fat content of foods offered and offering healthier beverages. The most challenging barriers experienced by research dietitians and food service staff were costs, availability of foods and student acceptance. Forming strong relationships between the research dietitians and food service staff was identified as a key strategy to meet HEALTHY nutrition goals.

Introduction

Background and rationale for HEALTHY (overall study)

Poor dietary intake and physical inactivity are primary contributing factors for several major diseases including type 2 diabetes mellitus (T2DM). T2DM is a devastating disease with complications that include cardiovascular disease, renal failure, blindness and limb amputation. Historically, T2DM was rarely found in children and adolescents; however, there have been significant increases in the global prevalence of overweight/obesity and T2DM in recent decades in the pediatric population [1–3].

Recent research indicates that healthy eating and regular physical activity are essential for the prevention of T2DM [4]. The Dietary Guidelines for Americans 2010 include both healthy eating and physical activity in one of their key recommendations: ‘Prevent and/or reduce overweight and obesity through improved eating and physical activity behaviors’ [5]. The Guidelines also state, ‘Maintain appropriate calorie balance during each stage of life—childhood, adolescence, adulthood, pregnancy and breastfeeding, and older age’ [5].

In response to the considerable increase in pediatric T2DM, the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) of the National Institutes of Health (NIH) sponsored a primary prevention trial called HEALTHY [6]. Though ‘HEALTHY’ is not an acronym, the study name emerged from formative research with members of the target population, and was used to brand all associated study activities.

The objectives of the overall HEALTHY study were to develop and test a comprehensive, school-based intervention aimed at reducing modifiable risk factors for T2DM in middle school youth by promoting physical activity and healthy nutrition.

HEALTHY study design (overall study)

The HEALTHY study was a multi-component, school-based, cluster randomized controlled trial with 42 middle schools participating (21 intervention, 21 control). Seven field centers across the United States administered the study and were overseen by a steering committee, comprising the principal investigators from the seven field centers, a coordinating center and representatives of the NIH (NIDDK), which funded the study through a cooperative agreement. Youth participating in the study included 4603 middle school students (2307 intervention; 2296 control) who were assessed multiple times throughout the study for various outcome measures; however, the main outcome measures, body mass index, fasting insulin and fasting glucose levels were evaluated during the first semester of their sixth grade year and again during the second semester of eighth grade. The HEALTHY intervention consisted of four integrated components: behavior, communications/social marketing, nutrition and physical education. The intervention began during the second semester of cohort students’ sixth grade year and continued until the end of their eighth grade year. Details of the HEALTHY Study research design and methods as well as details regarding each intervention component have been reported elsewhere [6–10].

HEALTHY nutrition intervention

The purpose of the nutrition intervention was to improve the quality of the foods and beverages offered to students by changing the total school food environment [8]. The nutrition intervention focused on five goals: (i) lower the average fat content of food offered in schools; (ii) offer at least two servings of fruit and/or vegetables per student participating in the National School Lunch Program (NSLP) and at least one serving per student participating in the School Breakfast Program (SBP) each day; (iii) offer only dessert and snack foods with ≤200 kilocalorie per single serving and/or package; (iv) eliminate milk containing ≥1% fat and added sugar beverages and decrease the serving size of 100% fruit juice and (v) offer at least two servings of high fiber (≥2 g of fiber per serving) grain-based foods and/or legumes per student on NSLP and at least one serving per student on SBP each day [8]. Multiple strategies were used to help achieve each goal. The intervention was coordinated by a research dietitian at each field center. Each research dietitian was a registered dietitian, whose main focus was to implement the nutrition goals of the study. In addition to changing foods and beverages offered in the cafeteria, the research dietitians worked with the food service managers and staff to improve food and beverage choices within the a la carte lines, vending machines and school stores. They worked with school personnel to eliminate foods of minimal nutritional content and/or to discontinue giving foods and beverages high in fat and added sugar as rewards. Specifically, the research dietitians worked with school staff to locate and order foods that met HEALTHY goals and were within cost limits of the schools. They also worked with school and HEALTHY staff to organize activities that encouraged students to try the new foods offered at breakfast and lunch.

Schools were required to hold at least one taste test each semester. These events allowed students to taste newly added foods/beverages and to pilot foods/beverages under consideration. Each semester, one cafeteria learning lab (CLL) was to be implemented by HEALTHY staff to promote and teach students healthy nutritional behaviors. In addition, the research dietitians observed meals and suggested a variety of changes school staff could make to encourage students to choose new foods.

HEALTHY process evaluation

Process evaluation is an assessment of the implementation of an intervention and is useful in understanding the dynamics of a trial and ensuring that the study interventions were delivered as designed [11–13]. The design of the HEALTHY process evaluation was based on a conceptual framework outlined by Linnan and Steckler [11]. Process evaluation helped ensure that components were implemented successfully and consistently across all 21 intervention schools. In a complex, multi-component intervention such as HEALTHY, process evaluation data can be used to document the extent to which various components were actually implemented and how the intervention was received by the target group. A variety of components were assessed in the process evaluation of the HEALTHY nutrition intervention, including fidelity, implementer participation and barriers [11, 14]. Briefly, pilot and formative processes were conducted to inform the intervention. During the main HEALTHY Study, qualitative and quantitative methods were used to evaluate the fidelity of the study [14]. As this was a multi-center, multi-faceted trial, all components of the trial were monitored using planned observations to ensure that the intervention was delivered as intended [14].

The purpose of this article is to report the process evaluation findings concerning the extent to which the HEALTHY nutrition intervention was implemented as intended during the main trial. This article also discusses how process evaluation methods were conducted in a complex, multi-faceted intervention.

Methods

Process evaluation measures and procedures

Process evaluation data for the HEALTHY Study were collected from each intervention semester through the course of the intervention, which lasted five semesters as the cohort progressed through middle school. The process evaluation utilized a mixed method that combined quantitative and qualitative approaches to gather information. Implementation of the nutrition intervention was assessed via structured observations and interviews, which have been shown to provide the most instructive data [15]. A comprehensive description of the methods for process evaluation staff training, instrument development, data collection, data entry and data management has been reported elsewhere [14].

School food environment observations

Trained research staff not involved in the implementation of the intervention observed the school food service environment at each intervention school twice each semester for five semesters, once during the first half of the semester and again in the latter half of the semester, providing a total of 210 observations. Dates of the observations were randomly determined. The research assistants had to work with the principals to decide the date of the observations to ensure that it would represent a typical day. Each observation directly assessed all food and beverage points of services, including the cafeteria serving line, a la carte, vending machines and school stores.

Document analyses

Cafeteria menus, work production sheets and nutrition specification sheets in each food service manager’s file as well as the product information notebook compiled by the research dietitian were used to supplement the observations. The observation instrument included dichotomous scale items to assess whether or not the nutrition core strategies that corresponded to the nutrition goals were implemented. For example, if an item with a high fat content was being offered for lunch in the cafeteria during the intervention period, then it was recorded that the school did not implement the strategy to reduce high fat entrees under goal 1 (refer to the goals previously stated earlier in this article). These observations assessed the fidelity of the nutrition intervention.

Food service personnel interviews

Structured interviews were conducted by trained staff with the food service managers at each school and district-level food service directors or supervisors at the intervention schools during the last year of the study. These interviews were audiotape recorded and transcribed. Each interview lasted ∼30–45 min and was conducted at times requested by school staff members. The interviews consisted of Likert-type ratings scales and open-ended, non-leading questions. The goals of the interviews were to assess: (i) the perceived effectiveness of intervention components; (ii) efficiency of implementation of intervention components; (iii) attitudes toward the intervention; (iv) recommendations for dissemination; (v) receptivity of individuals in the school to the intervention and (vi) recommendations for intervention improvement. A total of 27 individual interviews were conducted with 32 food service personnel [for some schools, more than one food service personnel were interviewed (e.g. pairs of food service personnel)].

Research dietitian interviews

Structured interviews were conducted with the HEALTHY research dietitians from each field center at the end of each semester. Trained interviewers took detailed notes during the audiotape recorded interviews. The audio recordings were used to supplement notes and capture illustrative quotes. Full-length transcripts were not created because process evaluation data had to be rapidly analyzed and reported to the study group at the conclusion of each intervention semester. The interviews lasted 30–45 min and consisted of Likert-type rating scales and open-ended questions. During the first three semesters, one interview was conducted per intervention school and during the last two semesters, one longer interview was conducted to encompass all three intervention schools at a given field center. The goal of the interviews was to evaluate: (i) which strategies were the focus of that particular semester; (ii) barriers encountered during the semester; (iii) perceived effectiveness of taste tests and CLLs; (iv) quality of research dietitians’ relationships with school food service managers and staff and (v) integration of intervention components. A total of 77 interviews were conducted with all seven research dietitians over the course of the study.

Data analyses

Quantitative observational data collected at each field center were electronically transferred to a central database maintained by the study coordinating center. The statistical analysis software (SAS; version 9.2; SAS Institute, Cary, NC, USA) was used to analyze these data and provide descriptive and longitudinal results [via means, standard deviations (SDs) and percent values]. Interview data collected at each field center were sent to the Qualitative Data Core at the University of North Carolina, Chapel Hill and entered into a computerized database. ATLAS.ti (version 5.2; Scientific Software Development GmbH, Berlin, Germany) was used for data coding and analysis. Interview data were open-coded by interview guide topic using qualitative content analysis [16]. Review of coded data revealed emergent themes and trends. One of the authors (W.J.H.) independently and systematically coded all of the interview data relevant to this study.

Results

The 21 HEALTHY schools that received the nutrition intervention had, on average, 265 students (SD = 96) per school and were composed, on average, of 45.6% Hispanics, 30.7% of non-Hispanic Blacks, 17.8% non-Hispanic Whites and 5.9% other race/ethnicities. The percent of students eligible for receiving free/reduced meals in the 21 schools was 78.8% (SD = 14.2).

Fidelity of nutrition intervention implementation

Quantitative observational data from the school food environment were used to calculate nutrition intervention fidelity values based on the percent of time intervention schools met the HEALTHY nutrition goals. Table I lists each of the nutrition intervention goals and the number and percentage of intervention schools that met the goals at both observation time points during the five intervention semesters as well as across all semesters. The data show that, by the end of the first semester, only one of the five nutrition goals was met by >50% of the intervention schools (goal 2, met by 65% of the schools). By the end of the second semester, four of the five goals were met by >50% of the intervention schools (goal 1 was met by 76% of the schools, goals 2 and 4 by 81% of the schools and goal 3 by 86% of the schools). The most difficult goal to meet was increasing grain-based foods and legumes.

Table I.

Nutrition goal implementation (N = 21 schools for each semester, N = 105 schools overall)

	Spring 2007	Fall 2007	Spring 2008	Fall 2008	Spring 2009	Overall
	Met n (%)	Met n (%)	Met n (%)	Met n (%)	Met n (%)	Met n (%)
Goal 1: Lower fat content	9 (43)	16 (76)	21 (100)	21 (100)	21 (100)	88 (84)
Goal 2: Increase fruits and vegetables	13 (62)	17 (81)	18 (86)	15 (71)	20 (95)	83 (79)
Goal 3: Limit dessert and snack kilocalories to <200 kilocalories	10 (48)	18 (86)	19 (90)	21 (100)	21 (100)	89 (85)
Goal 4: Increase healthier beverages	6 (29)	17 (81)	17 (81)	21 (100)	21 (100)	82 (78)
Goal 5: Increase fiber content	7 (33)	9 (43)	11 (52)	18 (86)	19 (90)	64 (61)

Met (%) = total number of schools that met the goal in that semester and the percent of all of the schools that met the goal in that semester.

Note: Must have met both observations for that semester; if ‘not met’ for one or both observations, then the goal was not met.

Based on data from documentation logs of intervention events implemented, a total of 180 taste tests were delivered across the 21 intervention schools during the study. Intervention schools were required to implement at least one taste test per semester, and an average of 1.7 taste tests (SD = 0.7) were delivered at each intervention school each semester. Overall, all of the intervention schools participated in at least one taste test per semester (the minimum requirement); however, 56.2% of the schools completed more than the minimum requirement across all semesters (Table II). The events enabled students to taste foods such as kiwi, whole grain waffles, whole wheat pizza and 1% milk, among other foods. It has been reported that taste tests are a valuable approach for nutrition education [15, 16].

Table II.

Number of taste tests provided each semester (N = 21 schools for each semester, N = 105 schools overall)

Semester	Number of taste tests
	One taste test per semester	Two taste tests per semester	Three taste tests per semester
	N (%)a	N (%)	N (%)
Spring 2007	9 (42.9)	6 (28.6)	6 (28.6)
Fall 2007	6 (28.6)	15 (71.4)	0 (0)
Spring 2008	8 (38.1)	9 (42.9)	4 (19.0)
Fall 2008	9 (42.9)	9 (42.9)	3 (14.3)
Spring 2009	14 (66.7)	4 (19.0)	3 (14.3)
Overall	46 (43.8)	43 (41.0)	16 (15.2)

^aN (%) = Total number of schools that conducted the stated number of taste tests for that semester and the percent of all of the schools that conducted the stated number of taste tests for that semester.

Each semester, one CLL was also organized by HEALTHY intervention staff to educate students on healthy nutrition. Documentation logs showed a total of 104 CLLs were implemented across the 21 intervention schools; thus, approximately one CLL was delivered at each intervention school each semester (one intervention school did not complete a CLL in Fall 2008). CLL topics included: (i) the sugar content of various popular beverages, (ii) the amount of activity time it takes to expend calories of popular snacks, (iii) nutrition facts labels, (iv) healthy portions of common foods and (v) healthy lifestyle choices. In addition, the research dietitians frequently observed meals and suggested a variety of changes that the school staff could make to help encourage students to choose new foods.

The research dietitians were asked during the interviews to discuss their relationships with food service managers and staff. They reported that the quality of the relationship was key to ensuring food service managers’ and staffs’ cooperation in implementing the intervention. Overall, the research dietitians reported ‘great’ relationships with staff. The research dietitians learnt throughout the intervention that incentives and communication were vital to successfully implementing elements of the nutrition intervention. Simple incentives, such as gift cards and t-shirts, demonstrating appreciation for extra work required by the intervention were greatly appreciated by school staff. The research dietitians also reported that including food service staff in decisions related to intervention changes helped enhance the level of acceptance from all staff members.

Food service staff also discussed their relationships to the study staff during interviews. Several noted the background information on nutrition provided during HEALTHY trainings was helpful. One mentioned, ‘It was such a great education for me, and I would hope this education would be for a lot of the managers. We, as managers feeding this many children on this level, we should be educating all of the managers, should be having the opportunity to be exposed to this nutrition education that we got. It really, benefited me, training, teaching children how to eat healthy’.

Barriers to nutrition intervention implementation

Food service staff and research dietitians were asked to discuss the barriers associated with implementing the nutrition component. The most frequently reported barriers were the costs associated with making the changes, availability of healthier foods and student acceptance of new foods and beverages. Both the food service staff and research dietitians discussed the challenge of finding foods that met HEALTHY goals were within cost limits of budgets and would be well received by students.

At the beginning of the study when food service familiarity with the HEALTHY Study was not as strong, it was difficult to implement changes that were perceived as possibly decreasing the revenue of the cafeterias. As one research dietitian noted in an attempt to eliminate sweetened beverages, ‘Sweetened beverages were popular with students and brought in revenue’. As the study continued, the lack of availability and prohibitive costs of higher fiber foods were significant barriers to achieving goals. One research dietitian said during an interview, ‘Our economic crisis has more of an effect on the food service department than I had anticipated, which is what I believe is part of the decision-making process for the next school year, and why they may not be purchasing some of our higher-cost, healthier items. It makes me sad, but this is our reality’.

Food service staff mentioned costs as a barrier, but also mentioned student reception. One food service worker mentioned in an interview, ‘From the food point it was getting them to change their idea about the way they ate. When we cut out the frying, which I was very much for, they started baking the fries, the kids whined. They really did. It was hard getting them to change their way of thinking about foods in the beginning’. They mentioned that student acceptance occurred gradually over the course of the study. In addition, food service staff also mentioned that finding foods that met the HEALTHY nutrition goals, especially the fiber goal, was challenging.

Discussion

The primary purpose of this article was to report the process evaluation findings concerning the extent to which the HEALTHY nutrition intervention was implemented during the main trial in terms of fidelity and barriers. In addition, we discussed how process evaluation methods were conducted in the midst of a complex, multi-faceted intervention.

Overall, the results of observations showed improvement in the observed fidelity of implementing nutrition strategies from baseline to the end of the study. By the last semester of the study, only two of the five nutrition intervention goals failed to be met consistently across all schools. The most challenging goal to implement was serving high fiber foods, including grain-based foods and legumes. Mobley et al. [17] reported that the higher cost of high fiber foods compared with other foods available from the United States Department of Agriculture was the main reason that food service personnel did not purchase high quantities of high fiber foods. Interestingly, Treviño et al. [18] reported that, despite the HEALTHY intervention, schools meeting most of their nutritional goals to implement healthier foods in the schools, there were no statistical differences between HEALTHY intervention schools and control schools in school food service finances.

The easiest goals to implement were lowering the fat content of foods offered and offering healthier beverages. Forming strong relationships between the research dietitians and food service staff was observed as key to meeting HEALTHY goals. The most challenging barriers experienced by research dietitians and food service staff were costs, availability of foods and student acceptance.

The Los Angeles Unified School District Nutrition Network (LAUSDNN) is a school-based project that focused on improving nutrition and physical activity in more than 200 schools. Kratz et al. [19] reported on the process evaluation data for LAUSDNN and similarly noted the importance of strong relationships required with school staff to implement such an intervention. They stated that ‘effectiveness and institutionalization of the program might be positively affected by fostering local ownership, allowing school personnel (who apply for the grant) to tailor the program to their individual schools’.

Most recently, Siega-Riz et al. [20] reported a 10% higher intake (138 g or approximately two servings versus 122 g, respectively, P = 0.0016) of fruit among students in the HEALTHY intervention schools compared with students in the control schools.

In addition, Mobley et al. [17] presented data from the HEALTHY Study using the Nutrition Data System for Research (NDSR), a more rigorous approach to collecting and analyzing dietary data over a much longer time frame. NDSR utilized data collected on specific foods that were offered to students over a 21-day period at three time points of the study: at the beginning of sixth grade, before the intervention began; the second semester of seventh grade and in the second semester of eighth grade. Using this rigorous approach to evaluating intake (NDSR), the percentage of intervention schools that were able to meet the high fiber goals was lower than that detected by the process evaluation data. In NDSR, Mobley et al. [17] reported their data for the SBP and NSLP separately, and found that the high fiber goal was met 14% and 0% for SBP and NSLP, respectively, compared with our process evaluation data, where we found that the high fiber goal was met 64% of the time. Note that the process evaluation data were collected for SBP and NSLP combined, not for each one separately, which could be one cause for such a large discrepancy between the NDSR and the process evaluation data. Furthermore, the NDSR data are based on what the students actually chose, whereas the process evaluation data are based on what the intervention schools offered. In addition, at times, a school would state that a food was high fiber, but when analyzed in NDSR, it did not meet our high fiber goal of 2 g of fiber per serving. Despite the dissimilarity in rates, both sources of data show that the high fiber goal was the most difficult to attain.

The results of the observations and interviews were shared with HEALTHY staff (not school staff) at the conclusion of each semester. The results helped research dietitians highlight specific goals and strategies that required extra focus. Research dietitians worked directly with the food service managers and staff and used other tools to evaluate whether goals were being met. This helped research dietitians not only highlight specific goals that needed attention, but the feedback provided to food service staff was instrumental in improving and/or maintaining strong relationships between the research dietitians and food service staff. For example, the high fiber goal was observed as ‘not met’ more frequently than other goals. As a result, research dietitians worked with food service staff to help develop additional strategies to meet this goal. One specific strategy was a salad bar that was implemented in several HEALTHY schools. Food service staff mentioned an increase in labor (e.g. food preparation) as a concern for implementing changes during interviews. Research dietitians worked with food service staff to develop strategies that helped maintain changes, such as salad bars, while decreasing excess labor (e.g. food preparation).

Others have reported that introduction of new foods and changes in a school cafeteria did take some time; however, once changes were implemented, they were accepted by the food service employees [12, 16]. Reynolds et al. [12] reported that the nutritionists from their High-5 School-Based Nutrition Intervention anecdotally stated how important the degree of ownership and pride by the food service staff was in successful implementation of the intervention to improve the dietary habits of children.

Overall, the rate of fidelity discussed in this article is quite high. However, it must be noted that the number of observations was limited. Thus, the data have all the limitations associated with taking periodic snapshots of a process that varied daily. The primary use of the data was to help provide feedback to the research dietitians regarding the implementation of the nutrition goals and strategies.

Though the fidelity of the high fiber goal was not as high as expected, the fidelity and implementation of the rest of the nutrition goals were. Steckler et al. [21] reported that, in the Pathways study, students received about 93% of their classroom curricula, whereas the implementation of the food service behavioral guidelines began at 51% in the third grade and increased to 87% in the fifth grade. In the Trial of Activity for Adolescent Girls (TAAG), the authors reported, through their process evaluation data, that the intervention components delivered from the intervention staff to the teachers had a high fidelity (84–97%); while the delivery of the intervention protocol from the teachers to the students had a much lower and more variable range of 18–93% [22]. Although the Pathways study and the TAAG showed a wide range of fidelity (25–98% in Pathways and 18–95% in TAAG), both multi-site trials showed overall high fidelity rate with their process evaluation data. The authors confirmed that their process evaluation data demonstrated the ability to successfully deliver a school- and community-linked physical activity intervention with high fidelity, demonstrating the fact that the majority of the intervention was implemented with high fidelity.

Implications for practice

The process evaluation methods and results discussed in this article have several implications for school-based programs. First, the relationship between study staff and food service staff is critical to making any changes in the foods offered to students. Food service staff need to be educated about the rationale for making changes and should be included in decisions of how and what changes to make. Second, even with a strong relationship, the cost associated with making healthier changes to foods offered in cafeterias is still a critical barrier (perceived or real [18]). Results of this analysis showed it was difficult to find products that met intervention guidelines and could be purchased at an acceptable cost by schools. Third, student acceptance takes time. Food service staff mentioned negative reactions by students when changes were first introduced, but noticed students seemed to eventually accept the changes. Finally, the process evaluation methods discussed in this intervention can serve as an effective strategy to monitor the implementation of food service interventions in other schools.

The sustainability of changes is the most difficult to achieve in any study. This was a multi-center, multi-faceted study, which resulted in many positive changes; however, there were no significant differences in the main outcome measures between intervention and control schools [23]. Marcus et al. [24] discussed the lessons learned from the HEALTHY study. One of the main points pertaining to this publication and to inform future publications is that the secular trends occurring in schools to increase physical activity and improve nutrition, will affect any intervention, even one as robust as the HEALTHY study [23, 24] . Therefore, Marcus et al. [24] state that future researchers should pay closer attention ‘to assessing the impact of temporal efforts and campaigns to increase public awareness, influence health practices, and mandate environmental change to distinguish study impact from external forces on both control and intervention groups’.

Funding

The National Institute of Diabetes, Digestive, and Kidney Diseases (NIDDK) of the National Institutes of Health (NIH; grant numbers U01-DK61230, U01-DK61249, U01-DK61231 and U01-DK61223); the American Diabetes Association.

Conflict of interest statement

None declared.