Little Association Between Wellness Policies and School-Reported Nutrition Practices

Abstract

Background

The Child Nutrition and WIC Reauthorization Act of 2004 mandated written school wellness policies. Little evidence exists to evaluate the impact of such policies. This study assessed the quality (comprehensiveness of topics addressed and strength of wording) of wellness policies and the agreement between written district-level policies and school-reported nutrition policies and practices in 48 low-income Michigan school districts participating in the School Nutrition Advances Kids study.

Method

Written wellness policy quality was assessed using the School Wellness Policy Evaluation Tool. School nutrition policies and practices were assessed using the School Environment and Policy Survey. Analysis of variance determined differences in policy quality, and Fisher’s exact test examined agreement between written policies and school-reported practices.

Results

Written wellness policies contained ambiguous language and addressed few practices, indicating low comprehensiveness and strength. Most districts adopted model wellness policy templates without modification, and the template used was the primary determinant of policy quality. Written wellness policies often did not reflect school-reported nutrition policies and practices.

Conclusions

School health advocates should avoid assumptions that written wellness policies accurately reflect school practices. Encouraging policy template customization and stronger, more specific language may enhance wellness policy quality, ensure consistency between policy and practice, and enhance implementation of school nutrition initiatives.

INTRODUCTION AND BACKGROUND

Schools are an important setting to improve children’s nutrition and physical activity and prevent chronic diseases such as heart disease and diabetes (American Dietetic Association, 2006; Michigan Department of Education, Michigan Department of Community Health, The Governor’s Council on Physical Fitness Health and Sports, & The Michigan Fitness Foundation, 2001; Story, Kaphingst, & French, 2006). The Child Nutrition and WIC Reauthorization Act of 2004 required all school districts receiving federal funding for school meals to establish a local wellness policy by July 1, 2006 (Section 204 of Pub. L. No. 108-265). Wellness policies were required to include the following: goals for nutrition education, physical education, and physical activity; nutrition guidelines for school meals that meet or exceed U.S. Department of Agriculture requirements; nutrition guidelines for all other foods available on campus (i.e., competitive foods); a plan for measuring implementation of the wellness policy; and involvement of key stakeholders (e.g., parents, students, school food authority, administration, school board, and the public) in policy development and communications (Section 204 of Pub. L. No. 108-265). Recognizing the unique circumstances of each school district, districts were allowed to tailor their policy to their needs. However, this lack of structure, in addition to no funding or enforcement of this mandate, resulted in a high degree of variability in the quality and implementation of such policies (Institute of Medicine, 2007; Story, Nanney, & Schwartz, 2009). The Healthy, Hunger-Free Kids Act of 2010 updates this legislation and includes provisions for technical assistance and requires evaluation of wellness policy adherence (Section 204 of Pub. L. No. 111-296).

Prior to the federal mandate, fewer than half of U.S. school districts had adopted policies to promote healthy eating and physical activity (Finkelstein, Hill, & Whitaker, 2008; Greves & Rivara, 2006; O’Toole, Anderson, Miller, & Guthrie, 2007). After the federal mandate took effect, nearly all districts had adopted a policy (Belansky et al., 2009; Chriqui et al., 2010; Chriqui et al., 2013; Longley & Sneed, 2009; Metos & Nanney, 2007; Moag-Stahlberg, Howley, & Luscri, 2008; Molaison et al., 2011; Probart, McDonnell, Weirich, Schilling, & Fekete, 2008; School Nutrition Association, 2006); however, policy language was often ambiguous, making it difficult to interpret, implement, and evaluate effectiveness (Belansky et al., 2009; Chriqui et al., 2010; Chriqui et al., 2013; Metos & Nanney, 2007; Probart et al., 2008). The purposes of this study were the following: (a) to describe the quality (comprehensiveness of topics addressed and strength of wording) of school district wellness policies, (b) to examine differences in wellness policy quality, and (c) to determine whether district-level written wellness policies reflect school-reported nutrition policies and practices.

METHOD

Participants

This study used cross-sectional analysis of baseline data collected through the School Nutrition Advances Kids (SNAK) intervention study. The SNAK project, included partners from Michigan State University (MSU), the Michigan Department of Education, and several partnering organizations of the Michigan Action for Healthy Kids coalition. The SNAK project aimed to improve school nutrition education, environments, policies, and student dietary behavior through Michigan’s Healthy School Action Tools self-assessment and action planning process (mihealthtools.org) and implementation of a Michigan State Board of Education nutrition policy. All study procedures and instruments were approved by the MSU institutional review board.

Schools were recruited to participate in fall 2007 (n = 32) and fall 2008 (n = 33) through an application for small-grant funding. Eligibility criteria included Michigan middle schools (seventh and eighth grades in the same building) with 50% or more of students eligible for free or reduced-price meals. Recruitment methods included direct mailings, e-mails, and phone calls to eligible schools and a posting on the Michigan Team Nutrition website. A total of 65 schools from 50 districts participated in the SNAK study. Wellness policies are typically written at the district level; thus, districts with multiple school buildings participating (5 districts, 15 schools) had one school randomly selected to represent school-reported nutrition practices. Two districts were not able to locate a written wellness policy, giving a final sample size of 48 school/district pairs.

Instruments and Procedures

School Wellness Policy Evaluation Tool

Policy quality was quantified using the School Wellness Policy Evaluation Tool (Schwartz et al., 2009), which contains 96 items within seven sections corresponding to the federal wellness policy requirements (Robert Wood Johnson Foundation Healthy Eating Research Program et al., 2008). Each item received 0 points if it was not addressed; 1 point if it was addressed, but the statement was suggestive or vague (e.g., schools should provide an adequate amount of time for lunch); and 2 points if the statement was specific and required (e.g., schools will provide at least 20 minutes daily to eat lunch). To collapse responses into dichotomous “yes/no” variables, items rated 1 or 2 points were designated a “yes,” and items receiving 0 points were designated a “no.” Each section, and the assessment as a whole, received a comprehensiveness score (percentage of items receiving 1 or 2 points) and a strength score (percentage of items receiving two points). Evaluation of the tool indicated that it has adequate internal consistency and interrater reliability (mean interrater reliability for total comprehensiveness and total strength scores intraclass correlation coefficient = .82; Schwartz et al., 2009). Two researchers independently scored each policy, and discrepancies between coders were reconciled with a third researcher. A comprehensive set of rules was developed, and all policies were rescored to ensure consistency.

The policies used by each district were categorized by the template used to develop policy. Templates were example documents created by various organizations to assist school districts in wellness policy development. Policies were divided into three major categories: (a) MASB: the Michigan Association of School Boards (MASB)–recommended policy template, which was developed by the Michigan Department of Education in collaboration with other state and local organizations, agencies, and citizens (n = 21); (b) Policy Company: a template developed by a company schools hired to manage their board policies (n = 21); and (c) Other: includes four schools that did not follow a recognizable policy template and two schools that used the National Alliance for Nutrition & Activity template (n = 6). The MASB policies were further categorized based on how districts modified the template—shortened, left as intended, or enhanced.

School Environment and Policy Survey

Within the 48 middle schools, 41 administrators (building-level principal or assistant principal) and 46 food service directors (FSDs) or managers completed the School Environment and Policy Survey (SEPS), which assessed school building–level wellness policies and practices. The first cohort of schools completed the SEPS between January and March 2008 and the second cohort between November 2008 and March 2009. School personnel completed the survey either on paper or online and received a $25 gift card incentive. Follow-up phone calls, e-mails, and mailings encouraged survey completion. Overall response rates were 85% for administrators and 91% for FSDs.

The SEPS assessed middle school building-level nutrition and physical activity policies and practices. It contained three modules with unique expertise-specific questions completed by school administrators, food service operations, and physical education teachers. Each module took approximately 30 minutes to complete. The SEPS was developed by Dr. Elaine Belansky and the Rocky Mountain Prevention Research Center for elementary schools, and preliminary validation findings suggest little reporting bias (Belansky et al., 2009). The SEPS was adapted for use in Michigan middle schools based on a literature review, best practice recommendations for schools, and experience working with middle schools. The adapted SEPS was reviewed by project team members with various types of expertise related to school nutrition and by several school food service and administration representatives to establish face and content validity.

Nutrition-related variables from the SEPS survey that corresponded to items in the Wellness Policy Evaluation Tool were used in the current analysis. From the administrator SEPS survey, five school nutrition policy variables were developed: prohibiting use of food as a reward or punishment, stipulating predominantly healthy foods in vending machines, a la carte, fund-raising, and class parties. To create dichotomous (yes/no) variables, these five policy items were categorized as “no” for “no written policy” or “written policy not enforced” response options and as “yes” for “unwritten policy always enforced” or “written policy sometimes or always enforced” response options. Four dichotomous school nutrition practice variables were created from the administrator SEPS survey: coordinate nutrition education throughout school (“yes” if reporting any of the following practices: posters encouraging healthy eating found throughout the school, bulletin boards feature healthy eating information, school announcements include messages about healthy eating), teachers role model healthy eating, integrate nutrition education into broader curriculum, and presence of a coordinated school health team prior to the starting the SNAK project. Six dichotomous nutrition practice variables were created from the FSD SEPS survey: having breakfast available, serving low-fat options in school meals (“yes” if at least 1 day/week), encouraging participation in school meals (“yes” if reporting any of the following practices: offering miniservings of new healthy foods, offering taste tests, providing incentives for school lunch, announcing the menu, discarding damaged produce, displaying foods in a way that is visually appealing, or surveying students about foods, beverages, preferences, time to eat, or their general opinion of food service), providing adequate time to eat lunch (“yes” if an average of 15 minutes or more to eat after students are served), providing training/education opportunities for food service staff, and holding of a nutrition-related degree by the FSD.

Data Analysis

All analyses were performed using Stata statistical software (Stata Corporation, Release 10.0, College Station, Texas, 2008). Analysis of variance was used to examine differences in wellness policy quality based on the policy template type. Fisher’s exact test (onesided) was used to evaluate degree of agreement between written wellness policies and the SEPS-reported school nutrition policy and practice variables. Districts with missing data for a single variable were removed from individual analyses.

RESULTS

The 48 schools included in this analysis had an average of 136 seventh-grade students (range: 23–431) and an average building enrollment of 473 students (range: 149–1217). In all, 54% of schools were located in urban settings, 27% were rural, and 19% were suburban. The majority of schools were public (noncharter; 83%), and 71% had >50% minority population. In all, 25% of schools used the same food service management company. Schools had an average of 66% of students eligible for free or reduced-price school meals (range: 50% to 98%).

Average comprehensiveness and strength of wellness policies for each section and the overall policy are shown in Table 1. Wellness policies addressed 40% of items (comprehensiveness score), whereas only 19% of items had specific and required strategies (strength score). The nutrition education section (comprehensiveness: 62%; strength: 31%) scored the highest, and the nutrition standards for competitive food section (comprehensiveness: 33%; strength: 5%) scored lowest.

TABLE 1.

Mean Wellness Policy Comprehensiveness and Strength Scores by Section and Template Type Used Among SNAK School Districts

Wellness Policy Section	All Policies (n = 48), M % [95% CI]	MASB Policies (n = 21), M % [95% CI]	Policy Company Policies (n = 21), M % [95% CI]	NANA + Other Policies (n = 6), M % [95% CI]
Nutrition education
Comprehensivenessa,b	62.0 [55.4, 68.7]	73.6 [66.1, 81.0]	58.7 [52.5, 64.9]	33.3 [2.8, 63.9]
Strength	31.5 [27.9, 35.1]	33.86 [30.6, 37.1]	32.3 [28.2, 36.3]	20.4 [−1.5, 42.3]
School meals
Comprehensiveness	35.1 [28.9, 41.3]	34.1 [24.6, 43.5]	32.6 [27.3, 37.9]	47.4 [14.2, 80.7]
Strength	21.2 [16.4, 25.9]	18.3 [11.5, 25.1]	20.51 [16.2, 24.8]	33.3 [7.5, 59.1]
Competitive foods
Comprehensivenessa	33.2 [25.7, 40.5]	48.9 [39.3, 58.6]	15.4 [8.7, 22.2]	40.2 [14.9, 65.6]
Strengtha	5.5 [1.8, 9.1]	6.1 [2.9, 9.3]	0 (n/a)	22.4 [−1.3, 46.1]
Physical education
Comprehensivenessa,b	35.4 [29.8, 41.1]	46.8 [37.6, 56.0]	28.6 [24.7, 32.4]	19.6 [2.4, 36.8]
Strengtha	24.1 [20.3, 28.0]	30.8 [24.9, 36.8]	19.6 [16.2, 23.0]	16.7 [1.6, 31.8]
Physical activity
Comprehensiveness	45.2 [37.4, 53.0]	53.3 [42.1, 64.6]	40.0 [30.5, 49.5]	35.0 [1.0, 69.0]
Strength	29.4 [23.5, 35.2]	37.1 [29.4, 44.9]	23.3 [15.7, 31.0]	23.3 [−1.4, 48.1]
Communication
Comprehensivenessa	42.4 [35.4, 49.4]	54.4 [43.4, 65.3]	34.1 [27.5, 40.8]	29.2 [3.7, 54.7]
Strengtha	25.9 [19.0, 32.8]	40.1 [30.1, 50.1]	12.7 [7.2, 18.2]	22.2 [−4.3, 48.8]
Evaluation
Comprehensiveness	54.9 [47.5, 62.3]	61.9 [50.1, 73.7]	54.3 [47.5, 60.5]	33.3 [0.9, 65.7]
Strength	12.5 [7.5, 17.5]	11.9 [7.8, 16.0]	8.7 [5.0, 12.5]	27.8 [−7.6, 63.1]
Total
Comprehensivenessa	40.3 [34.5, 46.1]	50.8 [41.8, 59.8]	31.5 [27.1, 35.8]	34.6 [9.9, 59.2]
Strengtha	18.8 [15.2, 22.4]	22.6 [18.3, 26.8]	13.8 [11.5, 16.1]	23.1 [0.4, 45.8]

NOTE: SNAK = School Nutrition Advances Kids; CI = confidence interval; MASB = Michigan Association of School Boards; NANA = National Alliance for Nutrition & Activity.

^a= MASB significantly > Policy Company.

^b= MASB significantly > NANA + Other.

Nearly all districts (44 out of 48) used a recognizable template for their policy, and template type was the primary determinant of policy quality. Districts using the MASB template had significantly higher comprehensiveness and strength than districts using other templates (Table 1). When districts shortened the MASB template (8 of 21 schools using the MASB template), their policies scored lower in both comprehensiveness and strength (results not shown). For policies within the other template category, the National Alliance for Nutrition & Activity–based policies generally had higher than average scores, and the policies that did not use a recognizable template were shorter and scored lower than average (results not shown).

Items from the School Wellness Policy Evaluation Tool were compared with matched items from the SEPS survey to determine the similarity between written and school-reported policies and practices. The percent of concordant policies and practice items (those similarly classified in the SEPS survey and in the written wellness policy as either yes/yes or no/no pairs) ranged from 9% to 71% (Table 2). The only variable with significant concordance (indicating a deviance from the null hypothesis of a random distribution across quadrants) was a policy regarding healthy foods in fundraising activities (71% similarly classified, p = .01).

TABLE 2.

Concordance Between Written Wellness Policies and Reported Nutrition Policies and Practices

Policy or Practice	Included in Written Wellness Policy	SEPS-Reported Practice		% Similarly Classified	Fisher’s p (One-Sided)
		No	Yes
Administrator-reported school nutrition policies
Prohibits use of food as a reward (n = 37)	No	8	14	49	.57
	Yes	5	10
Healthy food in vending machines (n = 21)	No	2	4	67	.45
	Yes	3	12
Healthy food in a la carte (n = 33)	No	4	4	67	.23
	Yes	7	18
Healthy food in fund-raising (n = 38)	No	18	3	71	.01
	Yes	8	9
Healthy food in class parties (n = 40)	No	11	4	50	.40
	Yes	16	9
Administrator-reported school nutrition practices
Coordinates nutrition education throughout school (n = 41)	No	0	8	66	.25
	Yes	6	27
Teachers role model healthy eating (n = 41)	No	6	3	44	.57
	Yes	20	12
Integrate nutrition education into broader curriculum (n = 41)	No	8	7	51	.55
	Yes	13	13
Presence of a coordinated school health team (n = 41)	No	7	12	56	.37
	Yes	6	16
Food service director–reported school nutrition practices
Breakfast available (n = 46)	No	0	34	26	NA
	Yes	0	12
Serving low-fat options in school meals (n = 45)	No	2	41	9	.91
	Yes	0	2
Strategies to encourage participation in school meals (n = 46)	No	3	40	13	.81
	Yes	0	3
Adequate time to eat lunch (n = 46)	No	4	18	54	.45
	Yes	3	21
Training provided for food service staff (n = 46)	No	7	12	46	.32
	Yes	13	14
FSD has a nutrition-related degree (n = 46)	No	20	14	50	.26
	Yes	9	3

NOTE: SEPS = School Environment and Policy Survey; FSD = food service director.

DISCUSSION

There is room for improvement in the quality of written school wellness policies. District wellness policies scored low in strength and comprehensiveness in most sections and for the overall assessment. Other studies using the same instrument also report low strength and comprehensiveness scores (Belansky et al., 2009; Chriqui et al., 2013; Schwartz et al., 2012). The most recent of these studies reports improvements with mean comprehensiveness of 48% and mean strength of 28% over the 5 years the wellness policy mandate has been in effect (Chriqui et al., 2013). The current study likewise found nutrition education to be the highest scoring section and nutrition standards for competitive foods to be the lowest scoring section (Chriqui et al., 2013). The Healthy Hunger Free Kids Act of 2010 required development of nutrition standards for competitive foods in schools participating in the National School Lunch Program (Food and Nutrition Service, U.S. Department of Agriculture, 2013). These standards went into effect July 1, 2014, and it is likely that wellness policy competitive food section scores will improve as a result.

The low strength scores are a result of the typical policy wording that included suggestive statements such as “shall offer and promote healthy foods in all venues,” which has also been found in other studies (Probart et al., 2008; Story et al., 2009). This ambiguous language can make implementation of wellness policy provisions difficult. The template policies that most schools are adopting without modification typically include this weak language. Districts may also intentionally keep written policies vague so that each building can tailor the policy to their specific needs or for fear of auditing of wellness practices. Policies may also be accompanied by a procedure manual (which may be deemed less restrictive for districts in which a lengthy and expensive process is required to change policy-related documents) outlining more specific requirements to implement the policy.

This is the first study to examine how templates influence wellness policy strength and comprehensiveness, and we found significant differences in quality based on template type used to create the policy. The MASB policy was developed and endorsed by a diverse coalition led by the Michigan Department of Education, which may have contributed to its higher quality. Previous studies indicate that the majority of school districts adopt the state-recommended wellness policy template (Belansky et al., 2009; Probart et al., 2008; Taylor, Francony, Beiting, Ritter, & Clutter, 2011), and in the current study, nearly half did so. Encouraging states to develop or recommend high-quality templates may be an effective tactic for enhancing wellness policy quality.

Recently, studies have attempted to determine the degree of implementation of written wellness policies, with mixed findings. The current study examined cross-sectional associations between written policies and matching school-reported policies and practices and found little concordance between the two. A similar study found extreme variations in concordance between fund-raising policies and practices, ranging from 15% to 68% (Kubik, Lytle, Farbakhsh, Moe, & Samuelson, 2009).

Several studies have found that wellness policy quality was associated with implementation. In Connecticut, wellness policy strength scores combined with a program that financially rewarded schools for implementing nutrition standards were associated with lower availability of unhealthy foods outside of school meals (Friedman, 2009). Another recent study found that higher wellness policy scores were associated with the healthfulness of foods and beverages in competitive food venues (Hood, Colabianchi, Terry-McElrath, O’Malley, & Johnston, 2013); however, both wellness policy provisions and availability of food and beverage items were self-reported by school personnel in this study (Hood et al., 2013). Another study found that written wellness policy quality was associated with administrator-reported implementation 1 year later (Schwartz et al., 2012). In contrast, one study found little change in physical activity provisions or school nutrition environments after wellness policy adoption (Belansky et al., 2009; Belansky et al., 2010). Another study found that wellness policy strength scores did not predict perceived implementation of nutrition standards in meals or competitive foods (Wall, Litchfield, Carriquiry, McDonnell, & Woodward-Lopez, 2012). With some wellness policies lacking a time line for implementation or details regarding evaluation, it is not surprising that implementation has lagged (Action for Healthy Kids, 2007; Gaines, Lonis-Shumate, & Gropper, 2011; Moag-Stahlberg et al., 2008).

Many potential explanations of the disagreement between written and school-reported policies and practices exist. Wellness policies are written at the district level and may not reflect the variability of policies and practices within each individual school building. Additional relevant health policies could be located in other documents such as staff and student handbooks. Thus, policy research should consider all available written policy documents. Administrators and FSDs may be unaware of some school policies and practices and thus fail to report them when surveyed. The lack of agreement in FSD-reported practices could indicate that FSDs are not included in wellness policy development and dissemination.

Alternatively, districts could have intentionally adopted wellness policies that include goals they hope to implement in the future. Written wellness policies may not have been implemented due to a variety of barriers. In a 2007 survey of Michigan school districts regarding wellness policy implementation, 86% reported that no changes had occurred or that it was too early to determine any changes (Michigan Department of Education, 2007). Seventy-seven percent of schools cited at least one barrier to implementing their policy, including lack of funding or staff time, no one in charge of implementation, and no system to track implementation (Michigan Department of Education, October 2007). In a national study, FSDs reported barriers to wellness policy implementation including the need to use sales of food as fund-raisers and a lack of time by administrators and teachers due to the No Child Left Behind Act (Longley & Sneed, 2009). Despite these barriers, these schools had significant improvements in implementation of nutrition components following the wellness policy mandate (Longley & Sneed, 2009).

Another recent study found that schools reporting a lack of coordination and lack of resources as barriers had significantly lower wellness policy implementation (Schwartz et al., 2012). Another plausible explanation supported by the current research is that districts are likely adopting a wellness policy template verbatim without regard for their district’s current or future practices. During policy analysis, it was observed that some schools failed to insert their district name where indicated. Furthermore, one template consisted of a checklist of statements from which districts could choose, with many statements being nearly identical. Some districts simply checked every box in the template without regard to the intention for the district to pick the statement that most closely reflected their practices. Although guidance was clearly necessary to aid school districts in creating wellness policies, the widespread availability of policy templates may have allowed districts to simply adopt a template, missing the spirit of collaboration, education, and personalization that was intended (Story et al., 2009).

This is one of the first studies to evaluate the degree of agreement between written wellness policies and school-reported nutrition policies and practices (Belansky et al., 2009; Kubik et al., 2009). Use of a validated tool to measure wellness policy quality adds strength; however, the lack of validated and reliability-tested instruments to assess school nutrition practices is a limitation of school-based nutrition research. Policy and practice data were self-reported by administrators and FSDs and may differ if additional school staff were surveyed. The nonrandom sample recruited through a small grant application may have resulted in self-selection bias. The study population included only low-income Michigan middle schools and may not be reflective of all school districts.

CONCLUSIONS

The federal wellness policy mandate has been successful at increasing awareness of the important role schools play in promoting health and well-being to students, staff, families, and the community. However, there is clearly room for improvement in the quality of policies and the translation of policies into health-promoting school practices. The Healthy, Hunger-Free Kids Act of 2010 provides an opportunity to address some of these shortcomings. The proposed rule accompanying this Act calls for increased school leadership, public participation, transparency, and evaluation of implementation of wellness policy initiatives. In essence, the wellness policy mandate was intended to stimulate a collaborative method in which a diverse team of stakeholders jointly determined which approaches complement their school’s unique circumstances to promote health. The widespread adoption of template policies in this study indicates that schools are not using this approach. Although policy templates are a valuable tool that can give schools a starting ground and ideas for best practices, encouraging schools to personalize template policies in order to reflect current practices and mutually agreed on goals is critical. Incentivizing a collaborative approach to developing wellness policies may be necessary. For the wellness policy mandate to be effective, technical assistance and best practice recommendations must be available and easily accessible to schools. Further research to determine how schools can successfully develop and implement wellness policies, and the barriers to this process, is necessary to ensure the effectiveness of this mandate.