Exploring the cross-sectional relationship between implementation supports for adopting nutrition standards and the nutritional quality of school lunches served

Abstract

Background:

The evidence-based nutrition standards of the Healthy, Hunger-Free Kids Act (HHFKA) of 2010 significantly improved school meal nutrition, yet little is known about how school food authorities (SFAs) were supported to translate the standards into practice in schools.

Objective:

This study tested whether two implementation supports, SFA receipt of training/technical assistance (TA) and purchase of new equipment, for implementing the nutrition standards were associated with the nutritional quality of school lunches.

Design:

This study used a cross-sectional design.

Participants/setting:

The study sample included 365 SFAs derived from the U.S. Department of Agriculture’s School Nutrition and Meal Cost Study (2014-15 school year), the only national data of school nutrition environments since the HHFKA.

Main outcome measures:

Implementation supports included reported participation in training/TA and the purchase of new equipment between the 2012-13 (year of policy adoption) and 2014-15 school years. The primary outcome, nutritional quality of school lunches served, was defined as low/high Healthy Eating Index (HEI) 2010 scores during the 2014-15 school year. Secondary implementation supports included the number of areas covered by training/TA, the adequacy of training/TA, and the degree of implementation challenges.

Statistical analyses:

Multivariable-adjusted logistic regression models assessed if receipt of implementation supports was associated with lunch HEI-2010 scores.

Results:

The median lunch HEI-2010 score was 81.7 (95% CI: 80.4, 82.9). Most SFAs (78.4%, 95% CI: 72.0%, 85.0%) reported having participated in training/TA and one third (33.8%, 95% CI: 24.4%, 43.2%) reported having purchased new equipment—neither were associated with the odds of having higher HEI-2010 scores for lunches served at the time of data collection.

Conclusions:

Many SFAs accessed implementation supports to adopt the nutrition standards. Information on dose, quality, and nutritional impacts of implementation supports using measures of change are needed to determine how best to support SFAs with implementation of new nutrition standards.

INTRODUCTION

More than 80,000 public schools in the United State (U.S.)¹ during the 2012-2013 school year were required to adopt the U.S. Department of Agriculture’s (USDA) nutrition standards for school meals required by the Healthy, Hunger-Free Kids Act (HHFKA) of 2010.² These standards, informed by nutrition science, emphasized more fruits, vegetables, and whole grains, fewer refined grains and starchy vegetables, and less sodium and added sugar.² The HHFKA was a significant policy change for schools participating in the National School Lunch Program as this was the second comprehensive revision of school lunch nutrition standards³ and it offered the potential to improve school meal nutrition for more than 30 million schoolchildren.⁴

Multiple studies since this time have indicated the HHFKA improved nutrition outcomes for students, including increased selection of fruit⁵ and improved diet quality.⁶ However, none have applied an implementation science lens⁷ to consider differences in how schools nationally were supported to implement the HHFKA nutrition standards, and whether differences in implementation might be associated with helping schools offer more nutritious school meals. Implementation strategies, including training and/or equipment, could have been critical to successful implementation considering the shifts in food service practices for menu planning (e.g., food purchasing, recipes) and food preparation (e.g., scratch cooking, food safety) prompted by the HHFKA nutrition standards, particularly given the substantial need for implementation supports prior to HHFKA’s implementation. At the start of the 2012-13 school year when the revised HHFKA nutrition standards went into effect, the Kids’ Safe & Healthful Foods Project⁸ issued a report that described training and equipment needs among a nationally representative sample of school food authorities (SFAs), the entities responsible for administering the National School Lunch Program for one or more schools. Nearly 60% of SFAs reported having received training, though most still reported the need for training especially for SFAs in suburban areas and the Southeast.⁸ Nearly all SFAs reported needing new equipment to implement the HHFKA standards despite previous federal investments in school kitchen infrastructure through the American Recovery and Reinvestment Act of 2009.⁹ Equipment needs were lowest for smaller SFAs and those in the West⁹ though the reasons for this are unknown since this is an understudied topic.

Aside from supporting feasibility of policy implementation, implementation supports like training and equipment could also contribute to school meal nutrition.^8-11 Training for school food service directors¹² and partnerships with chefs¹³ have been suggested as strategies for improving school meal nutrition though evidence is limited. Kitchen equipment and design have also been proposed as pathways for promoting nutrition^14,15 but evidence is limited—the impacts of salad bars in schools, for example, on student nutrition is mixed.^16,17

The present study applied Proctor’s model for implementation outcomes⁷ to adoption of the HHFKA nutrition standards in a national sample of SFAs. In the absence of pre- and post- data that would have allowed for a study of change in nutritional quality, study aims were to assess the cross-sectional relationship between SFA-reported past participation in training/technical assistance (TA) and prior purchase of equipment to implement the HHFKA nutrition standards between the 2012-13 school year (when schools were required to adopt the standards) and 2014-15 school year (when data were collected), and the study outcome of school lunch nutrition quality (i.e., the goal of the HHFKA). The study hypothesis was that participation in training/TA or equipment purchases between the 2012-13 and 2014-15 school years would be associated with school lunches of higher nutritional quality. Secondary analyses explored associations between additional implementation outcomes and nutritional quality.

METHODS

Study Design and Population

This study analyzed data from the USDA School Nutrition and Meal Cost Study (SNMCS),¹⁸ a cross-sectional study of U.S. public schools during the 2014-15 school year. The SNMCS is the only nationally representative, comprehensive assessment of school nutrition practices and operations since the adoption of the HHKFA nutrition standards.

The SNMCS team sampled across three groups of SFAs and then up to six schools within select SFAs. The present study utilized data from 518 SFAs who completed SFA Director Surveys and 1,207 schools who completed Menu Surveys. Because the authors were interested in use of implementation supports at the SFA level, relevant school-level observations in the Menu Surveys were averaged and merged with data for SFAs that also completed SFA Director Surveys to reach an analytical sample of 365 SFAs. It is possible the averaged SFA-level data may not be representative of all schools in the SFA; however, these were the only national data available to utilize for the present implementation study.

Secondary analyses were conducted for a subset of 272 SFAs that reported participating in training/TA. Further details on design and measures can be found in the SNMCS Study Design, Sampling, and Data Collection report,¹⁸ SNMCS Volume 1 for SFA Director Surveys,¹⁹ and SNMCS Volume 2 for Menu Surveys.²⁰

This study was classified by the Institutional Review Board at the Harvard T.H. Chan School of Public Health as non-human subject research and as such was exempt from review.

Study Measures

This study explored cross-sectional relationships between implementation outcomes (defined as outcomes of implementation supports for implementation processes) and service outcomes (defined as outcomes for the quality of the intervention delivered) by mapping SNMCS data onto Proctor’s model for implementation outcomes (see Figure 1 for conceptual model).⁷

Figure 1.

Conceptual model of implementation support received by school food authorities between the 2012-13 and 2014-15 school years and the nutritional quality of school lunches served, informed by Proctor’s implementation outcomes.⁷ Outcomes measured in the present study are displayed in blue boxes. Based on the available School Nutrition and Meal Cost Study data,^19,20 the authors chose to define implementation outcomes as the degree of implementation support received. Service outcomes were defined as the quality of the service provided to students. ^aTA = technical assistance. ^bSFA = School Food Authority.

Implementation outcomes.

The authors leveraged data from SFA Director Surveys issued by SNMCS in 2014-15 to assess SFA operations including the implementation supports SFAs used to implement the HHFKA nutrition standards. The present study’s primary independent variable was the “adoption” of implementation supports including: 1) Past participation in training/TA: SFA directors indicated (yes/no) if they had received any types of food services operations training/TA to implement the HHFKA nutrition standards since the 2012-13 school year; and 2) Prior purchase of equipment: SFA indicated (yes/no) if they had purchased equipment in order to implement the HHFKA nutrition standards since the 2012-13 school year.

Secondary implementation outcomes included exploratory measures of “adoption” and “acceptability” of implementation supports, as well as “feasibility” of implementing the HHFKA nutrition standards. The authors defined “adoption” as a training/TA score: SFA directors that reported participating in training/TA selected up to 16 areas for which training/TA was received (e.g., menu planning, food production, food purchasing). The authors summed the number of areas covered by training/TA received to create training/TA score for each SFA. “Acceptability” was measured with the SNMCS variable of adequacy of training/TA: SFA directors that received training/TA rated the adequacy of food services operations training/TA received using a response scale from one (not at all adequate) to five (more than adequate). “Feasibility” was assessed with a challenges score: SFA directors rated the degree to which seven potential barriers (e.g., understanding meal requirements, food costs) were currents challenge to implementation using a scale from one (not a challenge) to five (a significant challenge). The authors assigned each SFA a challenges score by summing the number of current challenges rated as “somewhat of a challenge” to “a significant challenge.”

Details on training/TA topics offered, types of challenges reported, and types of equipment purchased by SFAs in the SNMCS sample have been published elsewhere.¹⁹

Service outcomes.

The primary service outcome of “effectiveness” was defined as the nutritional quality of school lunches served to measure progress towards meeting the HHFKA’s goal of improving school meal nutrition. The USDA’s Healthy Eating Index 2010 (HEI-2010) score²¹ was selected as the measure of nutritional quality to reflect alignment with the 2010 Dietary Guidelines for Americans (DGAs).²² School nutrition managers within selected SFAs completed online SNMCS Menu Surveys to provide five days of lunch menus served during the 2014-15 school year. Food and beverage items were linked to food components in the USDA’s Food and Nutrient Database for Dietary Studies²³ and in the Food Pattern Equivalent Database 2017-18 per 100 grams.²⁴ These data were used to calculate HEI-2010 scores²¹ across 12 components (total fruit, whole fruit, total vegetables, greens and beans, whole grains, dairy, total protein foods, seafood and plant proteins, fatty acids, refined grains, sodium, and empty calories) averaged over the five days of lunch menus using the HEI simple scoring method algorithm.²⁵ HEI-2010 scores range from 0 to 100 where a score of 100 reflects total alignment with the DGAs.^21,22 The authors averaged school-level HEI-2010 scores by SFA and then coded each SFA as having either a lower or higher (i.e., below vs. at or above the median of 81.7) HEI-2010 score for the study outcome.

SFA sociodemographic characteristics.

SFA sociodemographic characteristics available for public use in the SNMCS data included: region (West, Midwest, Mid-Atlantic, Mountain, Northeast, Southeast, Southwest) from the USDA Food and Nutrition Service SFA Verification Summary Report 2012-13;¹⁸ poverty level (less than 20% of students eligible for free- or reduced-priced lunch, 20% or more) from school district files of the 2011 U.S. Census Bureau’s Small Area Income and Poverty Estimates;²⁶ size (less than 1,000 students, 1,000 to 5,000 students, more than 5,000 students served by the SFA) from the Common Core of Data 2011-12²⁷); share of students from racial/ethnic minorities (less than 20%, 20% or more) from the Common Core of Data 2011-12²⁷); and urbanicity (urban, suburban, rural) from the Common Core of Data 2011-12²⁷).

Statistical Analysis

Descriptive statistics were used to summarize participation in implementation supports, school lunch nutritional quality and SFA sociodemographic characteristics using PROC SURVEYFREQ and PROC SURVEYMEANS. Statistical significance levels were set at p<0.05. Significance testing was conducted in SAS version 9.4²⁸ and utilized sample weights provided by SNMCS.

Primary analyses assessed the cross-sectional association between SFA past participation in training/TA and prior purchase of equipment (“adoption”), and the “effectiveness” outcome of having a higher HEI-2010 score for lunches served using logistic regression methods (PROC SURVEYLOGISTIC), controlling for potential confounders associated with inequities in the public school system, including SFA size, region, share of students from racial/ethnic minorities, poverty level, and urbanicity. Secondary analyses tested for associations between SFA training/TA scores (“adoption”), challenges scores (“feasibility”), and adequacy of training/TA (“acceptability”) with the outcome of having a higher HEI-2010 score for lunches served (“effectiveness”) controlling for the same sociodemographic confounders.

RESULTS

Survey-weighted SFA sociodemographic characteristics can be found in Table 1. The sample included representation from SFAs in all regions, ranging from 6.3% in the Mid-Atlantic (95% CI: 3.9%, 8.7%) to 25.7% in the Midwest (95% CI: 18.5%, 32.8%). More than half of SFAs were located in rural areas (54.0%, 95% CI: 44.6%, 63.5%).

Table 1.

Sociodemographic characteristics of a national sample of school food authorities derived from the School Nutrition and Meal Cost Study^19,20 of the 2014-15 school year

Characteristic	Total sample (n=365), weighted % (95% CI)	SFAsa that accessed training/TAb (n=272), weighted % (95% CI)	SFAs that purchased new equipment (n=143), weighted % (95% CI)
Region
Midwest	25.7 (18.5, 32.8)	20.8 (14.0, 27.6)	28.1 (14.9, 41.3)
Mountain	19.7 (10.8, 28.6)	22.4 (11.5, 33.3)	23.6 (4.9, 42.3)
Southwest	16.1 (9.0, 23.2)	17.5 (9.1, 26.0)	9.7 (2.6, 16.7)
Northeast	12.1 (5.1, 19.2)	11.5 (2.8, 20.3)	9.8 (2.8, 16.8)
West	11.5 (4.8, 18.1)	13.1 (4.7, 21.6)	7.2 (3.6, 10.8)
Southeast	8.7 (4.7, 12.7)	8.6 (3.9, 13.4)	14.1 (3.7, 24.6)
Mid-Atlantic	6.3 (3.9, 8.7)	6.0 (3.3, 8.7)	7.5 (2.4, 12.6)
Size
Less than 1,000 students	43.4 (36.5, 50.4)	47.7 (39.6, 55.8)	39.0 (21.5, 56.5)
1,000 to 5,000 students	41.6 (34.3, 48.9)	40.0 (31.7, 48.4)	43.6 (28.9, 58.3)
More than 5,000 students	15.0 (12.5, 17.6)	12.3 (9.5, 15.1)	17.4 (10.9, 23.9)
Poverty level
Less than 20%	56.6 (49.2, 64.1)	52.7 (43.9, 61.5)	57.0 (40.8, 73.2)
20% or more	43.4 (35.9, 50.8)	47.3 (38.5, 56.1)	43.0 (26.8, 59.2)
Share of students from racial/ethnic minorities
Less than 20%	40.9 (31.7, 50.1)	40.5 (29.5, 51.6)	35.1 (21.3, 49.0)
20% or more	59.1 (49.9, 68.3)	59.5 (48.4, 70.5)	64.9 (51.0, 78.7)
Urbanicity
Rural	54.0 (44.6, 63.5)	55.3 (43.9, 66.7)	44.4 (29.2, 59.7)
Suburban	39.0 (29.6, 48.4)	37.5 (26.1, 48.9)	44.6 (28.8, 60.4)
Urban	7.0 (4.3, 9.7)	7.2 (3.8, 10.5)	10.9 (4.0, 17.9)

Notes: ^aSFA = school food authority. ^bTA = technical assistance.

The average SFA HEI-2010 score for lunches served was 81.3 (95% CI: 80.3, 82.3). Average HEI-2010 scores differed by SFA region and size (data available upon request). Since the 2012-13 school year, most SFAs (78.4%, 95% CI: 72.0%, 85.0%) reported having participated in training/TA to implement the HHFKA nutrition standards, whereas only one third reported having purchased equipment (33.8%, 95% CI: 24.4%, 43.2%).

SFAs used for secondary analyses (n=272 SFAs that reported having participated in training/TA) differed significantly from the SFAs that were excluded (n=93 SFAs that reported not having participated in training/TA) by SFA region (p<0.05), size (p<0.05) and poverty level (p<0.05). Specifically, the sub-sample included more SFAs in the Midwest, with fewer than 1,000 students, and poverty levels of 20% or more. Among these SFAs that reported participating in training/TA, SFAs had average training/TA scores of 10.6 (95% CI: 9.8, 11.4) out of 16 training/TA topics. This subset of SFAs rated the adequacy of training/TA as more than adequate (24.6%, 95% CI: 16.1%, 33.1%), adequate (43.9%, 95% CI: 34.7%, 53.0%), neither adequate/inadequate (28.4%, 95% CI: 22.4%, 34.4%), somewhat inadequate (2.5%, 95% CI: 0.7%, 4.4%), and not at all adequate (0.0). SFAs had average challenges scores of 3.5 (95% CI: 3.2, 3.9) out of 9 possible challenges to implementation.

Primary analyses found no statistically significant cross-sectional association between the odds of having a higher HEI-2010 scores for lunches served in 2014-15 and SFA past participation in training/TA or prior purchase of equipment (Table 2). Secondary analyses among the SFAs that reported participating in training/TA also did not find cross-sectional associations between HEI-2010 scores and the number of topics/areas covered by training/TA, the adequacy of training/TA, or the degree of implementation challenges (Table 2).

Table 2.

Cross-sectional association of implementation supports accessed by a national sample of school food authorities between the 2012-13 and 2014-15 school years and the nutritional quality of school lunches served during the 2014-15 school year

Implementation supportsa	Unadjusted odds ratiob (95% CI)	Adjusted odds ratiob,c (95% CI)
Primary analysis (n=365 SFAs d )
Past participation in training/TAe
No	Reference category	Reference category
Yes	1.05 (0.45, 2.4)	0.79 (0.35, 1.78)
Prior purchase of equipment
No	Reference category	Reference category
Yes	0.86 (0.38, 1.94)	1.00 (0.47, 2.15)
Secondary analysis (n=272 SFAs f )
Training/TA score	1.04 (0.89, 1.22)	1.04 (0.87, 1.24)
Challenges score	0.99 (0.91, 1.09)	0.96 (0.87, 1.06)
Adequacy of training/TA
Somewhat inadequate	Reference category	Reference category
Neither adequate/inadequate	1.08 (0.18, 6.39)	2.32 (0.45, 12.06)
Adequate	0.51 (0.12, 2.09)	1.22 (0.26, 5.68)
More than adequate	1.93 (0.42, 8.85)	2.96 (0.60, 14.73)

^aReported uptake between 2012-13 and 2014-15 school years.

^bOdds of having a higher Healthy Eating Index 2010 score for school lunches served during the 2014-15 school year.

^cAdjusted for SFA region, size, share of students from racial/ethnic minorities, poverty level, and urbanicity.

^dSFAs = school food authorities.

^eTA = technical assistance.

^fSubset of SFAs that reported participating in training/TA.

DISCUSSION

This study aimed to highlight implementation strategies SFAs reported accessing as they translated the HHFKA nutrition standards—which have resulted in improved school meal nutrition^5,6—into practice. Despite finding most SFAs reported participating in food services operations training/TA, accessing this implementation strategy alone between the years of 2012-13 and 2014-15 was not associated with the nutritional quality of lunches served in the 2014-15 school year. Equipment purchases during this time frame were not associated with more nutritious school lunches despite prior documentation of substantial equipment needs.⁹ These findings contribute to the literature gap on implementation supports but should be interpreted with caution given the cross-sectional study design that is subject to reverse causation.

While study findings suggest that reports of having received training/TA may not have contributed to school meal nutrition, it is possible the cross-sectional study design prevented the detection of this association. Participation in training/TA prior to 2012-13 and HEI-2010 scores before 2014-15 among sampled SFAs is unknown, and thus, changes in study variables over time were not analyzed. There are also several possible explanations, though speculative, for the absence of a significant relationship between equipment purchases and school lunch nutrition. Nearly 90% of SFAs sampled by the Kids’ Safe & Healthful Foods Project reported needing equipment at the start of the 2012-13 school year.⁹ While the present study suggests some SFAs did make equipment purchases, the cross-sectional design precludes an understanding of initial differences in equipment needs and if equipment purchases were followed by improvements in school lunch nutrition. Another possibility, especially considering the limited evidence base¹⁴ for how the implementation support of equipment impacts nutritional quality, is that SFAs were still able to serve nutritious school lunches even without adequate equipment. About half of SFAs from the Kids’ Safe & Healthful Foods Project reported not having necessary equipment during the 2012-13 school year but that they were able to “make do” with existing equipment.⁹ SFAs in the present study sample may also have been able to work around equipment limitations though it would be important to understand how these activities impact costs and efficiency of policy implementation, especially since SFAs are required to operate on cost-neutral budgets.²⁹ Similarly, a final possible consideration is that the HHFKA itself is a strong policy that was effective at helping SFAs achieve an HEI score well above the average diets of U.S. children³⁰— it is possible strengthening policy implementation may have benefits for other implementation outcomes but may not be reflected in HEI scores for which SFAs have been able to achieve high scores on average.

To our knowledge, the present study is the first to describe the use of implementation supports specifically for implementing the HHFKA nutrition standards and to consider the association between these supports and school meal nutrition, both understudied topics within the implementation science literature in school settings. Implementation studies of a policy requiring the replacement of less nutritious “red” menu items with more nutritious “green” menu items in Australian schools identified face-to-face meetings and menu audits/feedback as being helpful implementation supports;^31,32 however, it is unknown if the utility and benefits would translate to the U.S. context since the HHFKA nutrition standards have complex daily and weekly meal requirements.² The present study is also the first to apply Proctor’s model for implementation outcomes⁷ to the HHFKA nutrition standards. Future studies with measures of change using Proctor’s model are needed to better understand relationships between outcomes. Additionally, it would be informative to link SFA implementation supports to Proctor’s client outcomes (i.e., student dietary intake) which we were unable to do in the present study since student-level outcomes were only available for a limited number of SFAs in our analytical sample. It may also be important to consider if determinant implementation science frameworks and constructs could better explain determinants of implementing nutrition standards and meal quality. For example, multiple studies have used the Consolidated Framework for Implementation Research (CFIR) to identify determinants of school-based nutrition program implementation including perceptions of program benefits³³ and organizational culture in the inner setting,³⁴ and cosmopolitanism and student needs and resources in the outer setting.³⁴ Continued investigation of the relationships between determinants and contextual factors, implementation supports, and school nutrition is needed.

This study has several limitations. Though reports of similar questions about SFA training and equipment needs prior to the HHKFA have been published,^8,9 the authors did not have access to identifying information for SFAs that would have allowed for matching across datasets. Lack of baseline measures precludes any assessment of change in implementation supports and associated changes in meal quality from pre- to post- policy implementation. Unmeasured confounders, such as SFA budgets or the share of lunches served as free- or reduced- price meals, could influence the observed findings. Additionally, analyzing subsets of SFAs from the SNMCS, rather than the full nationally representative sample, may have diminished study power and reliability of study estimates. The use of a binary outcome (low/high HEI score) may also have limited the ability to detect an association given the loss of information near the outpoint.³⁵ It is also important to note that the schools within the SFAs, for which HEI lunch scores were averaged to the SFA level, may not be representative of the SFA; however, these data were still the best available national source to explore the aims of this implementation study. The authors also only had access to SNMCS data for implementation supports accessed by SFAs—there may have been training/TA delivered by SFAs to individual schools, for example, that influenced school nutrition. Another limitation is the potential for self-report bias. Cross-validation with observational methods, documentation of implementation supports delivered on multiple levels, and the use of implementation science frameworks to design study measures could be ways to strengthen data in future studies.

CONCLUSIONS

Many SFAs accessed implementation supports as they adopted the HHFKA nutrition standards. Continued research is needed to determine what factors are most effective for helping schools adopt new federal nutrition policies to ensure equitable implementation processes.

RESEARCH SNAPSHOT:

Research question:

What is the cross-sectional relationship between implementation supports (i.e., prior participation in food services training/technical assistance and purchase of equipment) for adopting nutrition standards of the Healthy, Hunger-Free Kids Act and the nutritional quality of school lunches among a national sample of school food authorities?

Key findings:

Between the 2012-13 (year of policy adoption) and 2014-15 (year of data collection) school years, most school food authorities completed training/technical assistance and one third purchased equipment. Neither implementation support was associated with the nutritional quality of school lunches served during the 2014-15 school year in this cross-sectional sample.