Inventory Versus Checklist Approach to Assess Middle School à la Carte Food Availability

Abstract

BACKGROUND

The purpose of this research is to evaluate 2 methods of assessing foods available on school à la carte lines for schools’ ability to assess the proportion of foods that are healthful options.

METHODS

This observational study used data collected at 38 middle schools, October 2006–May 2007. An inventory method was used to collect detailed information of items available on each school’s à la carte line, followed by a simplified checklist form. Using the detailed inventory method, the proportion of items meeting the Institute of Medicine’s (IOM) nutrition standards for foods available at each school was calculated. From the checklists, we calculated the proportion of categories representing more healthful foods. Schools were independently ranked according to the percentage of items meeting the IOM criteria, (inventory data) and the percentage of food categories considered “healthy” (checklist data). Wilcoxon rank sum test was used to compare school rankings.

RESULTS

The inventory and checklist approaches showed a good level of agreement when both methods were independently used to rank the level of healthy foods available on à la carte (Wilcoxon rank sum = 32.5, p = .62).

CONCLUSION

For purposes of ranking schools along a continuum of “healthfulness of foods on à la carte lines,” especially when resources are limited, a checklist approach appears to be satisfactory. This method may also be useful to school stakeholders needing an inexpensive à la carte assessment tool.

While the epidemic of childhood obesity is well documented, the etiology of the disease is poorly understood.^1,2 One venue that has been investigated for its contribution to the epidemic and called upon to help reduce the risk is the school environment.^3,4 Beyond food served as part of the United States Department of Agriculture’s (USDA) National School Lunch Program (NSLP) and School Breakfast Program (SBP), it is well recognized that youth have many other options for eating and drinking during the school day. An important source of this additional food that is largely unregulated is the competitive foods or foods that “compete” with the regular meal pattern lunch such as foods offered as fund-raisers or incentives, foods available in vending and in schools stores or foods available on the à la carte line.^5,6 À la carte foods are foods and beverages available in the cafeteria that are not part of the regular reimbursable meal pattern. Foods available through à la carte are of particular interest because of their availability in large numbers, the majority of which are high in fat and sugar.^4,5 The presence of an à la carte line in schools has been shown to impact the dietary intake of youth across the entire day.^7,8

Interest in understanding the role and extent of competitive foods in schools and attempting to encourage schools to offer more healthful options in à la carte and vending has been growing in the past 2 decades. In 1996, the Centers for Disease Control and Prevention published the Guidelines for School Health Programs to promote lifelong healthy eating, one of the first government documents to draw attention to other sources of food in schools and its potential to impact the health of school children.⁹ In 2004, USDA passed the Child Nutrition and Women, Infants, and Children (WIC) Reauthorization Act¹⁰ that required the development of a wellness policy that included, among other things, guidelines for competitive foods in schools. Most recently, the Institute of Medicine (IOM) published specific criteria to be applied to evaluate the healthfulness of foods offered outside of the NSLP.¹¹ With this interest, a variety of stakeholders, including school staff, community interest groups, public health nutrition advocacy groups, and researchers, have attempted to assess and evaluate what schools are offering through their à la carte programs and the healthfulness of those choices.

Current reports of à la carte foods tend to be descriptive in nature, for example, reporting the proportion of schools that offer salty snacks or soft drinks. While this information has been important in drawing attention to the types of foods that are offered in schools, more work needs to be done to increase our understanding of how the food environment of schools may be related to health.⁵ The intent of measuring the healthfulness of an environment is actually to assess an attribute of the environment that may affect health; for example an environment that offers more healthful options as compared to fewer healthful options. This characterization is most useful if it results in a measure that has construct validity (eg, being able to show that schools with more healthful foods in the environment have students who eat more healthful foods) or a measure that can rank order schools along a continuum that assesses the degree of obesogenic potential or conditions leading to obesity.¹² We know very little about how tools compare with regard to their ability to rank order schools along some continuum of “healthfulness” or “obesogeneity.”

A number of methods have been used to identify and quantify foods offered on school à la carte lines including the use of cash register data,¹³ inventories,⁷ and checklists.⁵ However, each method has its limitations with regard to expense, amount of data reduction required, validity, and reliability. There is a great need for tools that are efficient, reliable, and valid indicators of the “healthfulness” of the school food environment, both for school stakeholders and for researchers attempting to characterize school food environments along some continuum of healthfulness or obesogeneity. The purpose of this study is to compare 2 approaches for ranking schools with regard to healthfulness of foods available on the à la carte lines: a detailed inventory approach, and a checklist approach. The hypothesis is that a checklist approach is a valid method for assessing the healthfulness of the à la carte food environment of schools, particularly when resources for data collection may be limited.

METHODS

Subjects

Data for this analysis came from the Transdisciplinary Research in Energetics and Cancer-Identifying Determinants of Eating and Activity (TREC-IDEA) study,^14,15 which is a longitudinal cohort study funded by the National Cancer Institute with a goal of increasing our understanding of the etiology of adolescent obesity. Data were collected from 349 student and parent dyads from October 2006 to May 2007. Data collection included measures of body composition as well as potential etiologic factors at multiple levels of influence including psychosocial factors related to energy balance, dietary intake, and physical activity patterns and factors at the home, school, and neighborhood environmental levels. See Lytle¹⁵ for more information about study design. Data for this analysis came from the school-level data collected for TREC-IDEA (n = 116) but were restricted to middle schools only (n = 38) as only the middle schools had à la carte data collected in inventory format. The student body represented in these middle schools were predominantly white (77.8%), were evenly distributed according to sex (49.4% female), and, on average, 22.1% of their student population qualified for free and reduced-cost lunch (22.1%). Nearly all middle schools were public (92.1%) and suburban (89.5%). This study was approved by the University of Minnesota Institutional Review Board.

Instruments

Several measures were included to assess the school environment and some data collection methods differed between middle and high schools. In middle schools, an inventory approach was used to measure foods available on à la carte lines. In the inventory method, a trained-staff member visited all middle schools and recorded all à la carte prepackaged items, the type of item, total ounces of the package, calories and fat per serving as well as the number of servings per package. Product details were obtained by directly recording information from packages or through information collected on the Internet. Pilot data revealed that the interrater reliability of this method of data collection was excellent (ICC = .9974).

Because of the vast amount of products available on high school à la carte lines, the inventory approach for recording à la carte items in high school was cost-prohibitive. Therefore, a checklist format was used for high schools. The checklist was modified from the CDC’s School Health Policy and Practice Survey¹⁶ and contained categories of common foods found in school environments. Pilot data showed excellent interrater reliability for data collection (kappa = .99). This research compares data collected using the inventory method with data from this checklist. Tables 1 and 2 represents the 2 data collection instruments.

Table 1.

Inventory Data Collection Form for Middle School à la Carte Food Environment

Inventory Form Example
Full Product Name (eg, Nacho Cheese Doritos)	Internal Code†	Price	Package Size (oz or g)	# Servings per Package*	Calories per Serving*	Fat Grams per Serving*

^*This information may need to be entered into the form after returning to the office.

^†These codes will need to be entered prior to data entry.

Table 2.

Checklist Data Collection Forms for Middle School à la Carte Food Environment

Checklist Form
Available?	Food and Beverage Items
(1)	100%fruit juice or 100%vegetable juice?
(1)	Diet soft drinks?
(0)	Sweetened beverages such as regular soda pop, sports drinks, or fruit drinks that are not 100%juice?
(1)	Fruit (fresh, frozen, canned, or dried)?
*	Breadsticks, rolls, bagels, pita bread, or other bread products?
(1)	Low-fat cookies, crackers, cakes, pastries, or other low-fat baked goods?
(0)	Cookies, crackers, cakes, pastries, or other baked goods that are not lowin fat?
(1)	Yogurt and smoothie
*	Pizza, hamburgers, or sandwiches
(1)	Lettuce, vegetable, or bean salads?
(1)	Other vegetables?
(0)	French fried potatoes?
(0)	Chocolate candy?
(0)	Other kinds of candy?
(1)	Salty snacks that are lowin fat, such as pretzels, baked chips, or other low-fat chips?
(0)	Salty snacks that are not lowin fat, such as regular potato chips or cheese puffs?
(1)	Low-fat or fat-free ice cream, frozen yogurt, or sherbet?
(0)	Ice creamor frozen yogurt that is not lowin fat?
*	Milk?
(1)	Water?

^*Items in these categories excluded from checklist due to ambiguity or lack of Institute of Medicine comparison.

( ) indicates points assigned for each category.

Procedure

The use of these 2 instruments provided an opportunity to examine the level of concordance in ranking middle schools using the 2 different methods. Using inventory data available from the middle schools participating in the research, study staff completed a corresponding checklist for each middle school. Interrater reliability for the results of the transcription of the data yielded 100% agreement.

To determine the healthfulness of à la carte offerings in the middle schools using the inventory method, each food and beverage item on the inventory was classified as not meeting (score = 0) or meeting (score = 1) IOM¹² criteria. IOM criteria include (1) food servings less than 35% of calories from fat, (2) food servings equal to or less than 200 calories per serving for food, (3) a serving size of less than 4 ounces of 100% fruit juice for middle school students, and (4) water without additives or carbonation. A total score representing the proportion of foods and beverages offered that met the IOM criteria was created for each school.

For each category on the à la carte checklist, less-healthy items such as sweetened beverages, regular-fat cookies, crackers or cakes, and French fries were given a score = 0. More healthy items were given a score = 1 and included 100% fruit or vegetable juice, water, vegetables, fruit, low-fat salty snacks and low-fat frozen desserts. As there was no distinction on the checklist for regular or reduced-fat milk, milk was removed from this analysis (n = 410 items). Additionally, as bread products or entrées such as pizzas and sandwiches could not be characterized because of their unknown fat content, these products were also removed from the analysis (n = 256 items), leaving 17 scorable categories. A ratio of healthy food categories available per total food categories offered was calculated for each school.

Data Analysis

Using the independent scoring procedures for the inventory and checklist methods, schools were sorted by score, then ranked by the percentage of healthy versus less-healthy à la carte items, with a rank of “1” being most healthy. The Wilcoxon rank sum test was used to determine if there was a difference in the “healthfulness ranking” of each school based on the inventory versus checklist mode of data collection using SAS (version v.9.1, 2003, SAS Institute, Cary, N.C.). Significance was determined at p = .05 or less. In addition, the analysis was rerun to test if limiting the sample to schools with at least 20 à la carte items available (an indication of variety) affected the level of concordance between the 2 methods.

RESULTS

The total number of middle schools in the full analysis was reduced to 37, as 1 school only offered milk as an à la carte item and milk was excluded from the analysis. Using the inventory method, the mean total number of food item categories offered as à la carte items per school ranged from 1 to 103 items (mean = 37.4 items) with a total of 1384 items offered across schools (Table 3). On average, fewer than 50% of foods offered across schools met the IOM criteria. The lowest ranking school (“37”) had only 6.3% of à la carte items offered meeting the IOM criteria.

Table 3.

Ranking of School According to the Proportion of Food Items Meeting the Institute of Medicine (IOM) Criteria Using the Inventory Approach Compared to the Ranking of Healthy Items Using the Checklist Format

School Code	Inventory Approach			Checklist Approach
	Number of à la Carte Items	Meeting IOM Criteria* (%)	Healthiness Ranking: Inventory	Percent of Checklist Items Healthy (%)	Healthiness Ranking: Checklist
A	1	100.0	1	100.0	1
B	22	95.5	2	66.7	6
C	32	90.6	3	62.5	13
D	18	77.8	4	71.4	4
E	6	66.7	5	66.7	7
F	62	64.5	6	50.0	25
G	16	62.5	7	50.0	26
H	89	55.1	8	63.6	12
I	81	54.3	9	58.3	17
J	35	54.3	10	55.6	20
K	39	53.8	11	58.3	18
L	54	53.7	12	58.3	19
M	34	52.9	13	66.7	8
N	50	52.0	14	54.5	23
O	103	51.5	15	50.0	27
Q	2	50.0	16	100.0	2
R	32	50.0	17	66.7	9
S	34	50.0	18	62.5	14
P	2	50.0	19	50.0	28
T	41	46.3	20	60.0	15
U	43	44.2	21	70.0	5
V	16	43.8	22	50.0	29
W	62	43.5	23	60.0	16
X	45	42.2	24	40.0	36
Y	80	41.3	25	54.5	24
Z	27	40.7	26	55.6	21
AA	41	39.0	27	55.6	22
BB	26	38.5	28	42.9	35
CC	53	35.8	29	44.4	34
DD	34	32.4	30	50.0	30
EE	24	29.2	31	50.0	31
FF	45	28.9	32	45.5	33
GG	54	27.8	33	50.0	32
II	4	25.0	34	100.0	3
HH	48	25.0	35	12.5	37
JJ	13	7.7	36	66.7	10
KK	16	6.3	37	66.7	11
Total	1384	48.6	—	59.1

^*IOM criteria for healthy food is (1) food servings less than 35% of calories from fat, (2) food servings equal to or less than 200 calories per serving for food, (3) a serving size of less than 4 ounces of 100% fruit juice for middle-school students, and (4) water without additives or carbonation.

Using the checklist method, the proportion of healthy items versus total items ranged from 12.5% to 100% with a mean of 59.1% of food and beverage categories offered being healthy (Table 3). The Wilcoxon rank sum test showed a nonsignificant difference between rankings using the 2 modes of data collection (p = .62). Concordance improved (p = .95) when the analysis was restricted to schools with 20 or more à la carte items (n = 28 schools). However, for both full and reduced samples, only 1 out of the top 3 highest ranked schools using the inventory method also ranked in the top 3 using the checklist method. For the 3 lowest ranked schools by the inventory method, in the both full and reduced samples, there was concordance with 1 of the 3 schools. But, notable in the reduced sample, the least healthy (bottom 25%) schools were the same using either method, with only 1 exception.

DISCUSSION

This study compared 2 methods for assessing an attribute representing the healthfulness of foods offered on middle school à la carte lines. The findings suggested that checklists are valid in representing competitive foods offered on à la carte lines in a school and that they are sensitive enough to rank schools along a continuum of healthfulness. There was no statistically significant difference in healthfulness rankings between the more labor-intensive inventory method and the checklist method. Therefore, particularly for school personnel or community groups, the use of the checklist approach can provide a quick, inexpensive and useful tool to assess the general “healthfulness” of the à la carte food environment.

However, there are trade-offs. The use of an inventory provides potentially important detail such as the number and types of specific products offered cost and portion size. Using a more detailed inventory approach, nutrient level estimates can be examined such as the average caloric or fat content offered. Unfortunately, inventories are very labor-intensive and expensive, requiring data collection on site by a trained observer that may require 1 hour or more of data collection time. In addition, once the data are collected, there is a great deal of time and expense required to find missing data such as grams of fat per package or number of portions for each packaged product. The inventory approach produces a large volume of data requiring many data reduction decisions including (1) the level of detail to retain (ie, do we care about all of the types of sugar sweetened beverages available or do we just want to know if any sugar sweetened beverages are available?), (2) how to express the findings (ie, do we present nutrient data based on package size or serving size or 100 g weight?), (3) what to do with price data when the kinds and sizes of products vary widely?, and (4) the criteria of “healthfulness” to use (should we evaluate foods available using the IOM criteria, the Healthy Generations criteria,¹⁷ or some other way to classify foods based on healthfulness?).

Determining the criteria for healthfulness will likely influence rankings and the overall picture of the healthfulness of schools. As an example, our choice of using the IOM criteria likely resulted in greater misclassification of schools than if we had used another less-stringent criterion for healthfulness. One school in our study is ranked as #3 using the checklist approach but ranked #34 using the inventory approach. This misclassification occurred because 100% fruit juice is considered as a healthy beverage in the checklist but the portion size as set by the IOM (greater than 4 ounces for middle schools and 8 ounces for high schools) was exceeded and documented using the inventory approach. School stakeholders may want to consider petitioning suppliers for smaller portion sizes of 100% fruit juice in order to be more in line with IOM criteria. We also noticed that the checklist was more effective in classifying schools with a low percentage of healthy items than a high percentage of healthy items, but that trend is most clearly identified when looking at the reduced sample of schools.

Limitations

It is important to acknowledge the study limitations. The checklist was not completed in the school but completed post hoc using previously collected inventory data. While it would have been preferable to use 2 independent and simultaneous data collections, cost prohibited this additional expense. However, we believe that our findings would have been similar as the checklist approach was very simple and since test-retest reliability of all aspects of the data collection and data transcription was very high. In addition, the data presented reflect a snapshot of the items offered on one particular day and may not accurately represent foods usually offered on à la carte. Finally, the findings may not be generalizable to other schools or settings; the schools in our sample were largely suburban and public.

Conclusions

This study demonstrated that a simple checklist of foods available on à la carte does not differ substantially from a labor-intensive inventory approach on its ability to rank schools according to a measure of healthfulness. Future work should refine the checklist while attempting to maintain simplicity. For example, adding checklist items for portion size of juice or specifying types of milk available may allow for a more accurate assessment of foods meeting the IOM criteria. One may be able to better capture the distinctions between more and less-healthful entrées by creating separate categories for “sandwiches and other cold entrées” as healthier and “pizza, hamburgers, chicken nugget” and other hot entrées as less healthy. In addition, it would be helpful to evaluate the utility and criterion validity of a checklist to characterize foods available in vending and in school stores. This tool could be very useful to public health practitioners or researchers that are looking for a quick and inexpensive way to rank schools with regard to the healthfulness of products available.

IMPLICATIONS FOR SCHOOL HEALTH

The checklist approach for assessing the school à la carte food environment is a useful tool for school wellness councils or other school personnel needing to assess the healthfulness of food items in a quick and easy manner. Following some modifications of the checklist categories, the scoring of the items can be translated from “1” being equal to “healthy” and “0” to “less healthy” with subsequent calculation of a ratio of healthy to less-healthy items. The school wellness council’s goal will be to increase the ratio of healthy to unhealthy food items. Regardless of these potential alterations and keeping an eye toward simplicity, this tool has proven to be a valid and easy tool for assessing the healthiness of the school à la carte environment.