Validity and Interrater Reliability of the Visual Quarter-Waste Method for Assessing Food Waste in Middle School and High School Cafeteria Settings

Abstract

Background

Measuring food waste (ie, plate waste) in school cafeterias is an important tool to evaluate the effectiveness of school nutrition policies and interventions aimed at increasing consumption of healthier meals. Visual assessment methods are frequently applied in plate waste studies because they are more convenient than weighing. The visual quarter-waste method has become a common tool in studies of school meal waste and consumption, but previous studies of its validity and reliability have used correlation coefficients, which measure association but not necessarily agreement.

Objective

The aims of this study were to determine, using a statistic measuring interrater agreement, whether the visual quarter-waste method is valid and reliable for assessing food waste in a school cafeteria setting when compared with the gold standard of weighed plate waste.

Methods

To evaluate validity, researchers used the visual quarter-waste method and weighed food waste from 748 trays at four middle schools ànd five high schools in one school district in Washington State during May 2014. To assess interrater reliability, researcher pairs independently assessed 59 of the same trays using the visual quarter-waste method. Both validity and reliability were assessed using a weighted κ coefficient.

Results

For validity, as compared with the measured weight, 45% of foods assessed using the visual quarter-waste method were in almost perfect agreement, 42% of foods were in substantial agreement, 10% were in moderate agreement, and 3% were in slight agreement. For interrater reliability between pairs of visual assessors, 46% of foods were in perfect agreement, 31% were in almost perfect agreement, 15% were in substantial agreement, and 8% were in moderate agreement.

Conclusions

These results suggest that the visual quarter-waste method is a valid and reliable tool for measuring plate waste in school cafeteria settings.

POLICIES TO IMPROVE SCHOOL NUTRITION, SUCH AS the Healthy Hunger-Free Kids Act of 2010, may result in more students selecting healthy options at breakfast and lunch, including fruits, vegetables, and low-fat unflavored milk,¹ but it is not clear whether students consume these foods.^2,3 Measuring food waste in school cafeterias is an important part of a comprehensive evaluation of the effectiveness of school policies and interventions designed to increase consumption of healthier meals⁴ and to reduce food waste in schools.⁵

Weighing food waste (ie, plate waste) from individual plates or trays with a scale, considered the gold standard method,⁶ can be time-intensive, costly, and impractical in a busy cafeteria setting.⁷ Therefore, indirect forms of measuring waste have been developed,⁸ including aggregate selective plate waste,⁹ student self-report of consumption,¹⁰ visual estimation on-site,^10,11 and, more recently, visual estimation using digital photography.^12,13 Of these indirect methods, visual estimation by trained observers is relatively unobtrusive, efficient, and it safeguards against self-report bias, but may be subject to potential measurement bias.

Visual estimation is an assessment process in which trained observers estimate food waste using one of several developed scales.^{10–11,13–15} The visual quarter-waste method, which estimates food waste as 0% remaining, 25% remaining, 50% remaining, 75% remaining, or 100% remaining, has been used with increasing frequency.¹⁵ Studies that have assessed the validity of visual estimation have used intermethod comparisons and discussed situations where one method may be preferable.^{10,13,15–17} To do so, they have assessed the correlation between visual estimation and the weight of waste. By evaluating linear trends, correlation coefficients measure association but not necessarily agreement¹⁷ and should be interpreted with caution. Thus, the aims of this study were to assess the validity of the visual quarter-waste method in comparison to weighing plate waste, and the interrater reliability of the visual quarter-waste method, using a statistic that measures interrater agreement rather than correlation.

METHODS

Study Design

This study was conducted by trained research staff at the University of Washington Center for Public Health Nutrition as part of a larger study evaluating the influence of a school cafeteria intervention.¹⁸ In the intervention, cafeteria staff implemented low-cost strategies to improve students’ selection and intake of fruits, vegetables, and white milk within the National School Lunch Program. Study protocols were approved by the University of Washington Institutional Review Board. Data collection took place during May 2014 at four middle schools and five high schools in one school district in Washington State. Each school was visited once. In preparation for data collection, 10 graduate and undergraduate students received 2 hours of training in the visual quarter-waste method from the study’s project coordinator. Training included instruction regarding the tool and protocol as well as time to practice using the protocol with sample lunches until the team could accurately assess the food waste compared with test amounts.

Data Collection

At each school, the research team reviewed the day’s food options before the cafeteria opened to familiarize themselves with the size and shape of whole entrées. Some foods, such as whole fruits, vegetable side dishes, and low-fat white and chocolate milk, were the same at all nine schools, whereas entrées and specialty fruit items varied between schools. Beverages and most fruits and vegetables were prepackaged with the exception of whole apples and oranges. Cafeteria staff served hot entrées, and one school had a staff-operated made-to-order entrée salad bar on the day of data collection. Items were in standardized sizes due to being prepackaged or served by cafeteria staff. No foods included in the study were self-serve. In some cases, the team retained whole entrées or side dishes as references and agreed upon measurement standards such as the number of baby carrots in a prepackaged bag. Foods sold outside of the National School Lunch Program, called competitive foods, were not included in the study.

Researchers selected cafeteria tables during each lunch period based on a random cluster sample design. Once students were seated with their lunches, students at those tables were recruited to participate in the evaluation study.¹⁸ Each student provided verbal consent to participate. Research staff completed a card for each tray by marking the foods present, and taped it to the bottom of the student’s tray. Students were instructed to return their trays with all waste and wrappers to a marked rack in one section of the cafeteria when they finished eating. These trays were randomly distributed to members of the research team for visual assessment and then passed to another researcher who weighed the food waste.

Visual Quarter-Waste Method.

Each food on the tray was assigned a visual assessment estimate using the quarter-waste method, marking the amount of food waste as either 0% (none left), 25%, 50%, 75%, or 100% remaining (all left). Visual assessors noted on the data card any foods for which there was no evidence of expected waste such as an empty container or peel, as well as those that were added after the data card was filled. Entrées served with a side dish in the same container, such as chicken fingers and roasted potatoes, were visually assessed as separate foods. Visual assessors poured milk and juice into a measuring cup and measured the remaining fluid to the closest quarter-cup. A full milk carton is 8 oz and a full juice carton is 4 oz. Using simple calculations, the visual assessor determined the appropriate category for percent remaining. This approach for beverages differed from the standard visual quarter-waste method used because milk was served in cardboard cartons that were open in one corner and difficult to visually assess. For consistency, the pouring method was used for all beverages.

Interrater reliability testing was conducted at three schools with smaller student populations and/or fewer lunch periods, which allowed the team time to assess the same trays more than once. At these schools, the team tested a total of 59 trays for interrater reliability. After one visual assessor assessed a tray, she stapled the completed data card underneath a blank data card and marked on that card which foods were present on the tray before passing the tray at random to a second visual assessor for assessment. All members of the team were included in interrater reliability testing.

Weighed Method.

After visually assessing a tray, the researcher passed it to another researcher who weighed the food waste for each individual food on a scale (E Series E-160 Digital Portion Scale; Edlund) and noted the gram weight in the appropriate row on the data card. All scales were calibrated before data collection according to manufacturer directions. Beverages were weighed in their containers. For an entrée served in a cardboard boat that included more than one type of food item, such as chicken nuggets and french fries, each food was weighed separately and staff noted the presence or absence of the container for each weight. Once a tray’s food items had been weighed and recorded, they were discarded.

Throughout the data collection period, researchers weighed each item offered in its whole and/or uneaten state at least once. Foods accessible to the researchers at multiple schools before lunch service—all beverages and most side dishes—were able to be weighed multiple times. Multiple weights obtained for the same food or beverage were averaged together to provide a baseline, uneaten weight for each food. Foods weighed in containers or packaging were noted as such. Researchers also collected and retained at least one type of container or packaging for all food items, which were washed, dried, and cleaned before being weighed and recorded.

For foods served in a boat, container, carton, or plastic bag, the packaging weight was subtracted from the recorded weight of the food waste to provide a final weight of the food waste alone. This weight was divided by the average uneaten weight for the food and then multiplied by 100 to calculate a percent remaining for the food. These percentages were sorted into five categories to correspond with the visual assessment scale: 0% to 12.5% remaining (0% on visual assessment scale), 12.51% to 37.5% remaining (25% on visual assessment scale), 37.51% to 62.5% remaining (50% on visual assessment scale), 62.51% to 87.5% remaining (75% on visual assessment scale), and 87.51% to 100% remaining (100% on visual assessment scale). These bands reflect the distribution of the possible categories that could be assigned by visual assessors.

Data Analysis

Both validity and interrater reliability for each food were analyzed in Stata¹⁹ using a weighted κ statistic.²⁰ The κ is intended to compare the results of two raters. It measures the amount by which the agreement between two raters exceeds that expected by chance. A weighted κ assigns less weight to agreement as categories become further apart and is chance-corrected to account for the possibility that both raters could be assigning scores or estimates at random.²¹ In this case, one of the ratings is an actual weight and the percent remaining gram weight was calculated using a full, uneaten weight based on a random sample of trays.

The κ statistic has a maximum of 1 when agreement is perfect, 0 when agreement is no better than chance, and negative values when agreement is worse than chance. Although no single tool to evaluate κ is perfect, the most commonly cited interpretation of κ is by Landis and Koch,²¹ where κ=0 indicates agreement is no better than chance, κ of 0.01 to 0.20 indicates slight agreement, κ of 0.21 to 0.40 indicates fair agreement, κ of 0.41 to 0.60 indicates moderate agreement, κ of 0.61 to 0.80 indicates substantial agreement, and κ of 0.81 to 0.99 indicates near perfect agreement, and κ of 1.0 indicates perfect agreement. Standard error was calculated for all κ statistics. Standard deviation and coefficients of variation of baseline uneaten weight measurements were calculated for foods in the validity analysis for which at least four weights of whole, uneaten portions were recorded.

RESULTS

Food waste was assessed using the visual quarter-waste method and the weighed plate waste method for 748 trays. A total of 2,227 individual weights were collected for 39 foods, including beverages, fruits, vegetables, entrées, and side dishes. Any items that had weights but no corresponding visual assessment score were excluded from analysis (n=71). An additional three data points were excluded as outliers. The number of weights for a specific food ranged from a low of one (bean salad) to 388 (apple juice). The average number of weights per food was 57 and the median was 21.

Validity

The validity analysis excludes eight foods for which there were fewer than five pairs of ratings, and produced a total of 31 κ statistics. Fourteen of 31 foods had a κ in almost perfect agreement, 13 in substantial agreement, three in moderate agreement, and one in slight agreement with the gram weight of the waste (Table 1). The coefficients of variation suggest that foods served in prepackaged containers had the least variation in serving size, but that foods served by cafeteria staff, such as chicken tenders and french fries, had relatively low variation in serving size as well.

Table 1.

Validity: Variation in initial serving size and level of agreement between plate waste assessed by weight and the visual quarter-waste method for foods from 748 trays in 9 middle school and high school cafeterias in one Washington State school district during May 2014, by food type

	Agreement between Weight and Visual Quarter-Waste Method			Variation in Initial Serving Size
Food or beverage item Food: Entrees	n	κa	Standard errorbc	Standard deviation (g)d	Coefficient of variation (%)d
Hot dog	18	0.96	0.193
Cheese pizza	107	0.84	0.069
Fish taco	19	0.82	0.148
Canadian bacon and pineapple pizza	37	0.80	0.126
Seafood basket	56	0.80	0.095
Sub sandwich	72	0.75	0.077
Pepperoni pizza	57	0.74	0.100
Chicken tenders	67	0.64	0.084	13.1	16
Burger	26	0.59	0.153
Chicken Caesar entree salad	12	0.56	0.187
Food: Side items
Dinner roll	50	0.93	0.1101	1.36	3
Soup	58	0.92	0.095
Side fries	221	0.66	0.049	14.3	12
Fries with entree	108	0.63	0.067	19.3	16
Rice with entree	12	0.58	0.167
Food: Fruits and vegetables
Fruit cup	32	0.83	0.132	8.1	6
Potato salad	29	0.83	0.150	13.1	11
Fruit of the day	16	0.82	0.188
Baby carrots	53	0.82	0.096	2.6	4
Raisins	9	0.81	0.253	8.1	6
Grapes	10	0.80	0.283
Orange	17	0.80	0.217	17.4	10
Applesauce	30	0.79	0.132
Apple	61	0.79	0.088	8.6	7
Lettuce/tomato/pickle cup	22	0.77	0.138	8.0	11
Corn	12	0.73	0.191	6.6	8
Side salad	14	0.16	0.123	13.1	11
Beverages
Orange juice	109	0.85	0.073	0.9	1
Chocolate milk	333	0.83	0.041	1.4	1
White milk	114	0.83	0.070	1.3	0
Apple juice	388	0.82	0.038	1.5	1

^aMeasures the amount by which the agreement between two ratings exceeds that expected by chance. κ=1 when agreement is perfect and κ=0 when agreement is no better than chance.

^bBecause trays were randomly distributed to members of the research team, these standard errors do not account for differences between raters.

^cFor small numbers of pairs (<30) there is reduced confidence in the reliability of these standard errors.

^dTo examine the possibility of variation in initial serving sizes impacting validity analyses, standard deviations and coefficients of variation of “uneaten weight” measurement in grams was calculated for foods with at least four recorded weights for a whole, uneaten portion. These statistics are only related to one of the ratings, the gold standard weight, used to calculate the κ statistics. They are not related to the quarter-waste method rating.

Interrater Reliability

For interrater reliability, two raters assessed 59 trays at a total of three schools. Twenty-nine foods were assessed by two raters at least once, and 13 of these foods had at least five pairs of ratings. For six of the 13 foods, the assessments by two raters using the visual quarter-waste method were in perfect agreement, four were in almost perfect agreement, two were in substantial agreement, and one was in moderate agreement (Table 2).

Table 2.

Interrater reliability: Level of agreement between plate waste assessed by two raters in using the visual quarter-waste method for foods from 59 trays in 3 middle school and high school cafeterias in one Washington State school district during May 2014

Food or beverage item	n	κa	Standard errorbc
Food
Cheese pizza	11	1.0	0.302
Fries with entree	9	1.0	0.270
Hot dog	6	1.0	0.320
Baby carrots	5	1.0	0.338
Seafood basket	5	1.0	0.374
Sub sandwich	9	0.91	0.249
Canadian bacon and pineapple pizza	5	0.89	0.237
Side fries	13	0.69	0.233
Pepperoni pizza	7	0.59	0.344
Beverage
White milk	12	1.0	0.237
Chocolate milk	31	0.87	0.137
Orange juice	6	0.81	0.281
Apple juice	33	0.78	0.128

^aMeasures the amount by which the agreement between two ratings exceeds that expected by chance. κ=1 when agreement is perfect and κ=0 when agreement is no better than chance.

^bFor small numbers of pairs (<30) there is reduced confidence in the reliability of these standard errors.

^cBecause trays were randomly distributed to members of the research team, these standard errors do not account for differences between raters.

DISCUSSION

This study found that the visual quarter-waste method is a valid and reliable tool for measuring plate waste in a school cafeteria setting. For validity, compared with the measured weight, 45% of foods assessed using the visual quarter-waste method were in almost perfect agreement, 42% of foods were in substantial agreement, 10% were in moderate agreement, and 3% were in slight agreement. For interrater reliability between the two raters, 46% of foods were in perfect agreement, 31% were in almost perfect agreement, 15% were in substantial agreement, and 8% were in moderate agreement. A recent analysis of the interpretation of the κ statistic in health care studies proposes that κ≥0.60 demonstrates adequate agreement between raters.²² The vast majority—87%—of validity κ values in this study are above that threshold.

In the validity data, some of the highest κ values correspond to foods served in a single, solid, uniform shape such as hot dogs, pizza and dinner rolls, or prepackaged foods with many pieces (eg, baby carrots or raisins). One might expect certain foods to be difficult to assess visually due to characteristics related to the food’s appearance and/or flow when agitated,²³ such as soups that involve particles suspended in liquid and change shape easily. However, foods that were creamy, chunky, or included more liquid (eg, potato salad or fruit cup), also performed very well with the visual quarter-waste method. For example, soup (eg, chicken penne or clam chowder) had a κ=0.92, indicating nearly perfect agreement with the recorded weight. This suggests that visual assessment is able to accurately assess amorphous, soupy foods as well as those with firmer, more recognizable shapes.

For validity, the four foods with κ<0.61 (less than substantial agreement) were rice served with an entrée, chicken Caesar salad, burger, and side salad. Although most of these items were served rather than prepackaged, results showing relatively limited variation in initial serving sizes for some served foods (eg, chicken tenders and french fries) suggest that other served foods likely had limited variation in initial serving size as well, and that differences in initial serving amounts likely do not influence the validity assessment. The rice was served with another item—sweet and sour chicken with sauce—and may have been difficult to visually isolate and assess. The lower κ statistics for the other three foods may highlight a potential limitation—or at least a difference in perspective—of weighing as compared with the visual quarter-waste method: Weighing does not distinguish between different components of a food item, whereas visual assessors may. For salads that were entirely consumed except for leftover creamy dressing (added to a salad in varying and sometimes substantial amounts after purchase) and croutons, the actual gram weight of the food waste reflects a relatively small but potentially heavier part of the salad, as well as salad dressing weight potentially not accounted for in the baseline, uneaten weight denominator. Visual assessors may have ignored this and noted the lack of vegetable salad contents. A similar situation was noted with burgers. When determining a rating for the amount remaining of a hamburger or cheeseburger, the visual assessment may have taken into account the fact that some students left part of the cheese, condiments, and toppings but ate the burger meat and bun. Therefore, the low level of agreement for salads and burgers may be due to different ways of defining proportions of the whole. Each approach may have different benefits or limitations, although visual assessors can more clearly distinguish and account for added or remaining items, such as condiments and dressings.

In addition to its validity, this study demonstrated several strengths of the visual quarter-waste method. The research staff of undergraduate and graduate students was able to produce reliable results after only 2 hours of training, indicating that only brief amounts of training are needed. This presents the possibility that cafeteria staff and administrators could use the tool without the assistance of research staff or expensive equipment. In addition, the visual quarter-waste method is minimally disruptive to cafeteria flow compared with other food waste assessment methods and can be used to assess waste during meal service.

There are several limitations to this study. First, sample sizes for some foods in the validity analysis were lower than others due to those foods being selected less often by the students, and the accompanying κ statistics should be interpreted with caution. In addition, data for interrater reliability was only collected at three of the schools. This limited the sample size for individual foods because only a certain number of lunch items were offered at each school on the day of data collection. Therefore, the results for interrater reliability should be interpreted with caution, but preliminary data are supportive of the visual quarter-waste method as a reliable tool with regard to interrater reliability.

Second, because the nature of milk cartons made visual assessment of beverages with the quarter-waste method difficult, beverages were visually assessed by pouring them into a measuring cup and determining the amount remaining to the closest quarter-cup. The true equivalent to visual quarter-waste assessment for beverages would be to look at the fluid and estimate the percent remaining. In this study, the act of pouring fluids into a measuring cup took negligibly more time and was quite accurate (κ>0.81 for all four beverages), suggesting it is a reasonable modification to visual assessment of fluids that can be used in school cafeteria settings.

Finally, this study was conducted in one school district with a particular way of packaging and serving food. Other districts may serve foods in ways that could be more difficult to assess using this method, especially when serving sizes are less standardized (eg, self-serve salad bars and other self-serve foods).

This is the first known study to establish the validity and interrater reliability of the visual quarter-waste method using the chance-corrected κ statistic, which addresses methodologic concerns in comparing waste measurement methods. The strength of the results is especially important considering that data collection occurred during lunch service in multiple middle school and high school cafeterias with accompanying space, time, and noise limitations. These conditions are representative of those in which the visual quarter-waste method may be used to evaluate school meal consumption and therefore it is significant that even when used in a real-life setting, the method is reasonably valid.

CONCLUSIONS

A variety of plate waste measurement methods exist. Although weighing plate waste from every student on a scale is the gold standard, it is not always feasible. This study suggests that the visual quarter-waste method is a valid and reliable assessment tool for plate waste in a school cafeteria compared with weighing and, in certain cases, may actually be more accurate. These findings validate results of past studies in which the visual quarter-waste method was used as a tool to measure plate waste and support its use as an assessment method. Future research should explore the types and amounts of foods that are wasted, evaluate the effectiveness of food waste reduction interventions, and examine the use of this method in other settings with populations whose nutritional status is of interest, such as early child care and education settings.