Creation and validation of a NOVA scored picture set to evaluate ultra-processed foods

Abstract

There has been a rapid shift in the modern food environment towards increased processing in foods consumed in the United States (US) and globally. The NOVA system (not an acronym) for classifying food on degree of processing currently has the most empirical support. Consumption of foods in the NOVA 4 category, ultra-processed foods (UPF), is a risk factor for a host of poor health outcomes including heart disease, stroke, and cancer. Despite these poor health outcomes, UPF make up 58% of calories consumed in the US. Methodologies for assessing the reinforcing and rewarding properties of these foods are necessary tools. The Becker-DeGroot-Marschak auction paradigm (BDM) is a well validated tool for measuring value and is amenable to neuromonitoring environments. To allow for the testing of hypotheses based on level of food processing, we present a picture set of 14 UPF and 14 minimally-processed foods (MPF) matched on visual properties, food characteristics (fat, carbohydrate, cost, etc.), and rated perceptual properties. Further, we report our scoring of these foods using the NOVA classification system and provide additional data from credentialed nutrition professionals and on inter-rater reliability using NOVA, a critique of the system. Finally, we provide all pictures, data, and code used to create this picture set as a tool for researchers.

Introduction

In recent decades, the modern food environment has undergone a rapid shift towards increased processing. The most common classification system NOVA (not an acronym) defines these newly formulated foods as ultra-processed foods (UPF), as they are subject to several physical and chemical transformations along with the addition of additives only available in industrial settings (Monteiro et al., 2018). While these new additions are widely unstudied, the little that is known provides reason for concern. Common emulsifiers and sweeteners have been shown to affect metabolic health (Bhattacharyya et al., 2012; Dalenberg et al., 2020); meanwhile, the long shelf life of UPF allows for greater absorption of compounds from packaging materials which are associated with increased type 2 diabetes risk (Hwang et al., 2018). In addition, high sodium content and the absence of whole grains are hallmarks of UPF (Monteiro et al., 2018). These characteristics are top contributors to poor diet quality, which is the leading risk factor for death globally, resulting in over ten million lives lost a year (Afshin et al., 2019). More specifically, UPF consumption has been associated with increases in dementia (Li et al., 2022), cardiovascular disease (Juul et al., 2021), cancer (Romaguera et al., 2021), and all-cause mortality (Schnabel et al., 2019). Despite these known negative health effects, UPF dominate the modern diet, making up around 58% of total calories consumed in the U.S., with this trend increasing year over year (Steele et al., 2016, 2023). Understanding the factors that are driving UPF consumption is critical to improve world health.

To understand this recent shift in dietary patterns it is important to acknowledge known factors that influence the consumption of UPF, such as advertising, specifically when targeting children (E. Boyland et al., 2022; E. J. Boyland & Whalen, 2015; Sadeghirad et al., 2016). Additionally, food security and availability are factors that further increase UPF intake (Cooksey-Stowers et al., 2017; Leung et al., 2022; Serafim et al., 2022). Yet, without these factors, UPF still seem to have unique characteristics that drive intake. In the only randomized controlled trial on UPF consumption to date, participants were fed a nutrient matched ultra-processed and minimally-processed diet for 2 weeks while living in a metabolic ward. This study provided prepared ready-to-eat foods without packaging, thus dramatically reducing, or eliminating influences of marketing, food availability, and insecurity. Regardless, in the ultra-processed diet period, participants consumed ~500 kcal more a day and gained more weight and fat mass (Hall et al., 2019). These findings suggest the potential for UPF to have unique rewarding characteristics that drive consumption; however, this has yet to be directly experimentally tested.

One well established method to examine the influence of food characteristics on preference is the Becker-DeGroot-Marschak (BDM) auction task. This experimental economics paradigm typically has individuals bid for a range of food or non-food items represented in pictures, a way to estimate willingness to pay (WTP), which requires development of a set of visual stimuli (Becker et al., 1964; Blechert et al., 2019; DiFeliceantonio et al., 2018; Fromm et al., 2021; Newton-Fenner et al., 2023; Perszyk et al., 2021; Plassmann et al., 2007; Tang et al., 2014). This paradigm has decades of experimental evidence demonstrating it measures participants true maximum WTP for each item, as if it were the only item (Newton-Fenner et al., 2023). A recent meta-analysis encompassing 24 BDM studies using functional magnetic resonance imaging (fMRI) demonstrated brain activation associated with WTP predominantly in the ventromedial prefrontal cortex (vmPFC), dorsolateral prefrontal cortex, and ventral striatum (VS) (Newton-Fenner et al., 2023). More specifically, BDM studies that have used food images demonstrated a higher WTP for foods with greater energy density and foods that contain high levels of both carbohydrates and fats (DiFeliceantonio et al., 2018; Perszyk et al., 2021; Tang et al., 2014). Furthermore, these studies also show associations with WTP and greater blood-oxygen-level-dependent (BOLD) activation in vmPFC and VS (DiFeliceantonio et al., 2018; Tang et al., 2014). However, work investigating the role of degree of food processing remains notably absent.

Prior pictures sets that have been used to test the effect of energy density (Tang et al., 2014) or macronutrient composition (DiFeliceantonio et al., 2018; Perszyk et al., 2021), cannot be repurposed to investigate food processing. These existing picture sets do not have enough distribution across NOVA categories and were intentionally made to vary on either energy density or macronutrient composition. Currently the only picture set that was designed to investigate degree of food processing contains familiar foods for the United Kingdom and therefore is not suitable for use in the United States (Rogers et al., 2024). In addition, it has fewer foods than needed for a robust functional magnetic resonance (fMRI) design. The absence of a picture set to study food processing in the U.S is particularly notable, given UPF consumption in the U.S. is amongst the highest in the world (Marino et al., 2021). Acknowledging this methodological gap, here we propose a new picture set comprised of 14 minimally and 14 ultra-processed foods. In Experiment 1, we aimed to build upon the rigor of previous studies to establish a picture set of foods familiar to people living in the United States that differs on degree of processing but is matched on a variety of visual, nutritional, and perceptual properties. Next in Experiment 2, we validate the classification using the NOVA system and our food pictures while providing new data on the use of the NOVA system by lay people and individuals with nutrition credentials.

Experiment 1

Methods

Picture set creation

A multi-disciplinary team comprising psychologists, neuroscientists, and registered dietitians familiar with American dietary patterns, selected 57 foods 3 of which were represented twice in different arrangements resulting in 60 food pictures, in line with the picture selection process by Fromm et al. (2021). The NOVA classification system was used to classify foods as minimally-processed (NOVA categories 1–3) or ultra-processed (NOVA category 4). This resulted in 24 food pictures in the minimally-processed category and 36 in the ultra-processed category. All pictures were taken internally with professional lighting and a professional camera. Each picture contained foods arranged on a white plate, and when appropriate foods were also placed in a clear bowl (Figure 1). To improve the consistency between pictures Adobe Photoshop 2022 was used to correct for any lighting inconsistencies and crop all pictures to the dimensions of 3412 by 2704 with 350 pixels/inch.

Fig. 1. Final picture set.

The final selected picture set of 28 foods matched across 26 characteristics, forming two groups comprised of 14 MPF – minimally processed foods and 14 UPF – ultra-processed food.

Food characteristics

Nutritional information (calories, carbohydrate, fat, protein, sodium, and dietary fiber) was collected for all foods with Nutrition Data System for Research (NDSR; version 2020; University of Minnesota, Minneapolis, MN). All NDSR food metrics were derived from the food weights actually depicted in the pictures. Similar to that of prior work (Fromm et al., 2021) the food price for each picture was calculated from unit price (USD/g) derived from Kroger.com with the zip code set to that of the Fralin Biomedical Research Institute at Virginia Tech Carilion (laboratory location).

Visual properties

Visual properties were analyzed following methods developed by Blechert et. al. (Blechert et al., 2014) and used in the final picture matching process to ensure no visual confounders exist between final picture sets. All visual properties (red, green, and blue color content; brightness; spatial frequency; within-object contrast; complexity; and normalized complexity) were computed using Matlab R2022a (The Mathworks Inc., Sherborn, MA, USA).

MTurk task

Respondents were recruited through Amazon Mechanical Turk (MTurk), a crowdsourced platform that provides users monetary compensation for completion of human intelligence tasks (HITs). Eligible respondents had not previously participated in the survey as verified through cross checking MTurk IDs, had lived in the United States for the last 5 years without more than 6 consecutive months outside of the US, had no food allergies or restrictions (e.g. vegan, vegetarian, gluten free, kosher, halal), and did not report having taste or smell impairments. All respondents that completed the entire survey received $3.00; an additional $2.00 was given to respondents that also passed the quality checks. The experimental protocol was approved by the Virginia Tech IRB (22-030) and participants digitally consented to participation.

Scale training

After agreeing to participate, qualified respondents were subsequently trained on the use of the labeled hedonic scale (LHS). The LHS scale is a category-ratio scale that was derived using magnitude estimation, allowing subjects to draw across all hedonic experiences ranging from “The Most Liked Sensation Imaginable” to “The Most Disliked Sensation Imaginable” (Lim et al., 2009). To verify that all respondents were using the scale correctly, the following criteria were used: 1) Participants had to rate the smell of vomit as either neutral or disliked; 2) Participants had to rate the taste of their favorite chocolate as either neutral or positive; and 3) Participants had to rate the taste of plain bread between the scale values of −30 and 30, which are closest to the indicators “Like Very Much” and “Dislike Very Much.”

Perceptual properties

As previous work has indicated many other factors influence food choice and value as measured by the BDM (DiFeliceantonio et al., 2018; Fromm et al., 2021; Tang et al., 2014), respondents made ratings for estimated calories, familiarity, expected satiety, healthiness using a horizontal visual analog scale (VAS) and liking using the LHS they were trained on as described above. Additionally, subjects also estimated the price of each item, as prior work has demonstrated the potential for estimated price to influence willingness to pay in the BDM paradigm (Perszyk et al., 2021). The exact rating questions and scale anchors are provided in (Table S1).

Response validation

Respondents that met the qualification requirements were further screened to ensure data quality and remove computer automated programs (bots). First, responses were removed if they failed to provide an adequate text response on the final prompt of the survey: “Please write a few words on what this survey was about.” Examples of adequate text responses mentioned food or making ratings (e.g. “rating various foods”, “Rating different foods on a plate”) and irrelevant or absent responses were excluded (e.g. “good”, “NONE”, “Useful”). Additionally, throughout the survey, respondents were presented with three foods (Veggie Straws; Chex Mix; and Sausage and Egg Breakfast Sandwich) twice with slightly different visual arrangements and in addition they were presented with an empty plate to create quality control metrics. A sensitivity analysis was performed using the responses: frequency of consumption between the duplicated foods, how closely the empty plate was rated to $0, and how many times a rating scale was not moved from the default starting position. Due to no large changes in perceptual rating means in the sensitivity data sets, the strictest criteria were used for subsequent analyses. Responses were included if they had the same input for frequency of duplicate foods; moved their cursor from the default position on all continuous ratings but 5 ratings; and estimated cost within $0.10, satiety within 10% of scale, and caloric content under 50 kcal for the empty plate.

Picture selection and data analysis

Initial food picture selection was performed with Microsoft Excel (Mac Version 16.79.2). We sought to identify food picture groups that where similar on all 27 of our measured variables. We found there was no possible match on healthiness, and therefore excluded it from later matching, leaving us with 26 variables. We iterated (Excel) through possible groups using means and p-values from a Welch’s t-test as our outcomes of interest for all 26 properties including food characteristics, visual properties, and perceptual ratings (descried in detail above). Through this process we created two groups with equal numbers of pictures. Welch’s t-test is more appropriate for samples with unequal variance and small sample sizes (Winter, 2013). These two groups had no significant differences on all food characteristics, except for perceived healthiness. Next, the following analyses for picture selection were performed with R (version 4.2.2, (Team & others, 2021)). To finalize the matching process, 10,000 permutations of each group were checked to identify combinations of the foods with p-values greater than 0.05 as computed by a Welch’s two sample t-test across groups for all 26 characteristics, in line with prior work (DiFeliceantonio et al., 2018; Fromm et al., 2021). Importantly, the alpha level was not adjusted for multiple comparisons as the goal was to test the lack of difference between groups and not correcting would be a more a stringent test. In addition, we set parameters to not allow for duplicates of any foods that are too similar (e.g. same food with different arrangements or more than one trail mix). After running 10,000 simulations, 26 combinations were identified that did not significantly differ on all 26 criteria. Next, the sum of Hellinger distance for all 26 characteristics were computed and used to rank-order the 26 food groupings by lowest distance (Hellinger Distance - Encyclopedia of Mathematics, n.d.; Hellinger, 1909). Hellinger distance is a metric which describes how far apart two distributions are and can be calculated by first using a kernel density estimator on the variable for each of the groups and then using the Hellinger distance formula on those two estimated densities. The final picture set was chosen from these 26 options after consideration of food variety and Hellinger distance (Figure 1).

Results

Participants

A total of 5680 HITs were initiated, of which 939 qualified based on the inclusion criteria, 810 passed the LHS training, and of those 558 finished the survey. Of the 558 that finished the survey, 197 passed the quality assurance checks. For those that finished the survey, it took a median of 29.6 minutes (interquartile range = 15.13 minutes). After additional quality screening 197 respondents were left and used for all primary analysis. Participant characteristics for Experiment 1 (n=197) are provided in Table 1. The majority of respondents (n = 134) were female and identified as white (n = 157).

Table 1.

Participant characteristics and demographics of the final sample used in Experiment 1.

	Demographic	n = 197
Age mean (SD)		41.46 (13.39)
Sex n (%)	Male	63 (32.0)
	Female	134 (68.0)
Gender n (%)	Male	63 (32.0)
	Female	133 (67.5)
	Non-binary	1 (0.5)
Race n (%)	American Indian or Alaska Native	3 (1.5)
	Asian	12 (6.1)
	Black or African American	19 (9.6)
	Native Hawaiian or Pacific Islander	1 (0.5)
	Other	3 (1.5)
	White	157 (79.7)
	More than one	2 (1.0)
Ethnicity n (%)	Yes, of Hispanic, Latino or Spanish origin	16 (8.1)
	No, not Hispanic, Latino or Spanish origin	180 (91.4)
	Prefer not to say	1 (0.5)
Education level n (%)	High School	18 (9.1)
	Some College	68 (34.5)
	College	74 (37.6)
	Post College	37 (18.8)
Smoking Status n (%)	Yes, I smoke	29 (14.7)
	I sometimes smoke	12 (6.1)
	I have smoked in the past	49 (24.9)
	I have never smoked	107 (54.3)

Picture set matching

Prior to final picture set selection, the original 60 food pictures were reduced to 44 due to inadequate equivalents in NDSR, missing data, and/or pictures containing more calories than the maximum allowed estimated calories value on the MTurk survey. Using the methods described above, our two picture set groups contained 14 pictures each for a total of 28.

All nine visual properties (blue, green, red, brightness, complexity, normalized complexity, object size, spatial frequency, and within-object contrast) were not different (t < 2, p > 0.05) between the UPF and MPF groups (Table 2a, Figure 2a–c). Spatial frequency was slightly, but not significantly, greater in the UPF group (t(19.11) = −2.09, p = 0.0505).

Table 2.

Descriptive characteristics and statistical comparisons by group (UPF – ultraprocessed food, and MPF – minimally processed food) for food characteristics, and perceptual properties.

	MPF (n=14)	UPF (n=14)
	Mean (SD)	Mean (SD)	Cohen’s d	DF	p
a. Visual
Blue	0.17 (0.08)	0.15 (0.06)	0.272	23.78	0.479
Green	0.35 (0.03)	0.34 (0.03)	0.383	25.75	0.320
Red	0.47 (0.08)	0.51 (0.07)	−0.424	25.54	0.273
Brightness	15.23 (13.75)	15.1 (6.39)	0.012	18.36	0.976
Complexity	0.03 (0.02)	0.04 (0.03)	−0.266	25.59	0.488
Normalized complexity	0.13 (0.07)	0.18 (0.11)	−0.585	21.98	0.136
Object size	0.22 (0.14)	0.2 (0.09)	0.185	21.88	0.630
Spatial frequency	17.13 (2.14)	18.46 (1.07)	−0.789	19.11	0.0505
Within-object contrast	36.49 (25.62)	40.65 (13.63)	−0.203	19.82	0.598
b. Food characteristics
Actual cost (USD)	1.17 (0.71)	1.17 (0.56)	0.001	24.69	0.999
Carbohydrate (g)	25.69 (24.25)	39.89 (16.7)	−0.682	23.07	0.084
Protein (g)	11.78 (7.91)	7.61 (5.07)	0.628	22.12	0.111
Fat (g)	14.32 (8.66)	10.24 (5.14)	0.573	21.14	0.145
Saturated fat (g)	5.96 (4.69)	3.26 (2.7)	0.705	20.77	0.076
Dietary fiber (g)	3.53 (3.2)	2 (1.73)	0.594	19.99	0.132
Sodium (mg)	256.3 (221.14)	435.8 (285.41)	−0.703	24.47	0.075
Calories (kcal)	270.86 (63.17)	276.82 (49.97)	−0.105	24.69	0.784
Weight (g)	146.28 (89.09)	105.42 (67.4)	0.517	24.21	0.184
Energy density (kcal/g)	2.51 (1.42)	3.24 (1.23)	−0.546	25.46	0.161
Portion size (g/kcal)	0.54 (0.3)	0.39 (0.26)	0.535	25.37	0.169
c. Perceptual
Estimated calories (kcal)	210.88 (44.06)	242.18 (47.36)	−0.684	25.87	0.082
Estimated cost (USD)	1.72 (0.56)	1.6 (0.35)	0.266	21.78	0.489
Familiarity (% of scale)	76.75 (6.52)	79.06 (6.75)	−0.348	25.97	0.366
Expected satiation (% of scale)	62.50 (9.53)	57.09 (13.35)	0.466	23.52	0.230
Frequency (ordinal)	1.57 (0.29)	1.4 (0.18)	0.729	21.90	0.067
Healthiness (% of scale)	63.86 (11.37)	39.3 (14.42)	1.891	24.65	<0.0001
Liking (range of scale 100 to −100)	29.00 (10.43)	25.23 (13.88)	0.308	24.13	0.424

Mean, standard deviation, Cohen’s d, and p-value from Welch’s t-test for visual properties, food characteristics, and perceptual properties.

Fig. 2. Picture properties.

Violin plots for visual properties (a–c), food characteristics (d–i), and perceptual properties (j–o) of the final 28 pictures. Each dot represents the mean value of perceptual properties or extracted value of nutrition and visual properties for a single picture. Mean is represented with a dash and error bars represent standard error of the mean. The violin plot represents the smoothed histogram of individual picture values. UPF – ultra-processed food group in orange and MPF – minimally processed food group in grey.

Next, actual cost and the nutritional properties of the foods: saturated fat, dietary fiber, sodium, calories, weight, and portion size did not differ (t < 2, p > 0.05) between the UPF and MPF groups (Table 2b and Figure 2h–i). Importantly, previous work has shown macronutrient content and Energy density can influence WTP (Figure 2d–g ) (DiFeliceantonio et al., 2018; Perszyk et al., 2021; Tang et al., 2014). We did not observe significant differences between MPF or UPF for any macronutrient: Carbohydrate (t(23.07) = −1.81, p = 0.084), Protein (t(22.12) = 1.66. p = 0.111), and Fat (t(21.14) = 1.52 p = 0.145) nor Energy density (t(25.46) = −1.45 p = 0.161). Although UPF typically contain less dietary fiber than MPF, the MPF and UPF in our picture set were matched on dietary fiber (3.53g for MPF, 2.00g for UPF; (t(19.99) = 1.57, p>0.05)).

Finally, we observed no differences in the perceptual properties: estimated calories, estimated cost, familiarity, expected satiation, frequency, or liking (t<2, p>0.05; Table 2c and Figure 2j–n). However, respondents rated healthiness higher for the MPF pictures compared with the UPF (t(24.65) = 5.00, p < 0.0001; Table 2c; Figure 2o).

Due to the majority of our sample being female, we performed a sensitivity analysis for the 7 rated properties to examine if they differed between female respondents only, male respondents only, and the whole mixed sample (Table S2). In our sample that passed quality checks there were no participants that did not identify as either male or female. Means were generally nearly identical across MPF and UPF regardless of the sex of the sample. Except for frequency of consumption, which did not differ between MPF and UPF in the female and mixed group, but did differ when examining male respondents only (UPF=1.62, MPF=1.42, t(21.08)=2.15, p=0.0435).

Experiment 2

Methods

Participants

A common critique of the NOVA system is it can be difficult to classify foods based on its criteria. One recent study found nutrition professionals in France had low inter-rater reliability when classifying foods using the NOVA system (Braesco et al., 2022). To attempt to replicate these findings and obtain a more objective measure of NOVA classification for the final picture set, in Experiment 2 we had raters assess the NOVA score for the final 28 food pictures from Experiment 1. Raters were recruited through social media, email listservs of nutrition professionals, and word of mouth. Individuals that were willing to participate were asked if they considered themselves a nutrition professional, to provide how many years of prior nutritional education they had received, and if they had obtained a nutritional certification. The only inclusion criterion was completion of the entire survey. Raters that completed the survey were given the option to be entered into a drawing for a gift card by providing an email address. The experimental protocol was approved by the Virginia Tech IRB (22-030) and consent was digitally obtained.

NOVA validation testing

The survey for Experiment 2 was built using R (version 4.2.2, (Team & others, 2021)) and visualized with RShiny (Chang et al., 2023). In line with prior work, raters were first presented with information for each of the four NOVA categories ensuring they were familiar with the NOVA classification system (Braesco et al., 2022; Monteiro et al., 2016, 2019). Subsequently, raters were shown all 28 foods in a random order. Each food was represented with the color picture (Figure 1), food name, and a full list of ingredients as found on the foods’ packaging. Importantly, summarized information about all four NOVA categories was provided to the raters throughout the entire survey. For each food picture, raters had to select a NOVA category (1, 2, 3, or 4) and rate their confidence on a continuous visual analog scale for “How confident are you about the selected NOVA score?”

Statistical Analysis

The collected data was analyzed in two ways: firstly, using the full NOVA classifications system ranging 1–4; and secondly the binary NOVA classification MPF (NOVA categories 1–3) or UPF (NOVA category 4) as used in Experiment 1. Krippedorff’s alpha was used to assess inter-rater reliability across all raters for both the four NOVA categories and binary (NOVA 1–3 vs 4) (Krippendorff, 1970). An alpha of 0.8 is considered reliable, in line with recommendations by Krippendorff. An alpha less than 0.8 but greater than 0.667 is classified as tentative agreement, suggesting moderate reliability (Krippendorff, 2004). Next, the relationship between rated confidence and selection of the correct NOVA classification, for the binary NOVA classification of MPF or UPF was tested. The correct NOVA classification for each picture was determined based on the NOVA category most frequently chosen by raters, denoting rater consensus. To examine this relationship, a generalized linear mixed effects model with binomial outcome and logit link was used, effectively creating a logistic regression that accounts for repeated measures. To facilitate interpretation, confidence was modeled as a categorical variable as opposed to a continuous variable, therefore multiple different bins of confidence were examined: categorical with 5-unit bins, 10-unit bins, and collapsed bins (0–60, 60–80, 80–90, 90–95, 95–100). Akaike’s information criteria (AIC) was used to compare the models.

Results

Participants

A total of 67 raters completed the survey. Of the 67 raters, 50 self-reported having no nutrition credentials and the remaining 17 reported credentials as a Registered Dietitian (RD / RDN; n=16) or Nutrition and Dietetics Technician, Registered (NDTR / DTR; n=1). As expected, the credentialed individuals had significantly more nutritional education than the non-credentialed individuals (p < 0.001). A majority (82.4%) of credentialed individuals reported completing graduate coursework in nutrition, and a majority (58%) of non-credentialed individuals reported no undergraduate nutrition coursework. In addition, all but one (94.1%) of credentialed individuals self-identified as a nutrition professional, which was significantly more than the 7% of non-credentialed individuals (Table 3).

Table 3.

Distribution characteristics for nutrition credential information for experiment 2.

	Credentialed (n=17)	Not Credentialed (n=50)
	n (%)	n (%)	p
a. Nutrition credentials			<0.001
None	0 ( 0.0)	50 (100.0)
Nutrition and Dietetics Technician, Registered (NDTR / DTR)	1 ( 5.9)	0 ( 0.0)
Registered Dietitian (RD / RDN)	16 (94.1)	0 ( 0.0)
b. Nutrition education			<0.001
No undergraduate nutritional coursework	1 ( 5.9)	29 ( 58.0)
Some undergraduate coursework in nutrition	0 ( 0.0)	6 ( 12.0)
Completed undergraduate coursework in nutrition (associate’s degree)	0 ( 0.0)	1 ( 2.0)
Completed undergraduate coursework in nutrition (bachelor’s degree)	0 ( 0.0)	1 ( 2.0)
Some graduate coursework in nutrition	2 (11.8)	7 ( 14.0)
Completed graduate coursework in nutrition	14 (82.4)	6 ( 12.0)
c. Self identifies as nutritional professional			<0.001
No	1 ( 5.9)	43 ( 86.0)
Yes	16 (94.1)	7 ( 14.0)

Nutrition credential information for participants in Experiment 2. P-values from Fischer’s exact test.

NOVA classification – full range analysis

The inter-rater reliability analysis for the full range of NOVA classifications (1–4) for the 28 pictures demonstrated tentative agreement amongst all raters (α = 0.78) (Figure S1a). Raters with nutrition credentials had greater NOVA classification agreement (α = 0.80) than those without (α = 0.78) (Figure S1 b–c). Interestingly, raters with nutrition credentials used the NOVA 2 category less, which is comprised of culinary ingredients typically used only in recipes and not consumed in isolation (e.g. sugar, salt, and oil). For instance, rater consensus for Oatmeal and Honey was NOVA 3 for raters with nutrition credentials but was NOVA 2 for raters without nutrition credentials. The 14 pictures in the UPF group from Experiment 1 all had rater consensus for NOVA 4. Unsurprisingly, most misclassifications for these pictures were NOVA 3 (Figure S1). For the 14 MPF group pictures there was less consistency across the four NOVA categories, with the exceptions of Apples and Pecans, and Hardboiled Eggs, which had almost perfect consensus for NOVA 1. The remaining 12 food pictures had large contributions in multiple categories although NOVA 3 was the rater consensus (Figure S1).

NOVA classification – binary analysis

Next, we assessed the pictures with the binary MPF (NOVA 1–3) and UPF (NOVA 4) classifications. Rater consensus for all 28 food pictures agreed with our initial binary MPF and UPF classifications developed in Experiment 1 (Figure 1), demonstrating consistency in our categorization methodology. Across all raters, the binary classification inter-rater reliability had tentative agreement (α = 0.76; Figure 3a). Unsurprisingly, there was stronger agreement amongst individuals with nutrition credentials (α = 0.82, indicating strong agreement) than individuals without nutrition credentials (α = 0.74; table 3b–c).

Fig. 3. Rater consensus.

There is tentative agreement in categorizing the 28 foods from Experiment 1 as MPF – minimally processed foods (NOVA 1–3) or UPF – ultra-processed foods (NOVA 4) amongst (a) all raters (α = 0.76), (b) individuals with no nutrition credentials (α = 0.74), and (c) stronger agreement amongst individuals with nutrition credentials (α = 0.82). UPF – ultra-processed food in orange. MPF – minimally processed food in grey.

Rater confidence – binary analysis

Across all raters, confidence ratings were not uniformly distributed; very few responses were rated below 50%, as most responses had high confidence, specifically in the range of 95 to 100% (Figure 4). Correct responses, as determined by group consensus for the NOVA binary, were also unevenly distributed across the confidence levels, with the proportion of least correct responses being in the lower confidence levels. This trend is consistent across both individuals with and without nutritional credentials (Figure 4). Therefore, we broke up the confidence rating ranges into 5 intervals: 0–60%, 60–80%, 80–90%, 90–95% and 95–100% (Table 4). Using these ranges, an association was observed between rated confidence and the odds that the chosen NOVA classification aligns with the consensus across all raters (Table 4). When confidence ratings were between 95 and 100% there was 4.55 times higher odds that the chosen NOVA classification agreed with the consensus when compared to confidence ratings between 0 and 60% (z = 4.78, p < 0.001) among all raters (Figure 4a). Similarly, among individuals with no nutritional credentials, the odds that a rating made with a 95 to 100% confidence rating agreed with consensus was 5.38 times higher than a rating made between 0 and 60% (z = 4.78, p < 0.001). However, for individuals with nutrition credentials there is not a statistically significant difference for ratings made between 0 and 60% or 95 and 100% confidence. Although this might be explained by a greater percentage of confidence ratings in the 95 to 100% range and lower overall responses that do not agree with the consensus (Figure 4).

Fig. 4. Rater confidence.

Distribution of confidence ratings by agreement with the consensus for (a) all raters, (b) individuals with no nutrition credentials, (c) individuals with nutrition credentials. Shaded height of the bar represents level of agreement with the rater consensus. Numbers within the bars represent the percentage of ratings out of all ratings in the rated confidence bin.

Table 4.

Linear mixed effects model (logit link) results for correct binary NOVA classification regressed on evaluator confidence.

	All Participants				No Nutritional Credentials				Nutritional Credentials
Confidence	Odds Ratios	95% CI	z	P	Odds Ratios	95% CI	z	P	Odds Ratios	95% CI	z	P
0–60	Reference				Reference				Reference
60–80	2.26	1.26 – 4.03	2.75	0.006	2.13	1.16 – 3.92	2.42	0.015	2.8	0.54 – 14.64	1.22	0.221
80–90	2.89	1.31 – 6.38	2.63	0.008	2.79	1.22 – 6.40	2.42	0.016	4.02	0.36 – 44.39	1.14	0.256
90–95	3.8	1.17 – 12.32	2.23	0.026	4.56	1.21 – 17.11	2.25	0.025	2.32	0.19 – 28.08	0.66	0.508
95–100	4.55	2.45 – 8.47	4.78	<0.001	5.38	2.70 – 10.74	4.78	<0.001	3.01	0.71 – 12.81	1.49	0.137
N	67				50				17

Evaluator confidence on NOVA scoring for all participants, individuals with no nutrition credentials, and individuals with nutrition credentials for the Binary NOVA analysis. Odds ratio represents agreeing with the consensus compared to the 0–60 confidence group. CI- confidence interval.

Discussion

Given the ubiquity of ultra-processed foods in the diet of the average American, a set of pictures suitable for neuroimaging and common economic methodologies is an essential research tool. Here, we provide the methodology and validation of a picture set containing 28 (14 MPF, 14 UPF) foods matched on nutrition characteristics, to allow for the testing of the hypothesis that processing independent of macronutrient content is driving observed effects. These pictures are also matched on low level visual characteristics, making this picture set suitable for neuroimaging, including functional magnetic resonance imaging (fMRI), methodologies. Finally, these pictures are matched on rated perceptual characteristics from a large sample of raters. We combined previous (iterative testing) and advanced (simulations and Hellinger distances) methodologies to arrive at the final set, which is now available for download to researchers.

While our food picture set is matched on 26 characteristics, we were not able to match on perceived healthiness. Perceived healthiness has previously been shown to affect selection of portion size, estimated energy density, and ad libitum consumption (Faulkner et al., 2014; Labbe et al., 2017; Provencher et al., 2009). Although this pictures set was matched on actual energy density and portion size, perceived healthiness could still contribute to outcomes, such as estimated energy density, and should be considered in future experimental work using this picture set. However, it is also important to acknowledge that degree of food processing and perceived healthiness may be inherently interconnected and future work is required to determine if it is possible to dissociate these characteristics.

Ultra-processed food intake is emerging as one of the leading causes of cancer and all-cause mortality (Juul et al., 2021; Li et al., 2022; Romaguera et al., 2021; Schnabel et al., 2019). Despite many factors likely leading to high levels of consumption (E. Boyland et al., 2022; E. J. Boyland & Whalen, 2015; Cooksey-Stowers et al., 2017; Leung et al., 2022; Sadeghirad et al., 2016; Serafim et al., 2022), this trend persists even in the controlled environment of a metabolic ward (Hall et al., 2019). This indicates there are likely other factors driving their reinforcing value and over consumption. The post-ingestive effects of these foods is one potential mechanism (Small & DiFeliceantonio, 2019). Post-ingestive signals can change eating behavior and food choice and are conveyed to the brain (de Araujo et al., 2013; Ren et al., 2010; Veldhuizen et al., 2017). The potential for greater bioavailability of calories from UPF earlier on in digestion and absorption, could amplify these post-ingestive signals and lead to these foods’ increased rewarding and appetitive effects (Carmody et al., 2011; Lennerz et al., 2013; Wolever et al., 2019).

The NOVA system for classifying foods has the most empirical support in population wide studies (Hall et al., 2019; Huybrechts et al., 2022; Juul et al., 2021; Li et al., 2022; Monteiro et al., 2016, 2018, 2019; Monteiro & Astrup, 2022; Romaguera et al., 2021; Schnabel et al., 2019; Steele et al., 2016, 2023) and is employed here to classify the foods into UPF and MPF. However, the NOVA system is not without its limitations (Astrup & Monteiro, 2022). A common critique of the system is that it is difficult to classify foods. One recent study in France reported minimal agreement between raters on NOVA categories, even when ingredient information was provided to raters (Braesco et al., 2022). We wanted to ensure our NOVA classified groups of UPF and MPF agreed with other nutrition professionals and provide additional data on the utility of the NOVA system for classification. In addition, we collected NOVA ratings of a non nutrition credentialed population to better understand the generalizable use of the NOVA system. Overall, we found the classifications performed by our team were in line with other nutrition professionals, and in contrast to previous findings, our raters had tentative to strong agreement as measured by Krippendorff’s alpha. Credentialed nutrition professionals had strong agreement for both the binary classification (NOVA 1–3 vs 4) and full category classification. While individuals without nutritional credentials provided the same rater consensus for the binary classification (NOVA 1–3 vs 4) as the individuals with nutrition credentials but differed slightly for the full category classification. Explanations for differing findings could be we asked raters to rate fewer foods (28 vs ~120), our foods were designed to differ on NOVA score; and we presented raters with pictures (Braesco et al., 2022).

Limitations and Future Directions

Our MTurk sample for Experiment 1 had a greater representation of individuals that are female, highly educated, and white. We therefore recommend repeating key ratings in the sample of interest when completing a study using this picture set. We could not match our picture set on ratings of “Healthiness” as MPF foods were rated as significantly healthier than UPF foods. It will be useful to include this variable as a covariate in future analyses. In Experiment 2, it is likely our non-credentialed sample had greater nutrition education than the general population, therefore these results should be compared with those from other studies surveying those with nutrition education or credentials.

Here, we present a picture set of 28 foods commonly consumed in the US for use in behavioral economic and neuroimaging studies. All pictures, data, and code used to derive the matched groups (including pictures not chosen for the final set) will be made available to researchers. These tools will allow researchers to test hypotheses regarding the level of processing of foods independent of macronutrient content, including difficult to match nutrients such as protein and dietary fiber.