DietAI24 as a framework for comprehensive nutrition estimation using multimodal large language models

Abstract

Background

Accurate dietary assessment is essential for health research. While smartphone-based food image recognition offers a convenient alternative to traditional methods, existing computer vision approaches struggle with real-world food images and analyze only basic macronutrients, limiting their utility for comprehensive nutritional research.

Methods

We developed DietAI24, a framework for automated nutrition estimation from food images that combines multimodal large language models (MLLMs) with Retrieval-Augmented Generation (RAG) technology to ground the MLLM’s visual recognition in authoritative nutrition databases rather than relying on the model’s internal knowledge. In our work, we used the Food and Nutrient Database for Dietary Studies (FNDDS) as the authoritative nutrition database. Through this approach, DietAI24 enables accurate nutrient estimation without extensive data collection or model training.

Results

DietAI24 significantly outperforms existing methods when evaluated against commercial platforms and computer vision baselines using the ASA24 and Nutrition5k datasets. Performance is measured through mean absolute error (MAE). DietAI24 achieves a 63% reduction in MAE for food weight estimation and four key nutrients and food components compared to existing methods when tested on real-world mixed dishes (p < 0.05). Notably, DietAI24 estimates 65 distinct nutrients and food components, far exceeding the basic macronutrient profiles of existing solutions.

Conclusions

DietAI24 demonstrates that integrating MLLMs with RAG and standardized nutrition databases can substantially improve dietary assessment accuracy while enabling comprehensive nutrient analysis. This framework offers a scalable solution for nutrition research and clinical applications, potentially transforming large-scale epidemiological studies and personalized dietary interventions through more accurate and less burdensome dietary data collection.

Plain language summary

Taking photos of food to track nutrition is convenient, but current apps often guess wrong about what nutrients are in your meals. We created DietAI24, a system that combines artificial intelligence with a trusted nutrition database. When someone takes a food photo, our AI recognizes the foods and looks up their exact nutritional values instead of guessing. We tested DietAI24 against popular nutrition apps using thousands of food images. Our system was 63% more accurate and can identify 65 different nutrients, not just calories and protein, but also important micronutrients like vitamin D, iron, and folate that affect your health. This technology could help researchers better understand diet-related diseases and help doctors give personalized nutrition advice. It makes tracking what you eat easier and more reliable for everyone.

Yan et al. present DietAI24, a unified framework that identifies foods, estimates portion sizes, and computes 65 nutrients from real-world food photos using multimodal large language models. This tool outperforms existing methods and standardizes outputs for downstream analysis, enabling efficient, reproducible dietary measurement.

Introduction

Dietary assessment helps us understand how nutrition affects health, disease prevention, and public policy^1–3. The 24-h dietary recall (24HR) has become the gold standard for dietary assessment and has been incorporated into major health initiatives, including the National Health and Nutrition Examination Survey (NHANES) in the United States^4–6. However, the 24HR method is retrospective, requiring participants to recall all foods and beverages consumed during the previous day. This approach faces several inherent challenges: it relies heavily on participants’ memory, leading to omissions and underreporting; it causes cognitive fatigue; and whether conducted through interviewer-administered recalls or self-administered digital platforms, it remains time-consuming and resource-intensive^5,7,8.

In response to these challenges, researchers have increasingly turned to prospective methods that capture dietary intake in real-time using smartphone cameras and artificial intelligence for automated dietary assessment from food images^9–11. This approach has been explored through computer vision techniques for tasks such as food classification⁹, ingredient recognition¹², food segmentation¹³, and nutrition estimation^11,12. While this streamlines dietary assessment by enabling users to simply photograph and upload images of their meals for automated nutritional analysis, existing implementations face substantial limitations. Several commercial platforms now offer food image recognition capabilities, but these systems struggle with practical utility¹⁴. First, their reliance on predefined food categories fails to capture the nuanced variations in preparation methods and ingredient combinations that substantially impact nutrient content¹³. Second, most systems focus primarily on basic macronutrients, lacking the comprehensive nutritional analysis capability required for clinical research and specialized dietary interventions¹⁵. Third, these systems often struggle with accuracy in real-world conditions, where lighting, presentation, and portion sizes vary considerably¹⁶.

Beyond traditional computer vision approaches, recent advances in Multimodal Large Language Models (MLLMs) have shown potential for dietary applications. Sun et al. built a custom neural network integrated with GPT models to provide personalized dietary advice for diabetes using meal images¹⁷. Lo et al. demonstrated promising accuracy and generalization in food detection and dietary assessment for five common nutrients without fine-tuning¹. However, these approaches rely on the original MLLMs without incorporating nutrition domain knowledge. While Yin et al. attempted to specialize MLLMs for food tasks through prompt tuning, they still lacked the detailed nutritional values essential for comprehensive dietary assessment¹⁸.

To overcome these existing limitations, we developed DietAI24, a framework that combines MLLMs with Retrieval-Augmented Generation (RAG)¹⁹ technology. DietAI24 adopts a prospective-style paradigm where users photograph their meals in real time, reducing memory-related errors inherent in traditional 24HRs. Unlike existing tools that require extensive training data for each food type, DietAI24 uses MLLMs for visual recognition and RAG to query nutrition databases, enabling zero-shot estimation of 65 distinct nutrients and food components (for a full list, see ref. ²⁰). This approach addresses a fundamental challenge in using MLLMs for nutrition analysis: while MLLMs excel at understanding visual content and recognizing food items, they often generate unreliable nutrition values^21–24. This is because MLLMs lack access to authoritative nutrition databases during inference. This hallucination problem is particularly critical in dietary assessment, where incorrect values could compromise health research and clinical decisions.

RAG addresses this limitation by augmenting MLLMs with external knowledge bases, transforming unreliable nutrient generation into structured retrieval from validated sources. DietAI24 implements this approach by incorporating the Food and Nutrient Database for Dietary Studies (FNDDS, version 2019–2020), which provides standardized nutrient values for 5624 foods commonly consumed in the United States²⁵. Through RAG technology, DietAI24 grounds MLLM responses in this authoritative database, enabling accurate identification of specific food variants and comprehensive nutrient estimation. This approach not only enhances dietary assessment precision but also establishes a versatile framework applicable to other domains requiring systematic integration of visual analysis with specialized knowledge repositories.

In summary, the main contributions of our work include three key achievements. First, we present a successful integration of MLLMs with RAG technology for food image analysis, achieving a 63% reduction in mean absolute error (MAE) for nutrition content estimation compared to existing approaches, with superior accuracy across diverse food types and portion sizes. Second, we demonstrate zero-shot estimation of comprehensive nutrients without requiring food-specific training data. Third, we develop a scalable approach that can be adapted to different regional food databases and nutritional standards. In the following sections, we detail DietAI24’s architecture, evaluate its performance against existing methods, and discuss implications for dietary assessment research.

Method

Problem definition

The primary objective of diet assessment from food images is to estimate the nutrient content of foods consumed. Achieving this goal requires (1) recognizing the food items present and (2) estimating their portion sizes. We formalize the problem as three interdependent subtasks, presented in the logical order of their dependency and ultimate importance.

Nutrient content estimation

Given a food image I, estimate the nutrient content amount vector N = {N₁, N₂, …, N_l}, where each N_i represents the total amount of the i-th component (including nutrients and other food components) in the depicted meal, and l = 65 as defined in FNDDS.

Food recognition

Identify all food items present in the food image I as a set of food codes ${\mathcal{C}}_I=\left\{c_{I,1},c_{I,2},\dots ,c_{I,k}\right\}$, where each c_I,j belongs to a standardized food code ontology $\mathcal{C}$, and k is the number of distinct food items recognized in the image.

Portion size estimation

For each recognized food item c_I,j, estimate its portion size p_I,j, selecting from the set of FNDDS-standardized qualitative descriptors, such as 1 roll, 3 cups, 10 pieces (see Supplementary Table 1 for additional examples). Let ${\mathcal{P}}_I=\left\{p_{I,1},p_{I,2},\dots ,p_{I,k}\right\}$ denote the set of portion sizes estimated for all k detected food items in the food image I.

Food recognition is formulated as multilabel classification, requiring models to distinguish among thousands of visually similar food codes c_I,j. We frame portion size estimation as multiclass classification rather than regression to match how portion sizes are defined in the FNDDS database. For each recognized food item c_I,j, the system selects an appropriate portion size p_I,j from standardized options. Nutrient content estimation, the most challenging task, depends on accurate results from the previous two steps. Models must integrate the set of recognized food codes ${\mathcal{C}}_I$ and their estimated portion sizes ${\mathcal{P}}_I$ to compute the nutrient content vector N. As the number of estimated nutrients l increases, so do the potential applications in health research and clinical practice.

DietAI24 framework

Figure 1 shows DietAI24’s architecture. DietAI24 estimate the nutrient content of an input food image I through three main steps: indexing the nutrition domain database (as detailed in Supplementary Fig. 1), retrieving relevant food information (as detailed in Supplementary Fig. 2), and estimating nutrient content (as detailed in Supplementary Fig. 3). In the indexing step, the nutrition database is segmented into small chunks to provide concise and MLLM-readable food descriptions. The retrieval step identifies relevant food description chunks based on queries derived from the input image. For the estimation step, we uses an MLLM to predict nutrient estimations using the retrieved information. Specifically, FNDDS serves as the nutrition domain database, and the GPT Vision model is used as the MLLM for all image-to-text reasoning. We use LangChain²⁶ for efficient retrieval. The exact prompt templates used for each step of our process are provided in Supplementary Figs. 4 and 5. These prompts contain specific instructions for the MLLM to recognize food items, estimate portion sizes, and calculate nutrient content based on the retrieved information.

Fig. 1. An overview of proposed nutrition estimation framework DietAI24 integrating MLLMs and RAG technology.

DietAI24 processes food images through three stages. First, the FNDDS knowledge base is indexed into a vector database for efficient retrieval. When a food image is input, the GPT Vision model generates descriptions that are expanded into multiple queries to retrieve relevant food information from the database. Finally, the system identifies specific FNDDS food codes, estimates portion sizes, and calculates comprehensive nutrient content.

Indexing nutrition domain database

The retrieval knowledge source is the FNDDS database, which includes J = 5624 unique food items (4982 food and 642 beverages), each identified by an 8-digit food code c_j with a detailed textual description. The database also contains over 23,000 portion sizes of foods and beverages, reflecting common consumption patterns or frequently consumed measures, such as slices, pieces, or cups. For each food code, the FNDDS database provides values for l = 65 components comprising 65 nutrients and food components. Consider $\mathcal{F}$ as the FNDDS database, each food code c_j has a food description desc (c_j) including details on form, preparation, and source. These descriptions are transformed into embeddings e_j using OpenAI’s text embedding model (text-embedding-3-large) and stored in a vector database $\mathcal{V}$ for efficient similarity-based retrieval. The indexing step is only performed once. Incremental indexing can be conducted when the new version of the nutrition database is released.

Retrieving food information

Given an input image I, the retrieval process generates a detailed description d using the GPT Vision model (GPT_Vision_Model). This description is expanded into multiple queries {q₁, q₂, …, q_m} by a GPT model to improve retrieval robustness. Each query is embedded and used to perform a similarity search in the vector database $\mathcal{V}$, yielding a set of candidate food codes for the input image, denoted as ${\mathcal{C}}_I=\left\{c_{I,1},c_{I,2},\dots ,c_{I,r}\right\}$. Duplicates are removed, and this candidate set serves as input to the food code inference step.

Estimating nutrient content

This step infers which food codes are actually present in the image, estimates their portion sizes, and computes the nutrient content through four sequential processes. In the food code inference process, the GPT_Vision_Model analyzes the candidate food codes ${\mathcal{C}}_I$, and infer the final set of recognized FNDDS food codes actually present in the food image, denoted as ${\mathcal{C}}_I=\left\{c_{I,1},c_{I,2},\dots ,c_{I,k}\right\}$, where k is the number of recognized food items. During portion size estimation, for each recognized food c_I,j, retrieve the set of standard portion sizes ${\mathcal{P}}_{I,j}$ and units ${\mathcal{U}}_{I,j}$ from the FNDDS database. The GPT_Vision_Model selects the most appropriate portion size p_I,j (e.g., ‘1 roll’, ‘3 cups’), which is then mapped to its gram-equivalent value, grams(p_I,j), using the FNDDS database. If portion size adjustment is needed and no standard portion size ${\mathcal{P}}_{I,j}$ is suitable, the GPT_Vision_Model infers an adjusted portion size based on common measurement units ${\mathcal{U}}_{I,j}$ (e.g., ‘roll’, ‘cups’). Finally, during nutrient content calculation, for each recognized food item c_I,j, retrieve its nutrient vector ${\mathbf{v}}^{\left(j\right)}=\left[v_1^{\left(j\right)},\dots ,v_l^{\left(j\right)}\right]$ from the FNDDS database. As defined in FNDDS, each $v_i^{\left(j\right)}$ represents the amount of the i-th component per 100 grams of c_I,j. The contribution from each food item to each nutrient is calculated as:

$${\mathbf{N}}_{c_{I,j}}={\mathbf{v}}^{\left(j\right)}\cdot \frac{\mathrm{grams}\left(p_{I,j}\right)}{100},$$

where ${\mathbf{N}}_{c_{I,j}}$ is the nutrient vector for food c_I,j adjusted by its portion size.

The total nutrient content vector for the food image is then obtained by summing contributions across all recognized food items:

$$\mathbf{N}={\sum }_{j=1}^k{\mathbf{N}}_{c_{I,j}},$$

where k is the number of food items recognized in the food image I.

The system first selects a portion size for each food and looks up its gram-equivalent value as defined in the FNDDS database. By multiplying the nutrient content per 100 grams by the actual weight of each portion, the model provides an accurate estimate of how much of each nutrient is present in the entire meal shown in the input image.

Experiment setup

We compared DietAI24 to commercial platforms, computer vision models, and the original GPT Vision model using two datasets. We measured accuracy with MAE defined as the average absolute difference between predicted and reference nutrient values. To assess system usability, we used the success rates (SR), which represent the proportion of food images for which the system was able to generate valid nutritional estimates. Additionally, we conducted an expert review with two dietitians to assess the system’s practical performance on real-world food scenarios. A full description of evaluation metrics is provided in the “Evaluation metrics” section.

Datasets

We chose two public food image datasets for our evaluation: ASA24⁵ and Nutrition5k¹². The two datasets, developed following the United States Department of Agriculture (USDA) guidelines²⁷, contain accurate nutrient labels representative of American foods.

ASA24, developed by the National Cancer Institute, includes over 17,000 portion-size images linked to food codes and portion sizes in the FNDDS database⁵. Each image corresponds to a specific food code, and one food code can have several images with various portion sizes. All images were taken in a controlled photography setting. This dataset was created to provide accurate portion sizes for a wide variety of food items, resulting in high-quality standardized images. This dataset is ideal for testing the accuracy of DietAI24 in inferring food codes and portion size.

Nutrition5k offers 3500 RGB-D images of 5000 dishes made from over 250 unique food items¹². Each image is paired with detailed nutritional information, including dish weight, energy, and macronutrients (fat, carbohydrates, and protein). Compared to the ASA24, Nutrition5k contains a broader variety of mixed dishes that are commonly served in U.S. cafeteria environments. The foods and beverages that were used in developing this dataset were offered in Google cafeterias in California, USA. Additionally, the image angles in Nutrition5k resemble those commonly used when people take photos with their mobile phones, providing results that more accurately reflect real-life scenarios. Unlike ASA24, however, Nutrition5k does not provide ground truth labels for food codes or portion sizes. Instead, dish weight serves as the primary quantitative measure of the amount of food in each image.

As our study involved only secondary analysis of publicly available, de-identified datasets with no human subject interaction, additional IRB approval was not required.

Data processing

The ASA24 and Nutrition5k datasets each contain numerous food images. Due to time constraints and API usage limitations, we selected representative subsets from both the ASA24 and Nutrition5k datasets. For the ASA24 dataset, we identified the top 1000 food codes using NHANES consumption frequency data, representing 81.4% of daily U.S. food consumption to ensure experimental generalizability. Since each food code includes multiple images showing different portion sizes, we selected three images per food representing the largest, smallest, and median portions, yielding 3000 total images. This approach enables comprehensive evaluation of our method’s nutrient analysis capabilities across varying portion sizes, which is essential for accurate dietary assessment and nutrition planning. For the Nutrition5k dataset, we removed duplicate images with identical food item lists and randomly selected 1000 images from the remaining dataset. All images from both datasets were standardized to 256 × 256 pixel resolution for consistent baseline comparisons, with no additional preprocessing applied to preserve the original visual characteristics of food presentations.

Evaluation metric

We evaluate DietAI24’s performance from four aspects: nutrient content estimation, food code inference, portion size estimation, and usability. In this study, accuracy refers specifically to the validity of the model’s predictions, as measured by their agreement with ground truth values. Food code inference, portion size estimation, and usability each contribute to the overall accuracy of nutritional information estimation, which serves as the most important and ultimate result of our system. For nutrient content estimation, we calculate the MAE by comparing the model’s estimated nutrient amounts to the corresponding nutrient labels. The estimated values include food weight, energy, and nutrients, where DietAI24 calculates food weight by multiplying the estimated portion size by the corresponding weight (g) per portion from the FNDDS database, as described in the “Estimating nutrient content” section.

The ASA24 enables the evaluation of food code inference, as each image in the ASA24 dataset is linked to a food code label from the FNDDS database, which uses a systematic classification scheme. This scheme assigns codes where the first digit corresponds to one of nine major food commodity groups: Milk and Milk Products; Meat, Poultry, Fish, and Mixtures; Eggs; Dry Beans, Peas, and Other Legumes, Nuts, and Seeds; Grain Products; Fruits; Vegetables; Fats, Oils, and Salad Dressings; Sugars, Sweets, and Beverages. The first two digits further refine the classification by identifying subgroups within these main categories. Due to this systematic classification approach, we design four distinct metrics to assess the accuracy of the food code inference. The Exact Match metric represents the percentage of instances where the model accurately identified the food in the image, with the entire inferred food code matching the labeled food code exactly. The Close Match metric indicates the percentage of instances where the model identified the food with minor inaccuracies but correctly matched the first two digits of the inferred and labeled food codes, thereby placing the inferred food within the correct subgroup. The Far Match metric reflects cases where the model made substantial identification errors but still correctly classified the food within the same major food group as the labeled code. Finally, the Mismatch metric denotes the percentage of instances where the model entirely misidentified the food.

To evaluate the accuracy of portion size estimation, we use two main metrics: Classification Accuracy and Prediction Error. In this study, portion size is defined according to the standard descriptors used in the FNDDS database (e.g., 1 roll, 3 cups), each with an associated weight (g) specified by FNDDS. Classification Accuracy represents the proportion of food images for which the model correctly identifies the portion size as one of these predefined FNDDS categories, matching the ground truth label. Prediction Error is calculated as the absolute difference (in grams) between the model’s estimated portion size and the actual portion size for each image. Hallucinations are an important challenge when using MLLMs, especially in specialized applications like ours. In DietAI24, the GPT model sometimes encounters difficulties with tasks such as identifying foods from images and estimating their portion sizes. We evaluate the usability of DietAI24 using the SR, which measures the proportion of images where our system successfully perceives information, including food categories and sizes. For this evaluation, we consider only instances where DietAI24 generates responses related to food codes and portion sizes, regardless of their accuracy.

Expert review

We conducted an expert evaluation using the Nutrition5k dataset to validate DietAI24’s practical capabilities in real-world food analysis scenarios. Two expert dietitians assessed the system’s performance on ten nutritional components they identified as most reliably determinable through visual inspection: food items, energy, fats, proteins, carbohydrates, sugar, fiber, caffeine, alcohol, and water. The evaluation was based on a randomly sampled subset of images from the Nutrition5k dataset to ensure representative coverage of diverse meal types. Although DietAI24 can analyze up to 65 different nutrients and food components, this evaluation concentrated on these specific components to ensure meaningful assessment of the system’s visual inference capabilities. The evaluation utilized a standardized 4-point Likert scale to ensure consistent assessment. The scale included four categories: Exact Match (4), indicating complete alignment between prediction and ground truth, Close Match (3) for predictions with minor deviations but maintaining overall accuracy, Far Match (2) for predictions with notable deviations while preserving basic correctness, and Mismatch (1) for incorrect predictions requiring considerable improvement. For example, in food item identification tasks, experts applied these ratings systematically. They awarded Exact Match ratings when DietAI24 successfully identified all food items. Close Match ratings were given for generally accurate predictions that included some errors, particularly in cases where even human experts found food item identification challenging.

Baseline models

We compare DietAI24 against five baseline models for the accuracy of nutritional analysis, including three commercial mobile applications, a computer vision model we trained, and the original GPT Vision model API by querying specific nutrient quantities. The three commercial mobile applications, Calorie MAMA²⁸, Foodvisor²⁹, and SnapCalorie³⁰, can analyze food images to determine nutrient content, including food weight, energy, fats, carbohydrates, and proteins. The APIs of these applications were used to compare their performance with our method. We also train a Vision Transformer (ViT)-based multitask regression baseline model for nutrient content estimation³¹. In this baseline, the pre-trained ViT (google/vit-base-patch16-224-in21k) serves as the feature extractor, while a multitask ElasticNet handles the regression. We freeze the weights of the ViT and train the multitask ElasticNet on the official Nutrition5K training set. The nutrient estimation on ASA24 is detailed in our paper. Since the SnapCalorie and ViT baseline models were trained on the Nutrition5K dataset, they will not be included in the experiment on the Nutrition5K dataset for fairness.

MLLM selection

In selecting the appropriate MLLM for our dietary assessment framework, we conducted a comprehensive evaluation of available models. OpenAI’s GPT model has shown strong performance across various vision-language tasks, comparing favorably with other major models, including Google’s Gemini and Anthropic’s Claude^32–34. The model demonstrates effective capability compared to open-source MLLMs such as LLaVA and CogVLM in tasks requiring detailed scene understanding and complex reasoning^35,36. This consistent performance, particularly in zero-shot settings, made GPT Vision models a suitable choice for our framework.

To understand how advances in MLLM technology impact DietAI24’s performance in nutrient estimation, we specifically examine two implementations within OpenAI’s GPT model family: GPT-4 and GPT-4 Turbo Vision models. These models share the same fundamental architectural structure, but GPT-4 Turbo represents an advancement through extensive fine-tuning, algorithmic optimizations, and access to expanded datasets. By comparing these two models, we can directly measure how improvements in MLLM capabilities translate to enhanced accuracy in dietary assessment tasks. DietAI24 employs these pre-trained models in a zero-shot manner, without any fine-tuning.

Statistics and reproducibility

We compared DietAI24 against baseline models using paired t-tests on MAE across nutrients and food components. Our evaluation used 3000 ASA24 images (stratified across three portion sizes) and 1000 Nutrition5k images; details and justifications are present in the “Data processing” section. For DietAI24, two versions of the GPT model were used: GPT-4 Vision-Preview (GPT-4) and GPT-4 1106-Preview (GPT-4 Turbo). Each image was processed once per model with deterministic settings (temperature = 0 for all API calls) to ensure reproducibility. Two dietitians independently rated the same images using standardized criteria. While we didn’t use traditional replicates since each food image is unique, we assessed robustness through multiple portion sizes per food type.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Results

In this section, we compare DietAI24’s performance with baseline models in estimating energy, food weight, and several macronutrients, including fats, carbohydrates, and protein. Additionally, we provide a detailed breakdown of overall performance by assessing the accuracy of food code inference and portion size estimation. Detailed SR results for these components are presented separately by dataset. Finally, we emphasize our advantage in analyzing up to 65 nutrients and food components, compared to the limited scope of the original GPT Vision model.

Nutrient content estimation

Table 1 compares nutrition estimation accuracy on the ASA24 dataset. To assess statistical significance, we used paired t-tests to compare the MAE for each nutrient between DietAI24 and each baseline model across all food images. DietAI24 achieved lower errors than all baselines (p < 0.05 for most nutrients). The only exception is the comparison between DietAI24 and GPT Vision model for energy estimation per unit, where the difference is not significant on the Nutrition5k dataset (p ≈ 0.13). Our system consistently outperforms the competition in terms of food weight, energy, fats, carbohydrates, and protein. For instance, our MAE per dish for energy is considerably lower at 47.7 kcal, compared to 277 kcal for Calorie MAMA, 199 kcal for ViT, 169 kcal for SnapCalorie, and 168 kcal for Foodvisor. This superior accuracy is also evident in other nutrients. Our MAE per dish for fats is only 1.8 grams, which is substantially better than the 12 grams, 9.71 grams, and 8.65 grams reported by ViT, Calorie MAMA, and Foodvisor, respectively. Additionally, in terms of normalized error (MAE per unit), our system demonstrates enhanced precision, with a value of 0.196 for energy compared to 1.19, 0.84, and 0.719 for the other systems. This high level of accuracy, along with consistently low error rates across both per-dish and per-unit measurements, highlights the effectiveness and reliability of DietAI24, as also evidenced in the Nutrition5k dataset shown in Table 2.

Table 1.

Comparison of MAE across food weight and four nutrients on the ASA24 dataset

	ViT		Calorie MAMA		Foodvisor		SnapCalorie		DietAI24
Target	Per dish	Per unit	Per dish	Per unit	Per dish	Per unit	Per dish	Per unit	Per dish	Per unit
Energy (kcal)	199	0.84	277	1.19	168	0.719	169	0.728	47.7	0.196
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Fat (g)	12	1.17	9.71	0.993	8.65	0.867	8.34	0.854	1.8	0.174
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Carbohydrate (g)	23.9	0.741	31.9	0.992	24.1	0.752	21	0.653	6.95	0.208
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Protein (g)	11.8	2.07	5.5	0.989	4.13	0.747	4.36	0.784	1.38	0.233
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Food weight (g)	110	0.833	85.5	0.61	82.6	0.591	–	–	49.4	0.348
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)

MAE per dish reflects the error per dish, while MAE per unit normalizes the error by the average nutrient amount. The best performance is highlighted in bold. The evaluation includes 3000 unique dishes from the ASA24 dataset, sampled and preprocessed as described in the “Data processing” section. Values in parentheses indicate p values from paired t-test comparing to DietAI24.

MAE mean absolute error, ViT vision transformer.

Table 2.

Comparison of MAE across food weight and four nutrients on the Nutrition5k dataset

	Calorie MAMA		Foodvisor		GPT Vision		DietAI24
Metric	Per dish	Per unit	Per dish	Per unit	Per dish	Per unit	Per dish	Per unit
Energy (kcal)	966	3.07	168.5	0.536	81.8	0.244	68.2	0.197
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	( ≈ 0.13)
Fat (g)	14.5	0.982	8.24	0.559	5.49	0.382	4.01	0.226
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Carbohydrate (g)	30.4	0.986	21.9	0.711	9.31	0.44	5.98	0.212
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Protein (g)	16.5	0.983	12.9	0.766	5.27	0.426	3.88	0.188
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)
Food weight (g)	270	0.942	150.1	0.526	89.1	0.274	45.1	0.156
	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)	(p < 0.05)

MAE per dish represents the error per dish, while MAE per unit normalizes the error by the average nutrient amount. The best performance is highlighted in bold. The evaluation includes 1000 unique dishes from the Nutrition5k dataset, sampled and preprocessed as detailed in the “Data processing” section. Values in parentheses indicate p values from paired t-test comparing to DietAI24.

MAE mean absolute error.

As shown in Fig. 2, expert dietitians rated DietAI24’s accuracy from 2.63 to 3.98 on a 4-point Likert scale. On average, DietAI24 achieves Close Match or better performance for most nutritional components. The system showed exceptional accuracy in detecting caffeine and alcohol content, achieving the highest mean ratings (3.98 ± 0.16 and 3.97 ± 0.26, respectively) with Exact Match ratings in over 96% of cases. This high performance likely stems from the straightforward categorization of beverages and foods containing these substances, making them easier for dietitians to identify and verify. The system showed different levels of accuracy for other nutritional components. Fiber analysis achieved Exact Match ratings in 29.74% of cases (3.05 ± 0.81), while sugar analysis reached Exact Match ratings in 24.62% of cases (2.78 ± 0.92). In food item identification tasks, DietAI24 achieved Exact Match ratings in 34.87% of cases (3.30 ± 0.66). Protein assessment received the lowest mean rating (2.63 ± 0.93), reflecting the inherent difficulty in visual estimation. The high variations in expert assessments, such as 0.81 in fiber, 0.92 for sugar and 0.93 for protein, indicate considerable variability in expert assessments, highlighting that experienced dietitians struggle with consistent estimation of these nutrients and the complexity of visual assessment for these nutritional elements.

Fig. 2. Distribution of expert dietitians’ ratings: mean ± standard deviation across nutritional components on 4-point Likert scale.

The data underlying these distributions are available in Supplementary Data 1.

Food recognition

Table 3 evaluates the performance of GPT-4 and GPT-4 Turbo in recognizing FNDDS food codes from images of varying portion sizes, focusing on images where food was successfully detected. Accuracy is categorized as Exact Match, Close Match, Far Match, and Mismatch. Despite a considerable drop in the number of images processed by GPT-4 with decreasing portion sizes (as reflected by the SR shown in Supplementary Fig. 6), GPT-4 can maintain relatively stable accuracy in food code inference for those images where it can detect food presence. For example, GPT-4 shows an increasing trend in Exact Match rates from the largest (27.8%) to the smallest portion sizes (32.7%), with Close Match rates remaining high across all sizes, peaking at 86.5% for the smallest portions. The highest Far Match rate for GPT-4 is 96.2% for the smallest portions, with a low Mismatch rate of 3.8%. In contrast, GPT-4 Turbo exhibits fluctuating results, with its highest Exact Match rate of 29.1% for the largest portions decreasing to 22.1% for the smallest. Close Match rates for GPT-4 Turbo are generally lower, particularly for smaller sizes (59.3%), indicating challenges in accurately recognizing smaller portions. The Far Match rate decreases with reduced portion size, and Mismatch rates substantially increase to 28.1% for the smallest portions. Although GPT-4 Turbo sometimes fails to match the correct food code perfectly, it maintains a relatively stable SR across portion sizes as shown in Supplementary Fig. 6. These results highlight the challenges posed by smaller portion sizes in food recognition. Figure 3 provides a visualization of ASA24 food image samples with correctly recognized food codes and portion sizes. DietAI24 not only identifies the correct food category but also captures specific details such as flavor (e.g., Yogurt, fruit variety), preparation method (e.g., Whole potato, home fries), and even specific brands (e.g., M&M Chocolate Candies). Meanwhile, Fig. 4 shows Nutrition5k food image samples with inferred food codes and portion sizes.

Table 3.

Food and Nutrient Database for Dietary Studies (FNDDS) food code recognition accuracy on ASA24 dataset: GPT-4 vs. GPT-4 turbo across different portion sizes

	Portion size	Exact match (%)	Close match (%)	Far match (%)	Mismatch (%)
GPT-4	Largest	27.8	74.7	82.1	17.9
	Medium	28.8	82.2	87.7	12.3
	Smallest	32.7	86.5	96.2	3.8
GPT-4 Turbo	Largest	29.1	77.3	84.5	14.5
	Medium	26.3	68.3	78.4	21.6
	Smallest	22.1	59.3	71.9	28.1

Fig. 3. Example outputs from DietAI24 using the ASA24 dataset.

Each panel shows the input food image, the model’s predicted food code(s) and portion sizes, as well as the estimated nutrient content. a Mexican sweet roll (b) Deep-fried pork rinds (c) Mixed nuts with dried fruit (d) Plain chocolate candies (e) Croissant (f) Home-fried potatoes (g) Meat lasagna (h) Fruit yogurt.

Fig. 4. Example outputs from DietAI24 using the Nutrition5k dataset.

Each panel shows the input food image, the model’s predicted food code(s) and portion sizes, as well as the estimated nutrient content, illustrating the model’s application to real-world mixed dishes. a Almonds and roasted potato (b) Egg omelet with green beans and pork bacon (c) Grilled chicken breast with cauliflower, tomatoes, peppers, and onions (d) Grilled salmon with glazed carrots and roasted potato.

Table 4 illustrates the accuracy of food code inference for various food sub-groups, a crucial aspect for dietary studies, using the food image with the largest portion from the ASA24 dataset as an example to illustrate this accuracy. The algorithm shows commendable accuracy in several categories, such as Meat, Poultry, Fish, and Mixtures (79.4% accuracy), Dry Beans, Peas, Other Legumes, Nuts, and Seeds (78.5%), Grain Products (83.3%), Fruits (78.7%), Vegetables (77.9%), and Sugars, Sweets, and Beverages (73.7%). These high accuracies are vital for ensuring reliable dietary analysis, facilitating a better understanding of food intake patterns and their health impacts. However, the algorithm’s perfect score in identifying eggs (100%) is based on a single sample, casting doubt on its generalizability. Moreover, lower accuracy in categories like Milk and Milk Products (64.5%) and Fats, Oils, and Salad Dressings (0%), particularly those involving liquids, suggests challenges in distinguishing between similar visual textures.

Table 4.

Performance of DietAI24 in recognizing Food and Nutrient Database for Dietary Studies (FNDDS) food codes across different food sub-groups

Food sub-group	Close match accuracy (%)	N
Milk and Milk Products	64.5	24
Meat, Poultry, Fish, and Mixtures	79.4	44
Eggs	100	1
Dry Beans, Peas, Other Legumes, Nuts, and Seeds	78.5	17
Grain Products	83.3	101
Fruits	78.7	33
Vegetables	77.9	52
Fats, Oils, and Salad Dressings	0.00	5
Sugars, Sweets, and Beverages	73.7	38

Portion size estimation

Portion size is another crucial factor in accurately estimating nutrient content. To evaluate GPT’s capability in inferring portion sizes from images, we analyzed its performance on the ASA24 dataset, which includes ground truth portion size codes. As shown in Table 5, we used Classification Accuracy and Prediction Error metrics to assess its effectiveness across different portion sizes. Classification Accuracy, which measures the ability to correctly identify portion size from a list of commonly used portion sizes in the FNDDS database, was highest for the smallest portions (86.7%) and medium portions (85%), with the largest portions showing lower accuracy at 68.4%. Prediction Error, reflecting the discrepancy between inferred and actual portion sizes, was more pronounced in the largest portions at 0.72 units, decreasing markedly in smaller portions to 0.31 and 0.16 units. These results demonstrate GPT’s effectiveness in accurately classifying and predicting smaller portion sizes while highlighting challenges with larger portions.

Table 5.

Performance of DietAI24 in food portion inference

Portion size	Largest	Medium	Smallest	Total
Classification (%)	68.4	85	86.7	74.3
Prediction	0.72	0.31	0.16	0.55

Demonstration of identifying 65 nutrient and food component types

Table 6 presents the capabilities of DietAI24 in estimating over 60 nutrient and food component types from food images, comparing it with the original GPT Vision model. Results have been grouped into macronutrients, energy, vitamins, minerals, and other substances. The table reports the SR of the original GPT Vision model in nutrient content estimation, and the absolute difference (Abs Diff.) of the estimated nutrients between the original GPT Vision model and DietAI24. While the GPT Vision model achieves reasonable SRs for common components such as energy, protein, carbohydrates, fiber, and alcohol, its performance declines substantially for more complex components like fatty acids, certain vitamins, and minerals. This drop indicates limitations in the original GPT Vision model’s internal knowledge or access to detailed nutritional information. Moreover, the high Abs Diff. in nutrients like folate (25.67) and phosphorus (1.55) suggests inaccuracies in the original GPT Vision model’s estimations, which might indicate that the original GPT Vision model has learned some invalid knowledge or experienced hallucination. DietAI24’s accuracy has been validated on the ASA24 and Nutrition5k dataset, with both the SR and MAE calculated based on the same inferred food code and portion size as in Tables 1 and 2. Given the essential role of micronutrients in human health, DietAI24 establishes a benchmark in automated dietary analysis.

Table 6.

Comparative analysis of nutrient estimation accuracy between DietAI24 and the original GPT Vision model across various nutrients on Nutrition5k

Category	Nutrient name	Unit	GPT Vision SR (%)	Abs Diff.
Macronutrients	Energy	kcal	79	0.37
	Protein	g	61	0.43
	Carbohydrate	g	62	0.8
	Fat	g	26	0.45
	Water (Moisture)	g	0	–
Sugars & Fiber	Sugars	g	30	1.09
	Fiber	g	81	0.98
Fatty Acids	Saturated Fatty Acids	g	18	0.72
	Monounsaturated Fatty Acids	g	0	–
	Polyunsaturated Fatty Acids	g	6	0.46
	Cholesterol	mg	56	0.5
	Individual Fatty Acids (4:0, etc.)	g	0	–
Vitamins	Vitamin A	μg	8	0.57
	Vitamin C	mg	26	1.34
	Vitamin D	μg	4	0.5
	Vitamin E	mg	4	0.24
	Vitamin K	μg	0	–
	Vitamin B-6	mg	2	0.85
	Vitamin B-12	μg	16	0.29
	Carotenoids	μg	0	–
	Thiamin	mg	4	0.65
	Riboflavin	mg	0	–
	Niacin	mg	0	–
	Folate	μg	8	25.67
	Choline	mg	2	0.32
Minerals	Calcium	mg	12	0.57
	Iron	mg	4	0.51
	Magnesium	mg	2	0.41
	Phosphorus	mg	8	1.55
	Potassium	mg	18	0.55
	Sodium	mg	18	0.85
	Zinc	mg	4	0.42
	Copper	mg	4	0.64
	Selenium	μg	4	0.73
Others	Caffeine	mg	20	0
	Alcohol	g	45	0
	Theobromine	mg	0	–

SR success rate, Abs Diff. absolute difference.

Usability evaluation

In this section, we present the detailed results of DietAI24’s usability evaluation using the SR metric. For food code inference, SR represents the percentage of images where the model detects the presence of food, regardless of naming accuracy. This metric assesses the model’s ability to recognize content across various visual complexities, demonstrating the robustness of our system powered by the latest GPT-4 Turbo Vision model. We also compare it to the older GPT-4 Vision model to evaluate the impact of advancements in MLLMs on DietAI24. For portion size estimation, SR measures the feasibility of MLLMs to provide the portion size of a specific food item.

For the ASA24 dataset, Supplementary Fig. 6 compares the two models’ ability to recognize food items in images. Supplementary Fig. 6a shows their SR, while Supplementary Fig. 6b explains the difficulties they encountered. Considerable differences in SR are observed across three portion sizes. For the largest portion size, GPT-4 achieves an 88% success rate, while GPT-4 Turbo reaches 93%. With medium portions, GPT-4’s SR drops to 51%, whereas GPT-4 Turbo maintains 89%. For the smallest portions, GPT-4’s rate falls sharply to 37%, but GPT-4 Turbo remains resilient at 84%. These results show that GPT-4 Turbo consistently outperforms GPT-4, demonstrating superior robustness and reliability across all portion sizes. Failures fall into two categories: Low Image Quality and No Identified Food. Low Image Quality refers to instances where poor resolution or clarity impedes recognition, while No Identified Food includes cases where the models failed to detect food, often misidentifying non-food elements. GPT-4 struggles more with smaller portion sizes, with a considerable increase in No Identified Food errors, reaching 52% for the smallest portions. In contrast, GPT-4 Turbo performs better, with only 11.9% No Identified Food errors for the smallest portions. This suggests that GPT-4 Turbo is more proficient at handling smaller portion sizes, enhancing its effectiveness in identifying food items under various conditions. For Low Image Quality errors, we attribute the primary cause to low original resolutions.

For the Nutrition5k dataset, Supplementary Table 2 presents the SR for food code inference and portion size estimation with the latest GPT-4 Turbo Vision model. DietAI24 recognizes food from images with a very high SR of 98.9%. For portion size estimation, the algorithm accurately identifies the weights of all detected food items in 84.2% of dishes (per dish) and achieves an SR of 80.5% for identifying the weights across all individual food items (per food item). These results demonstrate the high applicability of DietAI24 for mixed dish images.

Cost efficiency

The cost per image for nutrient content estimation was determined by dividing the total API expenditure by the number of evaluated images for each system. For commercial nutrition tracking platforms-Calorie MAMA, Foodvisor, and SnapCalorie—the per-image cost is fixed according to each provider’s published API pricing. The cost for DietAI24 is calculated based on the actual OpenAI API charges, representing the average expenditure observed across all analyzed images. This value may fluctuate slightly depending on input size and prompt length. As shown in Supplementary Table 3, DietAI24 achieves competitive pricing compared to commercial nutrition tracking platforms.

Discussion

We present DietAI24, a framework integrating MLLMs with RAG technology and the FNDDS database for comprehensive nutrition estimation from food images. DietAI24 differs from consumer-focused applications by providing research-quality dietary evaluation capabilities. The findings have implications that extend beyond accuracy improvements.

DietAI24 offers important technical advantages over existing commercial platforms such as Calorie Mama, SnapCalorie, and Foodvisor. DietAI24 is designed to accurately recognize individual components within complex mixed dishes. Additionally, DietAI24 provides precise portion size estimation across the complete range of serving sizes defined by the FNDDS database, enabling accurate scaling of nutritional calculations. The integration with the FNDDS database enables DietAI24 to analyze 65 different nutrients and food components rather than the basic macronutrients typically provided by these commercial platforms. Unlike these commercial platforms that require time-consuming and resource-intensive retraining to accommodate new foods or updated nutritional information, DietAI24 can be easily adapted by simply updating the underlying nutrition database. Since FNDDS is updated every 2 years, DietAI24 can efficiently remain current with changes in the food supply and dietary trends. Moreover, these commercial applications employ proprietary, non-transparent algorithms that limit their suitability for scientific research applications.

Building on these technical capabilities, DietAI24 demonstrates substantial implications as a digital health solution in real-world nutrition practice. First, DietAI24 offers a powerful alternative to traditional 24HR methods, reducing the time and cost involved in nutrition research and minimizing participant burden. By enabling rapid and accurate dietary data collection through automated food image analysis, DietAI24 enables large-scale epidemiological studies and enhances the feasibility of repeated dietary assessments over extended periods. Second, DietAI24’s capability for continuous monitoring of dietary intake in daily living contexts allows dietitians and healthcare providers to leverage collected data for delivering personalized dietary interventions. This ongoing assessment could be combined with tailored nutritional counseling to support precision nutrition initiatives to promote healthier eating behaviors and ultimately enhance health outcomes.

The enhanced accuracy delivered by DietAI24 has important implications for health research and clinical practice. Protein estimation demonstrates remarkable precision (e.g., MAE 1.38g vs. 4.13–11.8g for baseline methods on the ASA24 dataset), facilitating accurate evaluation of dietary protein sufficiency. Such precision proves essential for muscle health investigations, metabolic function research, and clinical nutrition interventions. For example, this accuracy level supports sarcopenia studies, where protein requirements for elderly populations demand careful monitoring, alongside athletic nutrition research focused on optimizing protein intake distribution and timing³⁷. Similarly, our strong accuracy in detecting caffeine and alcohol content (expert ratings of 3.98 ± 0.16 and 3.97 ± 0.26 respectively) opens opportunities for behavioral health research. Precise monitoring of both caffeine and alcohol is crucial for sleep research, as caffeine consumption affects sleep quality hours after intake, while alcohol similarly disrupts sleep patterns in many individuals³⁸. Additionally, accurate alcohol detection strengthens public health surveillance and consumption pattern analysis³⁹.

Beyond dietary assessment, the core abstraction of DietAI24—translating image content to descriptive text for querying domain-specific databases through RAG—has potential applications in other scientific domains. This framework could be adapted for medical image analysis, materials science characterization, or environmental monitoring, where visual information needs to be systematically connected with specialized knowledge bases. The success of DietAI24 in dietary assessment suggests its potential value in these broader contexts where expert domain knowledge must be applied to visual analysis tasks.

Despite these advantages, several inherent limitations affect our approach. From a behavioral perspective, while DietAI24’s prospective approach reduces recall bias, it may introduce the Hawthorne effect, where awareness of being monitored alters behavior⁴⁰. Real-time food photography may influence eating patterns, as users aware of monitoring might underreport unhealthy foods, shift toward socially desirable choices, or forget to photograph meals during social occasions, leading to selective reporting. From a technical perspective, nutrient estimation accuracy depends on successful food recognition and portion size assessment, with errors propagating through the pipeline. MLLMs struggle with poor image quality (blur, lighting, small portions), while RAG-based systems face challenges with mixed dishes and may retrieve similar but incorrect food codes when composite foods lack database representation. To address these limitations, we propose several strategies. For behavioral challenges, future implementations could incorporate random meal sampling, batch photo uploads, and validation methods to detect reporting patterns. For technical challenges, we plan to implement super-resolution techniques, edge enhancement filters, and adaptive exposure normalization to improve image quality. Real-time camera guidance could help users capture better images from multiple angles and at different meal stages (before and after eating), enhancing both portion size estimation and overall system performance in real-world settings.

Our evaluation also faces specific constraints. We tested only OpenAI’s GPT models without comparing alternative MLLMs, and while GPT provides an effective development platform, its closed-source nature creates vulnerabilities, including variable inference costs, potential API discontinuation, and unexpected behavioral changes. However, DietAI24’s modular architecture enables straightforward migration to alternative MLLMs as they emerge. Additionally, our evaluation datasets present limitations-ASA24’s controlled photography may overestimate real-world performance, while Nutrition5k, though more realistic, primarily represents U.S. cafeteria foods. Both lack representation of international cuisines and foods not in the FNDDS database.

The transition from prototype to real-world deployment introduces further challenges. As users share images of their meals through DietAI24, they inadvertently disclose personal health or other information embedded in their dietary patterns. Such data requires robust protection measures. Generative AI processing personal information face unique privacy challenges and must address critical ethical and privacy challenges, such as data consent and potential biases^41,42. For DietAI24 specifically, this means implementing end-to-end encryption for food image transmission, restricting nutritional data storage to local devices when possible, and providing clear user controls for dietary data usage. During deployment, collected data can be leveraged to fine-tune MLLMs, enabling personalization to individual preferences while mitigating inherent biases.

In future work, we plan to evaluate and fine-tune the model on a more diverse set of food images, including those captured on various types of plates and in different background settings. Incorporating these diverse visual conditions is expected to strengthen the system’s robustness and ensure more reliable performance in practical, real-world settings. While the current implementation utilizes the FNDDS database focused on American dietary habits, we plan to extend DietAI24 by integrating additional regional nutrition databases like BLS⁴³, NEVO⁴⁴, CIQUAL⁴⁵, and McCance and Widdowson⁴⁶. This expansion would address equity considerations through improved cultural food diversity representation, ensuring fair performance across demographic groups—a critical factor for transitioning from research prototype to widely-adopted application.

In conclusion, DietAI24 demonstrates that integrating MLLMs with RAG and authoritative nutrition databases can substantially improve dietary assessment accuracy while reducing participant burden. The system’s modular design and competitive pricing further support its suitability for both research and real-world applications. Ongoing improvements in database integration, image quality handling, and privacy safeguards will continue to enhance its utility for diverse populations and use cases.

Author contributions

Conceptualization: R.Y., H.L., T.J.H., and X.H.; Methodology: R.Y. and H.L.; Data curation: R.Y. and H.L.; Validation: R.Y., J.L., D.L., H.P., and J.M.; Writing—original draft: R.Y. and J.L.; Visualization: R.Y. and D.L.; Writing—review & editing: R.Y., H.L., J.L., D.L., H.P., M.P.D., J.M., Y.Q., C.Y., T.J.H., and X.H.

Peer review

Peer review information

Communications Medicine thanks Vesna Knights, Sakib Shahriar and Kurt Gedrich for their contribution to the peer review of this work. [A peer review file is available].

Data availability

The ASA24 dataset is available through the National Cancer Institute (NCI) (https://epi.grants.cancer.gov/asa24/)⁵. The Nutrition5k dataset is publicly available at https://github.com/google-research-datasets/Nutrition5k¹². The FNDDS database (version 2019-2020) is accessible from the United States Department of Agriculture (USDA) at https://www.ars.usda.gov/northeast-area/beltsville-md-bhnrc/beltsville-human-nutrition-research-center/food-surveys-research-group/docs/fndds/¹². The National Health and Nutrition Examination Survey (NHANES) consumption frequency data is available through the Centers for Disease Control and Prevention (CDC) at https://www.cdc.gov/nchs/nhanes/⁴.

Code availability

All code for DietAI24 is available at https://github.com/Runz96/DietAI and archived at 10.5281/zenodo.17070373⁴⁷.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s43856-025-01159-0.