Potential of existing online 24-h dietary recall tools for national dietary surveys

Abstract

Objective:

To describe existing online, 24-h dietary recall (24-h DR) tools in terms of functionalities and ability to tackle challenges encountered during national dietary surveys, such as maximising response rates and collecting high-quality data from a representative sample of the population, while minimising the cost and response burden.

Design:

A search (from 2000 to 2019) was conducted in peer-reviewed and grey literature. For each tool, information on functionalities, validation and user usability studies, and potential adaptability for integration into a new context was collected.

Setting:

Not country-specific

Participants:

General population

Results:

Eighteen online 24-h DR tools were identified. Most were developed in Europe, for children ≥10 years old and/or for adults. Eight followed the five multiple-pass steps but used various methodologies and features. Almost all tools (except three) validated their nutrient intake estimates, but with high heterogeneity in methodologies. User usability was not always assessed, and rarely by applying real-time methods. For researchers, eight tools developed a web platform to manage the survey and five appeared to be easily adaptable to a new context.

Conclusions:

Among the eighteen online 24-h DR tools identified, the best candidates to be used in national dietary surveys should be those that were validated for their intake estimates, had confirmed user and researcher usability, and seemed sufficiently flexible to be adapted to new contexts. Regardless of the tool, adaptation to another context will still require time and funding, and this is probably the most challenging step.

National food consumption surveys are the main method used to monitor food consumption trends, nutritional status and exposure to hazardous substances in a population or to evaluate the impact of dietary policies. Ensuring the representativeness of the sample population and collecting accurate data are the biggest challenges⁽¹⁾. Since 2007, a decrease in response rates, defined as the ratio between the number of participants and all expected interviews (including unreachable and ineligible individuals), has been observed in many epidemiologic studies⁽²⁾, as reported in food consumption surveys in several European countries^(3,4), and the USA⁽⁵⁾. The reasons for refusal may include an increase in requests for study participation, declining trust in science, and increasingly complex research protocols^(2,3). As an example, in France, the previous 7-d self-administered paper food records methodology^(6,7) has shifted to interview-led 24-h dietary recalls (24-h DR) in the most recent cross-sectional Individual and National Study on Food Consumption 3 (INCA3) conducted in 2014–2015. The new protocol required four contacts to complete the dietary recalls after having agreed to take part, compared to two contacts in the INCA2 survey. This change may have had a negative impact on the response rate which decreased by about 20% points compared to the INCA2 study. This led to an increase in the duration of fieldwork and in costs to ensure representative population sample⁽⁸⁾. There is a need to shift towards more user-friendly tools and to adapt surveys to the population’s current lifestyle (e.g. longer working hours⁽²⁾), while maintaining high data quality at an acceptable cost.

A wide range of technological options for dietary assessments are available⁽⁹⁾. They can be categorised as computer-based (offline or online), mobile-based or image-based tools. Offline computer-based tools have already been used in several national surveys^(10–14) and have shown some limitations, in particular for data management^(8,15–17). For instance, adapting GloboDiet software to European national surveys, as well as checking and cleaning the collected data according to the FoodEx2 classification, was very time-consuming and costly^(16–18). Other technologies such as online computer-based, mobile-based or image-based tools have rarely been used in national dietary surveys, probably because of doubts about their acceptability within the population, or a lack of evidence about their validity and costs to collect data that are both nationally representative and accurate⁽¹⁵⁾.

Regarding mobile-based tools collecting dietary intakes, most were developed for commercial purposes^(9,19,20), often with the aim of helping individuals to manage their weight^(9,19,21). These tools may lack validity and transparency^(19,22), and they require that a large proportion of the population has a smartphone. A mobile-based solution not fully online, called INDDEX24, has been designed for low- and middle-income countries^(23,24) to fill the lack of tools meeting specific constraints in those countries (low smartphone penetration, low literacy, lack of connectivity, etc.)^(22,25). The tool includes a tablet and mobile application available online and offline, as well as a web platform for data management. This tool is currently in the process of being validated and represents potential for specific national dietary surveys. Barcode scanning applications usually used on mobile might be valuable for dietary assessments, but current tools are not reliable for use in national surveys without an extensive development phase and validation studies⁽²⁶⁾. As mobile-based tools, various technologies of image-based tools are available but all require further development to be validated on a wide range of food products and on a large sample size of individuals^(27–29).

Online computer-based tools (mainly using 24-h DR) appear to be the most mature technology to be adapted to national food consumption surveys without requiring long and costly development steps. Importantly, some of them have already been used in large-scale epidemiological studies^(30–34), and they were designed to be easily adaptable to other populations^(35–37). They can be adapted to smartphones, and many have been validated among children and/or adults^(22,38). To our knowledge, only one review focuses on web- and computer-based 24-h DR⁽³⁸⁾. In the Timon et al. review⁽³⁸⁾, common design features and the methods used to assess the ability of 24-h DR tools to accurately assess nutritional and dietary intakes have been fully detailed, but no information about user and researcher usability were reported⁽³⁸⁾.

To tackle the challenges encountered by national dietary surveys, such as maximising the response rate and collecting data from a representative sample of the population of interest while optimising the ratio between cost and data quality, existing online 24-h DR seem to have potential for the collection of good quality data while being less burdensome for the respondent and investigator. The aim of this study was to describe existing online 24-h DR tools in diverse aspects, such as functionalities, validation of nutrient estimations, user and researcher usability, and potential adaptability for integration into national dietary surveys.

Methods

Terminology

Here, validity means the extent to which a tool measures what it is intended to measure. The validity of dietary instruments is generally assessed by comparing nutrients and/or food intake estimates with another method considered the gold standard, which can be subjective (24-h DR, food diary, FFQ, etc.) or objective (biomarkers, observational studies, etc.)^(39,40). According to the ISO 9241-11:2018 Standard⁽⁴¹⁾, user usability is a measure of how well a user can learn and correctly use the tool’s functions, the ease of use, and user satisfaction in terms of whether a user can achieve his or her goals when using the tool. User usability is assessed using retrospective methods such as questionnaires, administered after the experience of the tool and/or real-time methodologies such as concurrent think-aloud protocols⁽⁴²⁾. In this paper, the term flexibility means the extent to which a tool can be easily modified and adapted to be used in a context other than the one for which the tool was developed. To simplify the manuscript, the term food is used instead of ‘food and beverages’ to describe the identification of all foods and beverages declared as consumed by the respondent.

Search strategy

Online computer-based self-administered 24-hD R tools were identified from reviews identified using a first search on Pubmed with the following terms, alone or in combination in the title or abstract: ‘survey’, ‘tool’, ‘instrument’, ‘assessment’, ‘questionnaire’, ‘measurement’, ‘diet’, ‘dietary’, ‘nutrient’, ‘food’, ‘intake’, ‘dietary pattern’, ‘dietary assessment’, ‘consumption’, ‘web’, ‘online’, ‘remote’, ‘digital’, ‘software’, ‘application’, ‘technology’, ‘ehealth’, and ‘review’, ‘meta-anal*’, and ‘systematic’. For the present paper, only two reviews including an evaluation and description of 24-h DR tools were retained (Timon et al. ⁽³⁸⁾ and Bell et al. ⁽²²⁾). Keywords were also used to identify relevant grey literature in Google, such as Timmins et al. ⁽⁴³⁾ and Coates et al. ⁽²⁵⁾, leading to the identification of two reports. From these four reviews or reports, focusing on tools published between 2000 and 2016, the authors identified a list of 24-h DR tools. An additional search with the same keywords (except ‘review’, ‘meta-anal*’, and ‘systematic’) was conducted to update the list and identify other tools published after the reviews or reports (published between 2017 and 2019) on PubMed and on Google in order to identify commercial tools without scientific publications.

Description criteria

For each tool, general characteristics, dietary intake collection methodology, as well as validation methodology and user usability were assessed based on the scientific literature and/or published reports. Functionalities and the method used to collect dietary intakes were described according to the United States Department of Agriculture (USDA) five-step multiple-pass 24-h DR method, a standardised and structured interview to record dietary intakes, during which several cues are used to help the respondent to remember and detail as accurately as possible of all foods consumed⁽⁴⁴⁾. Additionally, information on the tools’ flexibility to be adapted to another context was collected. All criteria chosen to describe the tool are reported in Fig. 1.

Fig. 1.

Criteria used to describe the tools. 24-h DR, 24-h dietary recall

*‘Eating occasion’ step is the collection of time, name and place of consumption of each food reported.

†‘Quick list’ step is the identification of all foods that the respondent consumed during the previous day.

‡‘Forgotten food list’ step provides cues about the consumption of often forgotten foods.

§ ‘Detail cycle’ step is the collection of detailed information on each food such as the fat content, brand name, preservation method and the consumed amount.

|| ‘Review and validation’ step is the final review of the 24-h DR.

Once tables were considered to be as complete as possible, based on available published papers or reports, phone or online video unstructured interviews were conducted with the corresponding authors of the studies, or the owner or developer of each tool in October 2019. The aim of the interviews was to check the already collected information, to validate specific points or to add information that could not be found in the literature. All collected information on validation and user usability studies as well as functionalities to collect dietary intakes were from published papers, whereas certain general characteristics (in particular available languages, last version and type of medium), and all information on flexibility were directly collected from the tool owner or developer.

A letter was assigned to each tool and used in the tables and text to refer to it when necessary.

Results

General description

The identification of online 24-h DR tools cited in the reviews and reports led to the selection of thirteen tools as follows (with the corresponding letter) (Fig. 2): Automated Self-Administered 24-h DR (A, ASA24)⁽⁴⁵⁾, Children’s and Adolescents’ Nutrition Assessment and Advice on the Web (B, CANAA-W) (previously Young Adolescents’ Nutrition Assessment on Computer, YANA-C)^(46,47), Computer-Assisted Personal Interview System (C, CAPIS)⁽⁴⁸⁾, Compl-Eat (D)⁽⁴⁹⁾, DietAdvice (E)⁽⁵⁰⁾, DietDay (F)⁽⁵¹⁾, Web-based Food Behaviour Questionnaire (G, FBQ)⁽⁵²⁾, Food Record Checklist (H, FoRC)⁽⁵³⁾, INTAKE24 (I, previously Self-Completed Recall and Analysis of Nutrition, SCRAN24)⁽⁵⁴⁾, Measure Your Food On One Day (J, myfood24)⁽⁵⁵⁾, NutriNet-Santé (K)⁽³¹⁾, Portuguese self-administered computerised 24-h DR (L, PAC24)⁽⁵⁶⁾ and Web-Survey of Physical Activity and Nutrition (M, Web-SPAN)⁽⁵⁷⁾. Five other online 24-h DR, published between 2016 and 2019, were added (Fig. 2): ClinShare (N), Creme Diet (O, published under the name foodbook24)⁽⁵⁸⁾, Web-based 24-h DR (P, R24W)⁽⁵⁹⁾, RiksmatenFlex (Q)⁽⁶⁰⁾ and Self-Administered Children, Adolescents, and Adult Nutrition Assessment (R, SACANA)⁽⁶¹⁾. In all, eighteen online 24-h DR tools were selected for this study (Fig. 2).

Fig. 2.

Flow chart for the selection of the online 24-h DR tools. 24-h DR, 24-h dietary recall

* The two reviews were the followings (38 and 43).

† The two reports were the followings (44 and 25).

A general description of the eighteen identified tools is available in Table 1. Among them, eleven (B⁽⁴⁶⁾, D⁽⁴⁹⁾, H⁽⁵³⁾, I⁽⁵⁴⁾, J⁽⁵⁵⁾, K⁽³¹⁾, L⁽⁵⁶⁾, N, O⁽⁵⁸⁾, Q⁽⁶⁰⁾ and R⁽⁶¹⁾) were developed in Europe, five (A⁽⁴⁵⁾, F⁽⁵¹⁾, G⁽⁵²⁾, M⁽⁷⁰⁾ and P⁽⁵⁹⁾) in North America (USA and Canada), one (E^(50,64)) in Australia and one (C⁽⁴⁸⁾) in South Korea. Five (A⁽⁷¹⁾, I⁽⁷²⁾, J⁽³⁶⁾, K⁽⁷³⁾ and R⁽⁶¹⁾) have already been adapted to be used in another country, and in particular, two (I⁽⁷²⁾ and J⁽⁷⁴⁾) have already been adapted for low-income countries (Middle East countries, Peru or the South-Asia region). Only one language is available in twelve tools (C–H, K, L–O and Q) (among them six in English: E–H, M and O), while the other six tools (C, D, K, L, N and Q) are in various languages. Three tools (D, I and Q) have been adapted or are being adapted for all populations (including infants), while the others were developed for teenagers and/or adults. Eight tools (A, I–K and N–Q) can (or will) be used on computers, mobiles and tablets, thanks to an automatic adjustment of the web page to the tool’s size (i.e. responsive design). Except four tools (G, H, K andf M), all have an integrated food composition database, allowing for automatic assessment of individual food and nutrient intakes for the researcher. Eleven tools (A–C, F, G, I–K, M, O and R) have a functionality to provide the respondent with a summary of their dietary intakes and for some tools, dietary advice^(75–78). While four tools (E, I, L and R) collect food intake data only, some tools collect other information such as dietary supplements (A, D, F, J and O), the level of physical activity (via a questionnaire) (B, C, K, M, N and Q), anthropometry (B, C, K, M and N), sleeping habits (A) or other information on food habits (G, H, K, M, N, P and Q).

Table 1.

General description of the online 24-hD R tools^*

Letter	Name†	Ref‡	Country of development	Available languages§	Owner	Adaptation to other countries	Latest version	Target population	Type of medium	Type of foods||	Food composition data integrated in the tool	Other data collected	Automatic report for
													researcher	respondent
A	ASA24	(45,62,63)	USA	English; Spanish (US version) English (Australian version) English; French (Canadian verison)	National Institutes of Health, Bethesda, USA. National Cancer Institute, Rockville, USA.	Y (USA, Australia, Canada)	1. ASA24–2020 (US version) 2. ASA24–2018 (Canadian version) 3. ASA24–2016 (Australian version)	≥ 10 years of age	C; M; T	G + B	Y	DS; Addition of sleep questions in a future release	Y	Y
B	CANAA-W	(46)	Belgium	10 languages including: English; German; Spanish; French; Italian; Swedish; Greek;… with translator system	Department of Public Health Ghent University, Ghent, Belgium and Research Foundation Flanders, Brussels, Belgium	N	NA	Validated for children (≥ 3 years of age) and adolescents (11 and 12 years of age)	C	G + B	Y	PA and ANT	NA	Y
C	CAPIS	(48)	South Korea	Korean	Seoul National University, Seoul, South Korea	N	NA	≥ 18 years of age	C	G	Y	PA and ANT	Y	Y
D	Compl-eat TM	NA	The Netherlands	Dutch	Wageningen University and Research, Wageningen, the Netherlands	N	New version available in 2021	≥ 6 months	C; T	G	Y	DS	Y	N
E	Diet Advice	(50,64,65)	Australia	English	University of Wollongong, Wollongong, Australia	N	NA	≥ 18 years of age	C	G	Y	N	Y	N
F	DietDay	(51)	USA	English	University of California, Los Angeles, USA.	N	NA	≥ 18 years of age	C	G + B	Y	DS and SD	Y	Y
G	FBQ	NA	Canada	English	University of Waterloo, Waterloo, Canada	N	NA	Validated for children between 11 and 13 years of age	C	G	N	FH	NA	Y
H	FoRC	(53)	UK	English	University of Aberdeen, Aberdeen, UK	N	NA	Adults ≥ 18 years of age	C	G	N	FH and SD	NA	N
I	Intake24	(54,66)	UK	English; Danish; Portuguese; Arabic	UK Open Government¶	Y (UK, Portugal, Denmark, New Zealand, the United Arab Emirates, South Asia region (Sri Lanka, India, Bangladesh and Pakistan) and Australia in progress)	2019	Originally developed for ≥ 11 years of age Adaptation for ≥ 1·5 years of age (not published)	C; M (adaptation in progress); T	G + B	Y	N	Y	Y
J	myfood24	(36,55,67)	England	English; German; Danish; Norwegian; French; Arabic (in progress); Spanish (in progress)	Dietary Assessment Ltd, Leeds, England	Y (Germany, Denmark, Norway, France and Caribbean, in progress for Middle East countries and Peru)	2019	≥ 10 years of age	C; M (adaptation in progress); T	G + B	Y	DS	Y	Y
K	NutriNet-Santé	NA	France	French	Nutritional Epidemiology Research Team (Inserm 1153/Inra 1125/Cnam/Université de Paris — Paris 13), Paris, France	Y; Belgium + Partnership with research teams and public institutes to transpose the technology to other countries (e.g. Brazil; Mexico; Canada)	01/09/2020	≥ 15 years of age	C; T; M	G + B	N (but direct match with an ad hoc food composition table)	PA, ANT, FH and SD collected from the NutriNet platform	N	Y
L	PAC24	(56)	Portugal	Portuguese	Instituto de Medicina Preventiva & Saúde Pública, Universidade de Lisboa, Lisbon, Portugal	N	2015	Validated for children between 7 and 10 years of age	C	G	Y	N	Y	N
M	Web-SPAN (based on FBQ tool)	(57)	Canada	English	University of Alberta, Alberta, Canada	N	2004	Validated for children between 11 and 15 years of age	C	G + B	N	PA, ANT and FH	N	Y
N	ClinShare	NA	France	French	MyGoodLife, Paris, France	N	2020	NA	C; M; T	G	Y	PA, ANT and FH; P	Y	N
O	Creme Diet (published under the name Foodbook24)	(58,68,69)	Ireland	English	Creme Global, Dublin, Ireland	N	NA	≥ 18 years of age	C ; T; M	G + B	Y	DS	Y	Y
P	R24W	(59)	Canada	French; English	Laval University, Quebec City, Canada	N	2015	≥ 18 years of age	C; M; T	G	Y	SD and FH	Y	N
Q	RiksmatenFlex	(60)	Sweden	Swedish	Swedish Food Agency, Uppsala, Sweden	N	2019	12 ≤ Teenagers ≥ 18 years of age; Adaptation in progress for all populations	C; M; T	G	Y	The tool has an integrated questionnaire function PA, FH, SD and food safety questions	Y	N
R	SACANA	(61)	Belgium, Germany, Cyprus, Estonia, Hungary, Italy, Spain, Sweden and Poland	Dutch (Flemish); German; Estonian; Hungarian; Italian; Spanish; Swedish; Greek; English and Polish	IDEFICS/I.Family Consortia	Y (Belgium, Germany, Cyprus, Estonia, Hungary, Italy, Spain, Sweden and Poland)	2014	≥ 11 years of age	C; T	G + B	Y	N	Y	Y

ANT, anthropometric data; ASA24, Automated Self-Administered 24-h diet recall; B, Brand level; C, computer; CANAA-W, Children’s and Adolescents’ Nutrition Assessment and Advice on the Web; CAPIS, Computer-Assisted Personal Interview System; DS, dietary supplement; FBQ, Web-based Food Behaviour Questionnaire; FH, food habits; FoRC, Food Record Checklist; G, generic; M, mobile; myfood24, Measure Your Food On One Day; N, No; NA, missing information; PA, physical activity; PAC24, Portuguese self-administered computerised 24-h dietary recall; SD, socio-demographic data; R24W, Web-based 24-h dietary recall; Ref, References; SACANA, Self-Administered Children, Adolescents, and Adult Nutrition Assessment; T, tablet; Web-SPAN, Web-Survey of Physical Activity and Nutrition; Y, yes.

^*All information was validated by the tools’ owners or developers, except for the tools Creme Diet, CAPIS, CANAA-W, Diet Advice, DietDay, FoRC and FBQ.

^†The name is underlined when information was validated by the developer/owner of the tool.

^‡Publications of tool development.

^§In the most recent version of the tool.

^||In the version published.

^¶Initially developed by Newcastle University, Newcastle, UK, with funding from Food Standards Scotland, Adaptation by the University of Cambridge, Cambridge, UK.

Method of dietary intake collection

Table 2 describes the main functionalities of the tools to collect dietary intakes.

Table 2.

Step number and method of the multiple-pass methodology and main functionalities to collect dietary intakes

Letter	Name *	‘Eating Occasion’	‘Quick List’	Prompts for the quick list	‘Detail cycle’, precise identification of the food	‘Detail cycle’, additional food descriptor	Detail cycle’, portion size estimation	‘Forgotten Foods List’	‘Review and validation’	Creation of recipe	Other functionality to identify the food	FFQ
A	ASA24	Time of consumption; place of consumption; social context; presence or not of a TV screen; question on eating habits; place of purchases	Keywords search engine; hierarchical tree by food group	3	N	Prepared dish; place of purchase; several descriptors (fat content, fortification, etc.) according to the selected food	Food picture; standard unit	Y	Y	Y	Saving favourite foods; suggestions for commonly consumed foods	N
B	CANAA-W	Time	Food group consumption reporting among 25 food groups	N	Hierarchical tree by food group	N	Food picture: 4 types of portion presentations with 260 generic foods photographed (Source: Belgian manual on food portions and household measures); household measurement units	Y	Y	N	N	N
C	CAPIS	Time; place of consumption	Keywords search engine; hierarchical tree by food group	3	N	N	Food picture; standard unit; free entry of g weight	N	Y	N	Suggestions for commonly consumed foods	Y
D	Compl-eat TM	Time of consumption; preparation method	Checking the box for the group consumed	N	Keyword search engine; hierarchical tree by food group	N	Standard unit; household measure; work for adding food pictures; free entry of g weight	Y	Y	Y	Inclusion of a (free) note to detail the food; manual entry for missing foods in the integrated food list	N
E	Diet Advice	N	Food group reporting	3	Hierarchical tree by food group	N	Food picture	NA	NA	NA	N	N
F	DietDay	Time of consumption	Keywords search engine; hierarchical tree by food group	3	N	Prepared dish; place of purchase, flavoured; method of food preparation	Food picture; household measures	Y	Y	N	N	N
G	Web-based food behaviour questionnaire	Time of consumption	N	N	Dropdown list	N	Food picture; standard units	Y	Y	N	Suggestions for commonly consumed foods	Y
H	FoRC	Time of consumption	Food group consumption reporting among 16 food groups	N	Hierarchical tree by food group	N	Food picture	Y	NA	N	N	N
I	Intake24	Time of consumption	Free keywords	N	Keyword search engine; hierarchical tree by food group	Source of food (purchased or home-based)	Food picture; cursor (for drinks only); standard units; household measure; food waste (for certain foods only)	Y	Y	Y (if missing foods)	Manual entry for missing foods in the integrated food list	N
J	myfood24	Time of consumption	N	N	Keyword search engine; filter by food category; filter by brand	N	Food picture; standard unit; free entry of g weight	Y	Y	Y	Recently added foods; saving recipes created; suggestions for commonly consumed foods	N
K	NutriNet-Santé	Time of consumption; place of consumption; social context; presence or not of a TV screen; place of meal preparation	N	N	Keyword search engine; hierarchical tree by food group	Type of food (commercial, restaurant or home-made); brand; salt consumed by food. Identification of organic food consumed	Food picture; standard unit; free entry of g weight	Y	Y	N	Suggested sample meals (related to previous user’s recalls); manual entry for missing foods in the integrated food list	N
L	PAC24	Time of consumption; place of consumption; social context; presence or not of a TV screen; meal preparation	Keywords search engine	3	N	Type of preparation	Food picture; standard unit; household measure; free entry of g weight for food with no pictures	N	Y	N	N	N
M	Web-SPAN (based on FBQ tool)	Time of consumption; place of consumption; social consumption; question on eating habits. Meal preparation	N	N	Dropdown list	N	Food picture; standard units	Y	Y	N	Suggestions for commonly consumed foods	Y
N	ClinShare	Time of consumption; place of consumption; social context; presence or not of a screen	N	N	Keyword search engine; hierarchical tree by food group	N	Standard unit; free entry of g weight	Y	Y	Y	N	N
O	Creme Diet (published under the name Foodbook24)	Time; place of consumption	Keywords search engine; hierarchical tree by food group	3	N	Home-made food, low-fat or not	Food picture; standard unit	Y	Y	N	N	Y
P	R24W	Time of consumption; place of consumption; social context; presence or not of a screen	N	N	Keyword search engine; hierarchical tree by food group	Fat content, cooked or raw, canned or fresh, etc.	Food picture; standard unit; household measure	Y	Y	For sandwiches and salads only	N	Y
Q	RiksmatenFlex	Time of consumption; place of consumption; social context	N	N	Keyword search engine	Y type of certain foods (e.g. meat, bread, etc.)	Food picture; standard unit; household measure	Y	Y	N	Pictures of foods commonly consumed in five food categories (bread, ready-to-eat sandwiches, breakfast cereals, ice cream and fat spreads); manual entry for missing foods in the integrated food list	Y
R	SACANA	Time of consumption; place of consumption; social context; activity during the meal	Food groups displayed in images or search using keywords in a pre-code food list	N	Keyword search engine; hierarchical tree by food group	Can be entered by the participant and then integrated manually by the survey centre to the main menu for general/future use	Food picture; standard unit; household measure	Y	Y	Y	Manual entry for missing foods in the integrated food list	N

ASA24, Automated Self-Administered 24-h diet recall; CANAA-W, Children’s and Adolescents’ Nutrition Assessment and Advice on the Web; CAPIS, Computer-Assisted Personal Interview System; FBQ, Web-based Food Behaviour Questionnaire; FoRC, Food Record Checklist; myfood24, Measure Your Food On One Day; N, No; PAC24, Portuguese self-administered computerised 24-hour dietary recall; R24W, Web-based 24-h dietary recall; SACANA, Self-Administered Children, Adolescents, and Adult Nutrition Assessment; Web-SPAN, Web-Survey of Physical Activity and Nutrition; Y, yes.

^*The name is underlined when information was validated by the developer/owner of the tool.

Eight tools (A, B, D, F, H, I, O and R) display the same steps as the USDA multiple-pass method, but not necessarily in the same order and not necessarily using the same method to collect the ‘Quick list’ (e.g. identification in a pre-defined list of foods, using free keywords or food group checkboxes). Other tools either do not include the ‘Forgotten food list’ step (n 3; C, E, L) or do not include the ‘Quick list’ step (n 7; G, J, K, M, N, P, Q). Tools without a ‘Quick list’ ask the respondent to provide all information (identification, description and quantification of the food) in one step for each consumption occasion of the day. The time of consumption is always requested, and other information, including the place of consumption (n 10; A, C and K–R), place of meal preparation (n 1; K), social context (n 8; A, K–N and P–R) and presence of a screen (n 5; A, K, L, N and P) can be requested depending on the tool.

The whole list of foods from which the respondent selects the one consumed depends on the study and version of the tool and can contain either generic foods only (often from national food composition databases), or generic and specific brand products (Table 1). In order to ease food selection by the user, the selected tools use different food identification systems (either in the ‘Quick list’ or ‘Detail cycle’ steps):

• - using a keyword search engine (n 13; A, C, D, F, I–L and N–R),

• - by selecting within a hierarchical tree (n 13; A–F, H, I, K, N, O, P and R),

• - by selecting within a dropdown list (n 2; M and G),

• - by filtering foods (n 2; A and J) by category, brand, type of food (generic or brand) or from a list of favourite foods,

• - by selecting from pictures (n 1, for specific food groups; R).

Five tools (B⁽⁷⁶⁾, I⁽³⁴⁾, J⁽⁵⁵⁾, O⁽⁵⁸⁾ and Q⁽³⁰⁾) have improved their keyword search engine by including synonyms and different spelling options or brand names to help participants find the correct food or to allow the identification of foods by matching more than one search term (e.g. chocolate biscuits). Other functionalities helping the respondent to report the correct food consumed were identified, such as the creation of personal recipes (n 7; A, D, I, J, N, P and R), or reporting a new food (free text entry) not yet in the integrated food list (n 5; D, I, K, Q and R).

Portion size estimation is requested, either directly after having identified a food (n 7; G, J, K, M, N, P and Q) or in the second step after having identified all foods consumed during the day (n 11; A–F, H, I, L, O and R). Quantification can be entered directly in grams or volumes (n 6; C, D, J–L and N), or using portion size estimation aids such as food portion pictures (n 16; A–C, E–M and O–R), standard units of consumption (n 14; A, C, D, G and I–R) or household measures (n 8; B, D, F, I, L, P, Q and R). Only two tools do not use food pictures (D and N). To our knowledge, only one tool (I) also requests, for some foods, the amount of food that is left over. The type of packaging or way of consumption can also be asked to refine the picture to display (e.g. consumption of an entire fruit or in pieces, consumption of a soda in a bottle, a can or a cardboard container)⁽⁵⁴⁾. For beverages, one tool (I) uses a cursor to fill the container chosen by the respondent (glass, bowl, etc.).

Method of validity assessment

Table 3 describes the methods used to validate nutrient and/or food group intake estimates using the tool, and Table 4 describes user usability assessment studies.

Table 3.

Methodological characteristics of the validation studies for the online 24-h DR tools

Letter	Name	Studies for the validation of food and nutrient intake estimates	Data collection	Population *	Number of recalls with the online tool	Subjective reference method	Objective reference method	Main criteria for comparison	Type of statistical analyses						Other criteria
									Mean comparison †	Bland and Altman ‡	Coefficients of correlation §	Intra-class coefficient correlations	Cross-classification ||	Other method	Proportion of matches, intrusions, omissions	Percentage of under- and over-reporting
A	ASA24 ¶	(79)	NA	n 93; ≥18 years old; USA	At least two 24-h DR	4-d weighted food diary (2 weeks before recalls with the tool)	N	Energy, nutrient estimates and HEI index between the tool and subjective measurements			X		X
A	ASA24 ¶	(80)	NA	n 81; 20–70 years old; USA	One 24-h DR (for half of the participants) filled out in the lab	One interview-led 24-h DR (for half of the participants)	1-d menu (3 meals) consumed in the lab the day before the recall with the tool. Plates were weighed before and after consumption	Energy and nutrient estimates between the tool, subjective method and objective measurements						Regression analyses to test the bias in nutrient intake estimation between the tools	X (difference between the tools using linear regression)
A	ASA24 ¶	(81)	NA	n 81; 20–70 years old; USA	One 24-h DR (for half of the participants) filled out in the lab	One interview-led 24-h DR (for half of the participants)	1-d menu (3 meals) consumed in the lab the day before the recall with the tool. Plates were weighed before and after consumption	Portion sizes between the tool, subjective method and objective measurements		X				Regression analyses to test the bias in portion size estimates between the tools
A	ASA24 ¶	(82)	2011–2012	n 1052; 20–70 years old; USA	One or two 24-h DR (depending on the randomised group)	One or two interview-led 24-h DR by phone	NA	Energy and nutrient estimates between the tool and subjective measurements	X (equivalence testing, using two one-sided test method with a bound equal to 20 %)					Difference in attrition rates by type of tool
A	ASA24 ¶	(83)	NA	n 627; ≥ 18 years old; USA; women	At least one 24-h DR	7-d weighed food diary (1–5 weeks after recalls with the tool) paper or online semiquantitative FFQ (Harvard or Willett FFQ)	DLW over 1 year, 4 urine samples (N, Na and K measurements), 2 blood samples (fatty acids, carotenoids, folate, tocopherol and retinol measurements)	Energy and nutrient estimates related to biomarkers between the tool, subjective method and objective measurements			X	X
A	ASA24 ¶	(84).	NA	n 1075; 50–74 years old; USA	Six 24-h DR over 1 year	Two 4-d food diaries 6 months apart; Two web-based Diet History Questionnaires, a FFQ	DLW over a 2-week period and two urine samples (protein, potassium and sodium) 6 months apart	Nutrient intakes related to biomarkers between the tool, subjective methods and objective measurements	X (no statistical tests)							X (no tests)
B	CANAA-W	Validation study on the offline 24-h DR software YANA-C(47), on which CANAA-W is based
C	CAPIS
D	Compl-eat TM	(85)	2013	n 47; 18–35 years old; the Netherlands; elite athletes	Three 24-h DR over 2–4 weeks	N	Total urinary N	Protein estimates between the tool and objective measurements	X	X	X
D	Compl-eat TM	(49)	2011–2015	n 514; 20–70 years old; the Netherlands	Three 24-h DR over a year (average number of days between the first and last recall = 354)	Three interview-led 24-h DR over a year	N	Energy, nutrient estimates and food group intakes between the tool and subjective measurements	X	X		X (Lin’s coefficients)
E	Diet Advice	(86)	NA	n 30; 23–60 years old; Australia	One dietary recall	One diet History One Food record	NA	Energy, macronutrient estimates between the tool and subjective measurements		X	X					X
E	Diet Advice	(87)	NA	n 30; 23–60 years old; Australia	One dietary recall	One diet History One Food record	NA	Energy, macronutrient estimates between the tool and subjective measurements		X	X					X
F	DietDay	(88)	2006–2009	n 53; 21–69 years old; USA	Six 24-h DR over a 2-week period	Dietary History Questionnaire (FFQ of 124 items)	DLW over 2 weeks	Energy estimates between the tools and objective measurement			X	X				X
G	FBQ	(52)	2006	n 201; 11–14 years old; Canada	One 24-h DR	One interview-led recall (same day as the tool)	N	Energy and nutrient estimates between the tool and subjective measurements	X		X	X		X
H	FoRC	(53)	NA	n 53; 17–49 years old; UK	Four 24-h DR	Four food diaries (after 24-h recalls using the tool)	N	Energy, fats and some food group intakes between the tool and subjective measurements	X	X	X
I	Intake24	(89)	12/2013–03/2014	n 168; 11–24 years old; Scotland	Four 24-h DR over 1 month	Four Interview-led 24-h DR (same day as the tool)	N	Nutrients and food group estimates between the tool and subjective measurements		X					X
I	Intake24	(72)	11/2015–09/2016	N 98; 40–65 years old; England	At least two 24-h DR over 10 d	N	Total energy expenditure by DLW over 10 d	Total energy estimates with objective energy expenditure		X	X	X				X
I	Intake24	Evaluation as part of the NDNS using DLW in progress (2019–2023)
J	myfood24	(90)	NA	n 212; 18–65 years old; England	At least three 24-h DR over 4 weeks	Three interview-led 24-h DR (2–4 d after the recall with the tool)	Total urinary N, urinary K, Na, fructose and sucrose concentrations, plasma concentration of total vitamin C, vitamin E and β-carotene Total energy expenditure	Energy and nutrient estimates related to biomarkers between the tool and objective measurements		X	X	X
J	myfood24	(91)	NA	n 212; 11–18 years old; England	At least two 24-h DR over 2 weeks	Two interview-led 24-h DR (same day as the tool) at school	N	Energy and nutrient estimates between the tool and subjective measurements		X		X	X
K	NutriNet-Santé	(31)	2009	n 147; 48–75 years old; France	One 24-h DR	One interview-led 24-h DR (the same day as the tool)	N	Energy, nutrient estimates and food group intakes between the tool and subjective measurements			X	X
L	PAC24	(92)	2013	n 41; 7–10 years old; Portugal	One 24-h DR at school (available assistance)	N	Recording type of foods and amount consumed at lunch by trained observers at school	Portion size estimates between tool and objective measurements	X	X					X
M	Web-SPAN (based on FBQ tool)	(70)	2005	n 459; 11–15 years old; Canada	Two 24-h DR, over 2 weeks	Three-day weighted food dairy (after recalls with the tool)	N	Energy and nutrient estimates between the tool and subjective measurements	X		X	X
N	ClinShare
O	Crème Diet (published under the name Foodbook24)”	(58)	NA	n 40; 18–64 years old; Ireland	Three 24-h DR	Four-day semi-weighed food diary (10 d after the recall with the tool)	Plasma concentration of carotenoids, ascorbic acid, fatty acids and total urinary N, urinary K and Na concentrations	Energy, nutrient estimates and food group intakes between the tool and subjective measurements Nutrient estimates and food group intakes related to biomarkers between the tool, subjective method and objective measurements	X	X	X		X
P	R24W	(93)	NA	n 107; 18–65 years old; Canada	Four 24-h DR over 20 d	N	Serum carotenoids	Food and nutrient estimates related to biomarkers between the tool and objective measurements			X		X
P	R24W	(94)	NA	n 107; 18–65 years old; Canada	Four 24-h DR over 20 d	Three-day weighed food diary (before recalls with the tool)	N	C-HEI score and components between the tool and subjective measurements	X		X		X
P	R24W	(95)	NA	n 107; 18–65 years old; Canada	Four 24-h DR over 20 d	Three-day weighed food diary (before recalls with the tool)	N	Energy and nutrient estimates between the tool and subjective measurements Energy estimates between the tool and estimated energy expenditure (Mifflin equations)	X	X	X		X			X
P	R24W	(96)	NA	n 62; 18–5 years old; Canada	Two 24-h DR recalls	N	Seven-day cyclic menu for 4 to 7 weeks, consumed outside, except lunch consumed in the lab	Portion sizes between the tool and objective measurements Energy and macronutrient estimates between the tool and objective measurements	X		X		X		X
Q	RiksmatenFlex	(30)	NA	n 78; 11–18 years old; Sweden	Two 24-h DR, 1 at school and 1 at home 1–2 weeks later	Two interview-led 24-h DR 2–4 weeks after the recall with the tool, 1 at school and 1 at home (by phone) 1–2 weeks later	Plasma alkylresorcinols and carotenoids biomarkers Energy expenditure using accelerometer ActiGraph GT3X over 7 d	Energy, food and nutrient estimates related to biomarkers between the tool and subjective measurements Food and nutrient estimates related to biomarkers between the tool, subjective method and objective measurements Total energy estimates and energy expenditure	X	X	X	X	X
R	SACANA	(97)	2013–2014	n 228; 5–18 years old; Belgium, Cyprus, Estonia, Germany, Hungary, Italy, Spain, Sweden and Poland	At least three 24-h DR	One FFQ of 59 items over the previous month	Total urinary fructose and sucrose concentrations	Sugar estimates between the tool, objective measurements and relative frequency of sweet foods			X			Method of triads using linear regression

ASA24, Automated Self-Administered 24-h diet recall; CANAA-W, Children’s and Adolescents’ Nutrition Assessment and Advice on the Web; CAPIS, Computer-Assisted Personal Interview System; FBQ, Web-based Food Behaviour Questionnaire; FoRC, Food Record Checklist; myfood24, Measure Your Food On One Day; N, No; NA, missing values; PAC24, Portuguese self-administered computerised 24-h dietary recall; R24W, Web-based 24-h dietary recall; Web-SPAN, Web-Survey of Physical Activity and Nutrition; SACANA, Self-Administered Children, Adolescents, and Adult Nutrition Assessment; HEI, Health Eating Index; C-HEI, Canadian Healthy Eating Index; DLW, doubly labelled water; NDNS, National Diet and Nutrition Survey; 24-h DR, 24-h dietary recall.

Grey cells are tools without publications on the tool.

^*Final sample size, age, country and specificity if needed

^†t-test or paired t-tests or Wilcoxon signed rank test;

^‡graphical method and limit of agreements

^§Spearman or Pearson, de-attenuated or raw correlation;

^||Cross-classification and weighted kappa coefficient;

^¶ASA24 was also validated among specific subpopulations such as low-income individuals, children 10–13 years of age, overweight and obese women, multi-ethnic older adults. All publications are available here: https://epi.grants.cancer.gov/asa24/resources/publications.html.

Table 4.

Methodological characteristics of the user usability studies for the online 24-h DR tools

Letter	Tool name	Studies on user usability and acceptability	Data collection date	Population *	Number of dietary recalls with the online tool	Method	Main result
							User satisfaction	Issues	Other
A	ASA24 †	(35)	NA	Study 1: n 40; 2–5 years old (parental reporting); Canada Study 2: n 294; 10–13 years old; Canada Study 3: n 98; 10–13 years old; Canada Study 4: n 331; 36–82 years old; Canada Study 5: n 264; 46–88 years old; Canada	Study 1: one 24-h DR after an observational feeding day (ASA24-Canada-2014) Study 2: one 24-h DR at school with observation (ASA24–2016, US) and one at home Study 3: one 24-h DR at school after an observational feeding day (ASA24–2016 US and ASA24–2014-Kids) Study 4: four 24-h DR over 4 months (ASA24-Canada-2014 and ASA24–2016 US), assistance available by phone or email Study 5: four 24-h DR over 3 months (ASA24-Canada-2014), assistance available by phone or email	Attrition and success and main technical issues in each study Study 1: NA Study 2: usability questionnaire and researcher comments at school Study 3: one 24-h DR at school after an observational feeding day (ASA24–2016 US and ASA24–2014-Kids) and researcher comments at school Study 4: NA Study 5: assistance comments	Study 2: majority indicated they found completing ASA24 ‘very easy’, ‘easy’ or ‘neutral’	Navigation; finding the correct food, in particular for multiple-word searches; language not child-friendly; log-in issues; assistance available only during office hours	Study 1: Median 35 min; Study 2–3: 34 min; Study 4: 34 min
A	ASA24 †	(82)	NA	n 942; 20–70 years old; USA; Focus on subgroup that completed both ASA24 and interview-led recalls	One 24-h DR One interview-led 24-h recall using AMPM	Questionnaire on the preference between ASA24 and interview-led recall using AMPM	70 % preferred ASA24 over AMPM, with a significant decrease with age	NA	NA
A	ASA24 †	(98)	NA	n 39; ≥ 18 years old; USA; low income	One 24-h DR	Comparison of attrition rates between unmoderator (no help), semi-moderator (audio and video recording) and moderator group (audio and video recording, think-aloud, help requests available) Analysis of audio and video recording among moderator and semi-moderator groups by categorising each task and issue Quantitative (number of task successes, number of issues, time, food item count) and qualitative analyses of each task and usability issue	NA	34·6 % of issues out of 286 related to effectiveness (ability to perform a task, e.g. submit incorrect information, next step unclear), 45·8 % related to efficiency (effort to complete a task, e.g. search item missing or inaccurate, mis-click), 4·2 % related to satisfaction (desired option not available), 15·4 % related to comprehension (e.g. question not understood)	Average time 27·4 min
B	CANAA-W	(46)	2011	n 65; 10–12 years old; Belgium Parents from primary school	Children: At least two 24-h DR (one under the supervision of researchers at school and one at home) Parents: Three 24-h DR of their children	Eight focus groups for children and parents Feasibility questionnaire for parents on user-friendliness, enjoyment, attractiveness and clarity of feedback.	More than 50 % agreed that the tool was clear, easy to complete, comprehensible and understandable	NA	Reasons for drop out: lack of time, lack of knowledge about child’s food, slow application
C	CAPIS	(48)	NA	n 200; ≥ 20 years old; Korea	One 24-h DR One paper 24-h DR	Difference of time to collect data between methods using t test Usability questionnaire (5 items)	Online tool was easier and faster than the paper 24-h DR	NA	Mean completion time: 14 min (28 min for the paper 24-h DR)
D	Compl-eat TM
E	Diet Advice
F	DietDay	(51)	NA	n 261; 21–69 years old; USA	Eight 24-h DR	Usability questionnaire (11 items)	75 % found the DietDay easier than the CASI-DH
G	FBQ	(52)	NA	n 11 dietician experts; Canada n 21; 11–12 years old; Canada	NA	Think-aloud method	Positive feedback about the content and appearance of the survey and the process of data collection	Finding the correct food	NA
H	FoRC
I	Intake24	(54)	NA	n 80; 11–24 years old; UK	Three rounds of 24-h DR using the tool followed by one interview-led recall	Think-aloud method Eye-tracking Usability questionnaire (10-item) using an adapted SUS-scale	Average SUS-score 83/100 for the latest tool version	Finding the correct food; Navigation	NA
I	Intake24	(99)	2015	n 182; ≥ 11 years old; Scotland	Four 24-h DR over 10 d	Usability questionnaire (general questions, usability of specific functionalities and satisfaction) Free comments	80 % agreed that the system was easy to follow and understand	Finding the correct food, in particular for multiple-word searches (‘mince, potatoes’, ‘ham sandwich’); Log in to the system	Reasons for refusal or drop out: no interest in the study, do not have time
I	Intake24	Evaluation as part of the NDNS using DLW in progress (2019–2023)
J	myfood24	(100)	NA	Study 1 (beta version): n 14; 11–18 years old; UK Study 2 (Improved live version): n 70; 11–18 years old; UK	Study 1: 24-h DR moderated by researcher for 50 % of participants and at home for the others Study 2: myfood24 24-h DR at school and one led-interview 24-h DR	Study 1: screen capture, verbal recording when doing standardised tasks usability-acceptability questionnaires (3 open-questions on myfood24, 5-likert scale questions on acceptability and satisfaction: SUS scale + 8 items) Study 2: same questionnaire as in study 1 and preference between methods	Study 1: average SUS score 66/100 Study 2: Average SUS score 74/100, 41 % preferred the myfood24 to the interview-led recall	Stage I: finding the correct food, using the recipe functionality, navigation	mean completion time: Stage I: 31·8 min Stage II: 16·2
J	myfood24	(36)	NA	Study 1: n 92; ≥ 18 years old; Germany Study 2: n 15; ≥ 18 years old; Germany	Study 1: Four 24-h DR (first recall with a researcher) Study 2: Enter in the online tool, 3 sample meals presented in a lab to assess the search functionality	Study 1: Usability-acceptability questionnaire (68 items): SUS scale, overall friendliness, willingness to use the tool, technical details and opinion on user interface or specific functionalities Free comments and overall suggestions Completion time Study 2: Analysis of screen video: number of search terms, way to search a product, number of exclusions, number of intrusions, impact of search behaviour on energy and nutrient intakes compared to the nutrient reference values of the real products	Median SUS score 78/100, lower in women than men User-friendliness as good or very good (67 %)	Finding the correct food, using the recipe functionality	Median completion time: 15 min, increase with age
K	NutriNet-Santé	(31)	NA	n 147; 48–75 years old; France	One 24-h DR and one food diary	Questionnaire on attitudes towards the web, time to complete the recall, opinion and method preference	The online method was preferred by 66·1 % of the subjects compared to food diary	NA	Mean completion time: 31 min
L	PAC24
M	Web-SPAN (based on FBQ tool)
N	ClinShare
O	Creme Diet (published under the name Foodbook24)	(58)	NA	n 118; 18–64 years old; Ireland	Three 24-h DR 4-d food diary	16-item questionnaire on user acceptability (acceptability of some functionalities, method preference, future use, overall satisfaction)	69·5 % reporting easy or ‘OK’ to use, 67·8 % preferred the online method compared to food diary	NA	NA
P	R24W	(59)	NA	n 29; ≥ 16 years old; Canada	One 24-h DR	Satisfaction questionnaire Free comments	A large majority of respondents (90 %) agreed that R24W was easy to access, to understand and to complete	NA	NA
Q	RiksmatenFlex	This was carried out as part of development of the tool
R	SACANA

ASA24, Automated Self-Administered 24-h diet recall; CANAA-W, Children’s and Adolescents’ Nutrition Assessment and Advice on the Web; CAPIS, Computer-Assisted Personal Interview System; FBQ, Web-based Food Behaviour Questionnaire; FoRC, Food Record Checklist; myfood24, Measure Your Food On One Day; PAC24, Portuguese self-administered computerised 24-h dietary recall; R24W, Web-based 24-h dietary recall; SACANA, Self-Administered Children, Adolescents, and Adult Nutrition Assessment; SUS scale, System Usability Scale; Web-SPAN, Web-Survey of Physical Activity and Nutrition; DLW, doubly labelled water; NDNS, National Diet and Nutrition Survey.

Grey cells are tools without publications on the user usability;

^*Final sample size, age, country and specificity if needed;

^†ASA24 usability was also assessed among children and multi-ethnic older adults.

Validation of nutrient intake estimates was assessed in twenty-seven studies (n 15 tools). Three tools (B, C and N) had no publication on the validation of nutrient intake estimates. Six tools (A^(79,83,84), E^(86,87), H⁽⁵³⁾, M⁽⁷⁰⁾, O⁽⁵⁸⁾ and P^(94,95)) compared nutrient intake estimates to those from food diaries, seven (A^(80–82), D⁽⁴⁹⁾, G⁽⁵²⁾, I⁽⁸⁹⁾, J^(90,91), K⁽³¹⁾ and Q⁽³⁰⁾) to nutrient intakes estimated by interview-led 24-h DR and three (A⁽⁸³⁾, F⁽⁸⁸⁾ and R⁽⁹⁷⁾) to estimates from FFQ. The number of days of dietary measurements as well as the time between data collection using the tool and the reference method varied widely between studies. For instance, from one (A^(81,83,101), E^(86,87), G⁽⁵²⁾, K⁽³¹⁾ and L⁽⁹²⁾) to six consumption days (A⁽⁸⁴⁾ and F⁽¹⁰²⁾) were collected using the online 24-h DR tool in validation studies. Four tools (G⁽⁵²⁾, I⁽⁸⁹⁾, J⁽⁹¹⁾ and K⁽³¹⁾) were validated against a reference method administered the same day^{(31,52,89,91)}, whereas other tools administered the reference method a few weeks before or after use of the tool. Ten tools (A^{(80,81,83,84)}, D⁽⁸⁵⁾, F⁽⁸⁸⁾, I⁽⁷²⁾, J⁽⁹⁰⁾, L⁽⁹²⁾, O⁽⁵⁸⁾, P^(93,96), Q⁽³⁰⁾ and R⁽⁹⁷⁾) had validation studies using objective measurements (biomarkers or energy expenditure n 10 studies, corresponding to nine tools A^(80,83,84), D⁽⁸⁵⁾, F⁽⁸⁸⁾, I⁽⁷²⁾, J⁽⁹⁰⁾, O⁽⁵⁸⁾, P^(93,96), Q⁽³⁰⁾ and R⁽⁹⁷⁾; feeding studies n 1 study: A⁽⁸¹⁾; or direct observation n 1 study: L⁽⁹²⁾), nine (A, D, F, I, J and O–R) of which also had a validation study with a subjective reference measurement (in the same or another study). Six tools (A^(80,83,84), F⁽⁸⁸⁾, J⁽⁹⁰⁾, O⁽⁵⁸⁾, Q⁽³⁰⁾ and R⁽⁹⁷⁾) were validated with both subjective and objective reference measurements in the same study, as recommended by Timon et al. ⁽¹⁰³⁾. Four tools (A⁽⁸⁰⁾, I⁽⁸⁹⁾, L⁽⁹²⁾ and P⁽⁹⁶⁾) assessed the proportion of exact ‘matches’, ‘omissions’ or ‘inclusions’.

Data were often analysed using a combination of statistical methods, measuring either the strength of an association at the individual level (correlation coefficients), the overall agreement between two measurements (mean comparisons), the agreement at the individual level (cross-classification and weighted Kappa coefficient), or the presence, direction and extent of bias between two measurements (graphics of Bland and Altman). The number of statistical analyses was between 2 and 5, with four studies out of twenty-seven (G⁽⁵²⁾, O⁽⁵⁸⁾, P⁽⁹⁴⁾ and Q⁽³⁰⁾) having more than three different statistical tests, as recommended by Lombard et al. to reflect each facet of validity⁽¹⁰⁴⁾. Publication results indicated overall moderate to good validity of online 24-h DR according to the statistical tests, and estimated nutrient intakes were comparable to the reference values. For instance, in a control feeding study, gaps between true and reported energy, nutrient and food group intakes were comparable between the online tool A and the interview-led offline AMPM software⁽⁸⁰⁾. Validation criteria were comparable between the online tool J and interview-led 24-h DR, with several biomarkers⁽⁹⁰⁾. Spearman’s correlations for urinary and plasma biomarkers were similar for both the online tool O and 4-d semi-weighed food diaries⁽⁵⁸⁾. Overall, based on their validation studies, each tool is valid to estimate nutritional intakes (data not shown).

User usability assessment

User usability was assessed in fifteen studies (n 11 tools, A–C, F, G, I–K and O–Q), among which one tool (Q) assessed usability but without publishing the results. In eight studies (n 7 tools, A^(35,82), C⁽⁴⁸⁾, F⁽⁵¹⁾, I⁽⁹⁹⁾, K⁽³¹⁾, O⁽⁵⁸⁾ and P⁽⁵⁹⁾), user usability was assessed only using a retrospective questionnaire administered after data collection. The System Usability Scale (SUS)⁽¹⁰⁵⁾, a validated questionnaire of ten items measuring the overall usability of a system (i.e. software, website and application) was used in three studies (n 2 tools, I⁽⁵⁴⁾ and J^(36,100)). SUS scores at least equal to 70 (out of 100) are considered ‘good’ by Bangor et al. ⁽¹⁰⁶⁾. Concerning methods other than questionnaires, we can mention focus groups⁽⁴⁶⁾ (n 1 tool, B), a retrospective methodology to collect qualitative information and real-time methods such as think-aloud protocols^{(52,54,56,107)} (n 4 tools, A, G and I) as well as eye-tracking⁽⁵⁴⁾ (n 1 tool, I). In four studies (n 3 tools, A⁽³⁵⁾, I⁽⁵⁴⁾ and J^(36,100)), both retrospective and real-time methods were used. Overall satisfaction could be considered good, but several common issues were reported: difficulties in identifying the correct food (A^(35,107), I^(54,99) and J^(36,100)), in particular when the respondent used several words (e.g. ‘mince, potatoes’), issues in navigating within the system (A^(35,107), I⁽⁵⁴⁾ and J⁽¹⁰⁰⁾), and difficulties logging in (A⁽³⁵⁾ and I⁽⁹⁹⁾).

Tool flexibility

Among the eighteen tools, thirteen (B–H and L–Q) have not been adapted for use in another country (Table 1). Information about how the tool could be adapted from the investigator of the study and/or from the tool’s technical support team was collected for eleven tools. For eight tools (A, D, I–K, N, O and R), changes to the food list and addition of full nutritional composition are feasible by providing the data to technical support, as a template file with a specific structure. Addition of another language is feasible for six tools (A, I–K, O and R). A web platform is available for the investigator of the study for eight tools (A, D, I, J, N, O, Q and R). On the platform, it is possible, depending on the tool, to edit certain parameters: adding new foods, changing nutritional composition, amending portion size pictures, activating functionalities or questions, and managing a study (sending invitation emails, checking responses and exporting the databases). Finally, tools A, I, J, O and R seemed to be the most easily flexible to a new context (web platform for the investigator of the study, possible addition of another language and modification of the input data). Only three tools (I, O and soon A) allow flexibility to store the collected data on a server of the investigator team. For two other tools (K and R), data can only be exported on request, limiting ongoing monitoring of the study.

Discussion

Eighteen online 24-h DR tools were identified and described in detail. Most were developed in Europe, for children 10 years of age and older and/or for adults. All tools are self-administered and collect time of consumption, identification of all foods and beverages consumed, and quantification of the amount consumed, before checking and validating the entries. The common information collected by all tools makes it possible to obtain high-quality intake estimates, showing promising capabilities for their use in national food intake surveys. Beyond these similarities, each tool has its own specificities regarding the order and functionalities of the multiple-pass steps to help identify and quantify the foods consumed. These specificities may have an impact on user usability, which was assessed for fewer tools than the validity of nutritional intakes. User usability should be assessed more often, especially for tools to be used in national dietary surveys because usability is a major driver of the response rate, a significant challenge in such surveys. Moreover, the ability of these tools to be adapted to new environments needs to be carefully evaluated, in view of implementing them in different countries. This point is, however, rarely addressed in reports or articles. This is why the authors of the present study needed to conduct unstructured interviews with the owner or developer of each tool to obtain more information.

Eleven tools were assessed regarding user usability, mainly through retrospective data collection of user satisfaction using questionnaires. Initiated by Eysenbach in 2005⁽¹⁰⁸⁾, the impact of design features on adherence, that is, the degree to which the user correctly uses the tool as designed and intended by the developer⁽¹⁰⁹⁾, has been studied in particular in online intervention programmes on mental health, lifestyle or chronic care, to prevent non-usage and dropout attrition^(110,111). For instance, it is recognised that personalisation of functionalities (e.g. using an avatar for children) or content (e.g. providing tailored messages) for a specific target group or individual increases user efficiency⁽¹¹⁰⁾. Theoretical models on adherence to web-based interventions have been developed⁽¹¹¹⁾ and could help to identify recommendations for designers to make the tool more attractive and easier to use. Among American adults, ASA24 (tool A) was preferred to interview-led AMPM software for 70 % of individuals⁽⁸²⁾. The attrition rate, defined as the percentage of individuals lost between the first and second 24-h DR, was slightly lower using ASA24 (tool A) (6 %) compared to AMPM (11 %), but no analyses were conducted to further understand the effect of the web-based system on this difference⁽⁸²⁾. More research is needed in this field to better identify, quantify and qualitatively describe issues and find opportunities to improve available tools.

Among the issues raised in user usability studies, a common one observed across tools is the ability to easily identify the correct food. Some tools have improved the keyword search engine^{(30,58,76,100)}, but optimising the search mechanism remains a field of development to improve attractiveness and user success. Doing so may improve user adherence, response rates and the validity of dietary data. Identifying the correct food is also highly dependent on the quality of the integrated food list, which must be diversified enough and representative of the population’s food habits. With the development of online platforms (e.g. OpenFoodFact⁽¹¹²⁾), dedicated to providing product labelling information on branded foods available on the market, the possibility of integrating these exhaustive databases into 24-h DR tools could be considered. There is no absolute agreement on the advantages of using branded products rather than generic foods in the database of the recall tools⁽³⁶⁾ but for the researcher, the collection of dietary data at branded level can provide many descriptors with less data management: the type of packaging, presence of a nutrition or health claim and fortification. However, when foods are at brand level, the challenge is to link each food to full nutritional composition (macronutrient and micronutrient content), generally available for generic foods. To reduce data management for researchers, automatic or semi-automatic procedures have recently been proposed to match foods with food composition tables, using fuzzy matching (comparison between two character strings) to provide a similarity score between food names and/or machine learning classifiers^(113,114), or by estimating the percentage of agreement based on the available nutritional content between the brand and generic food⁽¹¹⁵⁾. When the choice is to use a generic food database, the tool must be adapted to collect additional information about the food consumed concerning aspects relevant to the study aims (e.g. source of food: purchased or home-made). For instance, ASA24 (tool A) uses an extensive database of more than 13 million pathways to collect detailed information on the foods consumed⁽¹¹⁶⁾, but collection of the additional facets increases respondent burden. The development and integration of barcode scanning to identify foods⁽²⁶⁾ may improve usability in the next few years and could ease data collection for the user and investigator of the study. Barcode scanning is, however, not yet integrated in published online 24-h DR tools.

One challenge for 24-h DR tools to be used at national level is to ensure representativity and ideally to be adaptable to different countries. Ensuring representativity at the national level is challenging because studies have shown that age^{(34,55,82,117)} and income or educational level^(34,98,118) affected user usability with online 24-h DR. As a consequence, protocols must be tailored to the subpopulation (e.g. data collection at school^{(30,47,89,91)}, to provide 24-h support, to allow collection of data with an interviewer⁽³⁴⁾, to provide public internet access, to offer a specific version for children by simplifying the language and adding an avatar⁽¹¹⁹⁾). If the protocol or tool cannot be adapted, the dietary survey could be supplemented with an external study. For instance in France, the Nutri-Bébé 2013 survey, an observational cross-sectional study of children aged 15 d to 35 months living in France, collected detailed food consumption using food diaries filled by the parents and could supplement national INCA dietary surveys⁽¹²⁰⁾. Adapting 24-h DR to other countries can be very time-consuming and expensive, as previously shown with adaptation of GloboDiet^{(8,17,121,122)}. Most of the online 24-h DR tools reviewed in this study were developed for a highly specific context, limiting their potential adaptation. Furthermore, probably because our search criteria included online tools, most of the selected online 24-h DR were developed for high-income countries, as already highlighted by Bell et al. ⁽²²⁾. Therefore, the tools identified may not be suitable for countries with specific constraints, such as low- and middle- income countries, in which a limited literacy and numeracy may be source of error when using a self-administered tool⁽¹²³⁾, and where the tool may be unusable in some region with a low internet connectivity^(22,25). But, as mentioned in the results, some of the tools identified in this paper were already or currently being adapted for being used in some low- and middle-income countries. The development for a new population, such as a new country or age class, requires an update of the pre-integrated food list, food composition database and food portion pictures to be representative of the population’s food habits. This must be followed by new assessment of validity and user usability, as done by Koch et al. for adaptation of myfood24 (tool J) to the German population⁽³⁶⁾. The available languages must also be adapted, if needed. Even though some tools have developed a web platform, easing the integration of new data, or were specifically developed to allow simple updates using file templates, considerable work will be required to construct the integrated database.

A few limitations of this review should be noted. First, our descriptions of the tools were mainly based on information available in papers or reports. Except for six tools (tools A, I–K; O, R), which had a demo version freely available or a presentation video, the authors of this study did not test the tools, and some information may have been missed. However, for eleven tools, the owner and/or developer reviewed and validated the requested information, limiting inaccuracies. Second, we chose to describe only the method used in validation studies without providing the results, which may limit appraisal of each tool. As noted by Timon et al. ⁽¹⁰³⁾, high heterogeneity in the design of validation studies means that studies must be assessed in isolation, without any robust comparison between tools. Additionally, validation and user usability assessment studies are specific to the population studied and must be renewed when applied to a different context. Nevertheless, our results provide an overview of the quality of the validation and user usability studies conducted with each tool. Third, in all publications, there is little evidence that using 24-h DR is cost-effective, although this argument was often put forward in papers on new technologies^(38,124). Fourth, we choose to not assign a ranking of the tool, because each decision-makers have their own criteria and needs. Our objective was to describe as precisely as possible the tools, regarding various aspects, in order to provide enough information for decision-makers to identify the best opportunities. Finally, the aim of this review was to focus on online 24-h DR tools, but technologies are moving rapidly and other technologies, in particular smartphone applications with visual recognition could evolve quickly and be validated for use in large-scale surveys. Likewise, some new validation studies^{(93,125–128)} or user’s usability studies^(77,128) have been published since 2019, after the literature search conducted for this paper. Those articles published since 2019, not described in detail in this paper, are related to tools which were already described in this paper.

Conclusion

Eighteen online self-administered 24-h DR tools developed and validated in several contexts were identified. Tools that were validated to estimate nutritional and food intakes, that have confirmed user and researcher usability and that are sufficiently flexible to be adapted to different contexts, are probably the best candidates for use in national dietary surveys, as they are likely to improve response rates and to collect high-quality data. Regardless of the tool, adaptation to another context will require time and funding, and this is probably the most challenging step⁽¹³¹⁾.

Financial Support

The study was supported by ANSES.

Conflict of Interest

S.H. and C.D. have no conflicts of interest. R.G, S.M. and F.V. are employees of MS-Nutrition.

Authorship

R.G. designed the study, collected, analysed and interpreted the data, and wrote the manuscript. C.D. and S.H. designed the study and contributed to interpretation of the data. S.M. contributed to data collection and interpretation of the data. F.V. contributed to the analysis and assisted in writing the paper. All the authors reviewed the manuscript.

Ethical Standards Disclosure

Not applicable