Comparison of a web-based versus traditional diet recall among children

Abstract

Self-administered instruments offer a low-cost diet assessment method for use in adult and pediatric populations. This study tested whether eight to 13 year old children could complete an early version of the Automated Self Administered 24 (ASA24) hour dietary recall and how this compared to an interviewer-administered 24-hour dietary recall (24 HDR). One-hundred and twenty eight to13 year old children were recruited in Houston from June through August 2009, and randomly assigned to complete either the ASA24 or an interviewer-administered 24 HDR, followed by the other recall mode covering the same time interval. Multivariate analysis of variance, testing for differences by age, gender and ethnic/racial group, were applied to percentages of food matches, intrusions, and omissions between reports on the ASA24 and the interviewer-administered 24 HDR. For the ASA24, qualitative findings were reported regarding ease of use. Overall matches between interviewer-administered and ASA24 self-administered 24 HDR was 47.8 percent. Matches were significantly lower among younger (eight to nine year old), compared to older (10 to 13 year old) children. Omissions on ASA24 (18.9 percent overall) were most common among eight year olds and intermediate among nine year olds. Eight and nine year olds had substantial difficulties and often required aid in completing ASA24. Findings from this study suggest that a simpler version of a web-based diet recall program would be easier for children to use.

INTRODUCTION

Accuracy in dietary assessment has been recognized as critically important in understanding the influences of diet on health (1), evaluating current nutritional status (2), and assessing dietary behavior change (3). The interviewer-administered 24 hour dietary recall (24 HDR) is considered the gold standard diet assessment method among adults (4) and children (5). However, it entails measurement error (6) and due to its interviewer-administered nature, can be both expensive and logistically difficult for the interviewer to complete. Less expensive alternatives have been attempted (7). To be successful, however, any self completed dietary assessment instrument requires some minimum attention, memory, and categorization skills (8). Useful skills for successfully self-completing a 24 HDR is the identification of foods consumed by “browsing” among hierarchically organized food groups or by “searching” (typing in food names). Among eight to 13 year old children, child versus adult-generated food categories (9-12) enabled speedier, but not more accurate, categorization of foods (13). A graphically appealing browse procedure called “cover-flow” (i.e. an animated three dimensional graphic user interface for visually flipping through food picture categories and subcategories of items) was not any more effective for categorizing foods than the more common text-based tree-structure (13). Regardless of method, younger children (eight to nine year olds) tended to have lower accuracy and took longer to categorize foods (13). Observers reported that many children relied on pictures of foods (category collages) rather than text to make their selections in the cover-flow method (13). Multiple smaller food images in progressively larger amounts on the same screen enabled children to more quickly report food portion size than larger single-portion pictures presented one at a time (14). The presence or absence of visual cues such as a tablecloth and cutlery in images did not influence accuracy of portion size (14).

The National Cancer Institute convened a group of dietary assessment specialists to create a web-based automated self administered 24 hour dietary recall (ASA24) (15-17). Investigators incorporated validated methods, e.g. US Department of Agriculture's Automated Multiple Pass Method (AMPM) (18), and is currently being validated in adult populations. The beta version of ASA24 used in this study included standardized detailed probes, and up to eight food images in progressively larger sizes for portion size estimation (16). The graphic user interface was programmed to use an avatar to guide the respondent through completion of all tasks. The detailed probing used in ASA24 (e.g. type of fat used, sodium content, cooking method), however, may exceed the knowledge and reporting skills of children, and therefore should be tested with them.

The primary aim of this study was to test whether children of different ages could complete an early version of the ASA24 compared to report to a dietitian. A secondary aim was to identify improvements to enhance ASA24. The hypothesis tested was that substantial disagreements would occur regarding foods self-reported on the ASA24 and those reported to an interviewer, with more disagreements among younger children (8 or 9 year olds) versus older children (10 to 13 years old).

METHODS

From June through August 2009, a study was conducted using a quasi experimental design, randomly assigning 120 eight to 13 year old boys and girls to first complete either an early beta version of the ASA24 or an interviewer-conducted 24 HDR using the Nutrient Data System for Research software (19) (NDSR 2008 for collection; NDSR 2009 for analyses to benefit from time related updates, University of Minnesota, Minneapolis, MN) followed by the other method covering the same time interval. Sixty children started with one method. Food-by-food comparisons were made between the foods entered in ASA24 and those reported during the interviewer-administered recall. The interviewer-administered 24 HDR was used as the criterion, since the dietitian could help the child stay on task, explain the probes, and otherwise be responsive to the child and his/her way of reporting.

The inclusionary criteria were being a child eight to 13 years old (since eight is an age where children have experienced difficulties in dietary recall and 13 year olds appear to respond similarly to adults (13)); and being able to speak, read and write English. This study was conducted with eight to 13 year old children to determine if there was an age at which it was more difficult to use ASA24 specifically and/or at which greater discrepancies in reporting occurred between ASA24 and an interviewer-administered recall. Exclusionary criteria were having a physical, mental or visual limitation that would inhibit the child's ability to recall diet or use a computer. An effort was made to balance the sample on age and gender. Parents completed signed informed consent and each child provided signed assent. The protocol was approved by the Baylor College of Medicine Institutional Review Board.

Recruitment was conducted by a professional recruiter from the Children's Nutrition Research Center's research project volunteer data base. Additional recruitment was conducted with fliers distributed on public information boards around the Texas Medical Center. Preliminary screening for inclusionary and exclusionary criteria was conducted by the recruiter and then confirmed by project staff. Children were compensated $25 for their participation.

Parents completed demographic questionnaires. Interviewer-administered 24 HDRs were conducted by dietitians trained and certified annually by senior project staff. All recalls were reviewed by a senior dietitian (20). The target reference period for this study was midnight to midnight the previous day, identical to the time frame for ASA24.

ASA24 recalls were self-administered using the beta version available in February 2009, which used Food and Nutrient Database for Dietary Studies (FNDDS 1.0) developed by the US Department of Agriculture. During ASA24 administration a trained dietitian-observer made notes about difficulties encountered, solved major difficulties (that foiled further progress), and conducted a post ASA24 qualitative interview about all aspects of the experience, including what the children liked, did not like, and how to improve the tool. Children were encouraged to speak aloud about what they were doing and thinking while completing ASA24 to give the observer insight into the ease of task completion. Observations were recorded on forms structured by sequence of events in the ASA24. The interviewers evaluated ASA24 usability including the use of search (type in food name) versus browse (through hierarchically organized food categories) methods to find foods, portion size estimation method (portion size images presented in progressively larger servings), and children's understanding of terms used to identify foods, follow-up probes regarding the foods (such as how the food was prepared), how to report additions (such as butter to bread), and how to report multi-ingredient foods (such as salads or sandwiches). All interviewers were trained in qualitative data collection methods and followed a detailed written protocol to ensure consistency in data collection procedures. Observers were not the same individuals conducting the interviewer-administered 24 HDR.

To quantify differences between the interviewer-administered 24 HDR and the self completed 24 HDR using ASA24, output analysis files from both instruments were obtained and examined. These files reflected differences in the food coding databases, food descriptions and recipes used to represent foods reported by participants as well as differences in the foods reported by the children for the same recalled day. One senior dietitian paired foods from the two systems and classified them into five mutually exclusive categories: matches at the food level (e.g. plain hot dog versus plain hot dog), matches at the food category level (e.g. plain ham sandwich versus ham sandwich with pickles, mayo and cheese which are both in the same AMPM category), no match at any level (e.g. “barbecue beans” (NDSR) and “string beans, green, cooked, from a can” (ASA24) both reported for a dinner; “blueberries, frozen, unsweetened” (NDSR) and “pie, blueberry, one crust” (ASA24)) both reported for breakfast, intrusions (foods recorded in ASA24, but not NDSR), and omissions (foods recorded in the NDSR, but not ASA24). The denominator for all percents was the sum of items from NDSR plus the intrusions from ASA24. The categorizations were reviewed by one other staff member.

Univariate analyses of variance (ANOVA) were used to identify differences in percent matches and multivariate analysis of variance (MANOVA) was used to identify differences in percent category matches, omitted, intruded and not matched, by participant characteristics. For models with significant multivariate effects, meaning a difference among at least one of the dependent variables, results for each dependent variable (the univariate ANOVA) were subsequently examined while appropriately controlling for type I error. For significant univariate ANOVA effects, Bonferonni's post hoc pairwise comparisons were examined for significant univariate gender and race main effects. Because of the ordinal nature of age group, polynomial contrasts instead of pairwise comparisons were used in the investigation of differences. Analyses were conducted using Statistical Analysis Software (version 9.1.3., 2006, SAS Institute Inc, Cary, NC).

The qualitative interviews were structured by the sequence in which information was requested in the ASA24. Interviews were audio recorded and transcribed. Observer's comments and children's responses were coded by two dietitians. Conflicts in coding decisions were reviewed by all coders and investigators, and resolved by consensus. A thematic analysis was conducted of observer comments and interviews (21).

RESULTS AND DISCUSSION

Data were analyzed from 120 children. Forty-eight percent of foods reported were matches of specific foods (Table 1); 18.1 percent were matches of foods only at the food category level (a form of error); 19.0 percent were omissions; 12.5 percent were intrusions (reported on ASA24, but not the interviewer-administered 24 HDR); and 2.6 percent were non-matches.

Table 1.

Overall food matches, intrusions, omissions, and non-matches between automated self-administered and dietitian conducted modes of 24 hour dietary recall among children ages 8-13 (n=120).

Characteristic	n (%)	ANOVA	MANOVA
		% Foods matched at food level	% Foods matched at category level	% Foods omitted	% Foods which intruded	% Foods not matched at any level	Multivariate Main effects
		M (SE)	M (SE)	M (SE)	M (SE)	M (SE)
Gender							F(4,107)=2.98, p=.022
Boy	60 (50.0)	46.4 (2.9)	18.4 (1.8)	20.0 (2.2)	11.5 (1.8)	3.8 (0.7)
Girl	60 (50.0)	53.4 (3.0)	17.8 (1.9)	13.1 (2.2)	14.2 (1.9)	1.5 (0.7)
Univariate tests		p=.084		p=.026	p=.270	p=.026
Age Group							F(20,440)=1.70, p=.031
8	20 (16.7)	36.4 (4.9)	15.4 (3.1)	31.4 (3.7)	11.4 (3.1)	5.4 (1.2)
9	24 (20.0)	45.1 (4.4)	21.1 (2.8)	18.6 (3.4)	13.0 (2.8)	2.1 (1.1)
10	18 (15.0)	50.4 (5.3)	19.5 (3.3)	15.4 (4.0)	11.5 (3.3)	3.3 (1.3)
11	21 (17.5)	51.4 (4.8)	19.8 (3.0)	15.3 (3.6)	12.6 (3.0)	0.9 (1.2)
12	17 (14.2)	63.7 (5.4)	14.1 (3.4)	8.6 (4.1)	11.8 (3.4)	1.7 (1.4)
13	20 (16.7)	52.5 (5.0)	18.5 (3.2)	9.9 (3.8)	16.7 (3.1)	2.3 (1.3)
Univariate tests		p=.011 (linear trend p=.001)	p=.584	p=.001 (linear trend p<.0001)	p=.838	p=.160
Race/Ethnicity							F(12,327)=2.51, p=.004
Black	27 (22.5)	41.1 (4.2)	17.5 (2.7)	19.1 (3.2)	18.1 (2.7)	4.1 (1.1)
Hispanic	43 (35.8)	42.5 (3.3)	19.7 (2.1)	23.5 (2.5)	12.6 (2.1)	1.8 (0.8)
Other	16 (13.3)	60.5 (5.5)	20.1 (3.5)	4.5 (4.2)	13.3 (3.4)	1.6 (1.4)
White	34 (28.3)	55.5 (3.8)	15.0 (2.4)	19.0 (2.9)	7.5 (2.4)	3.0 (1.0)
Univariate tests		p=.004-Pairwise comparisons-none	p=.493	p=.002-Pairwise comparisons-	p=.042	p=.289

^aLevels of significance for univariate tests are 0.025 (Model 1) and 0.017 (Model 2)

*significant linear trend p=0.001

**significant linear trend p<0.0001

^bNo significant pairwise comparisons

Legend: ASA24=Automated Self-administered 24-hour dietary recall; NDSR=Nutrient Data System for Research; n=number; M=Mean; SE=standard error; p=level of probability; F=statistical F test; %=percent. “Foods matched” were those when the same food was identified in both the ASA24 and NDSR; “foods matched at the category level” were names for foods that were in the same category, but not the same foods; “foods omitted” were those reported on the NDSR, but not ASA24; “foods intruded” were those reported on ASA24, but not NDSR; “foods not matched” were different foods on the ASA24 and NDSR; the denominator for all categories was the sum of the foods reported in NDSR, including the intrusions from ASA24.

Significant results from the ANOVA (Table 1) indicated that the specific food matches had significant age group (p=0.011) and race/ethnicity (p=0.004) main effects (Table 1). A significant (p=0.001) linear trend for age indicated older age was associated with a higher percent of food matches. There were no significant two-way interactions (not shown).

For the MANOVA model with the four dependent disagreement variables, significant multivariate main effects for gender (p=0.022), age group (p=0.031) and race/ethnicity (p=0.004) were found (Table 1). Boys omitted more foods (p=0.026) and had more non-matches (p=0.026). There were significant age group (p=0.001) and race/ethnicity (p=0.002) main effects for the percent of food omissions, and there was a significant race/ethnicity main effect for percent of food intrusions (p=0.042). Older age was significantly linearly associated with a lower percent of food omissions (p<0.0001). Post hoc pairwise comparisons for race ethnicity did not yield significant effects. No two-way interactions were statistically significant (not shown).

Data from observers indicated that a broad diversity of problems occurred at every step in completing ASA24. For virtually each question asked, the eight to nine year olds (boys and girls) were more likely to report finding that aspect of ASA24 “hard” to understand. Overall fifty-seven percent of children reported that ASA24 was easy to use. Based on the observations Table 2 shows recommended enhancement and modifications to ASA24 considered by the observers to be most important for children.

Table 2.

Issues and resolutions associated with ASA24 diet assessment among children ages 8-13 (n=120)

Problem or perceived obstacle	Proposed enhancement or modification
Tutorial frequently ignored	• Replace tutorial with interactive training
Misspelling search terms resulted in search failures	• Use a phonetic spell checker to enhance the search engine
Incomplete wording resulted in no search output	• Use the “auto-fill” feature to complete a word and thus facilitate the search task
Foods searched for are not described in the same manner in the food list database	• Incorporate food tags to ensure an intelligent search (e.g. if “pasta” is the search term, foods such as noodles, spaghetti, ravioli, etc should be displayed as well)
Children deviate from the task to playfully explore the system	• Include validity checks such as “are you sure?” to limit the amount of playful search
Unintentional clicking of buttons without a clear purpose	• Include validity checks to confirm that the action had a legitimate purpose
Not enough fun or entertainment built into the system	• Use multiple avatars
	• Avatar could move around the screen if there is no key stroke in a given time period and say something funny such as: “Hey, I'm a busy avatar, let's get on with it!”
Unable to understand what to do at a given point in time	• Use the avatar as a tour guide
Lack of visual cues create confusion in respondents	• Make use of color or highlights to draw attention to the items of current interest
Between-meal snacking is not considered to be a meal	• Use examples of snacks in training

Although many challenges to dietary recall were reported by the observers and interviewers, matches by food (47.8 percent) were similar with a previously reported (49 percent) value for fruit and vegetables in an earlier (simpler) version of a computer assisted child 24 HDR, which employed observation of school lunch as the criterion measure (7). Matches at the food category, but not food, level have not previously been reported. The 18.1 percent of matches at the food category level, a form of error, suggests some confusion in the child's mind about what they ate, or how to report it. The high percentages of intrusions (12.5 percent) and omissions (19.0 percent) were consistent with previous research (7).

The pattern of results suggests that eight year olds had substantial difficulty in completing ASA24 with nine year olds having only slightly fewer problems, perhaps due to their cognitive immaturity, which was not assessed. Since this research did not assess what the children actually consumed, as might be determined from onsite observation, it is not clear whether the age-related differences applied to ASA24 only, or also to the interviewer-administered 24 HDR. Modifying and simplifying ASA24 according to the suggestions in Table 2 may make it easier for younger children to use.

Boys were somewhat more likely to omit foods or have no match at any level, while girls were marginally more likely to have matches. Some studies have reported greater dietary reporting accuracy by girls (22), but not other studies (23). Research is needed on the conditions under which this difference occurs, and what can be done to minimize it. There has been little reported on differences in accuracy of dietary reporting by racial/ethnic group among children. Such differences could reflect a variety of influences, e.g. attentiveness to food at meals, language or cultural differences in terms used for foods, or other factors.

Many of the probes inquired about details, such as cooking method or type of fat used in preparation, or additions to foods. Children often did not know the answer, causing confusion, frustration and fatigue, even though a “don't know” response that led to a default food code assignment was always available in the system.

This study was designed to identify problems children faced when completing ASA24, and to assess if there was an age below which children had encountered too much difficulty in completing it. A study limitation was not having a third, more objective, assessment of intake. Both recalls used in this study were self-reported using different modes of administration, one of which, the interviewer-administered mode, was assumed to be superior. Future research will benefit from having valid reference data. It may be valuable to include parental assistance in conducting any recall with the eight to eleven year old children (24) as is done in the national surveillance. Differences in food codes/databases between FNDDS and NDSR may also have induced some error.

CONCLUSIONS

Children were able to complete ASA24, but eight and nine year olds generally encountered more problems than older children, and most required help in order to complete the task. Future research will be needed to determine how to adapt and simplify the ASA24 to meet children's abilities and preferences.