Evidence-based development of a mobile telephone food record

Abstract

Mobile telephones with an integrated camera can provide a unique mechanism for collecting dietary information that reduces burden on record keepers. Objectives for this study were: (1) to test whether participant's proficiency with the mobile telephone food record (mpFR) improved after training and repeated use, and (2) to measure changes in perceptions regarding use of the mpFR after training and repeated use. Seventy-eight adolescents (26 males, 52 females) ages 11–18 y were recruited to use the mpFR for one or two meals. Proficiency with the mpFR was defined as capturing a useful image for image analysis and self-reported ease of use. Positive changes in perceptions regarding use of the mpFR were assumed to equate to potentially improved proficiency with the mpFR. Participants received instruction for using the mpFR prior to their first meal, and captured an image of their meals before and after eating. Following the first meal, participants took part in an interactive session where they received additional training on capturing images in various snacking situations and responded to questions about user preferences. Changes in the participants' abilities to capture useful images and perceptions about the usability of the mpFR were examined using McNemar, Wilcoxon rank-sum test, and paired t-test. After using the mpFR, the majority of participants (79%) agreed that the software was easy to use. Eleven percent of participants agreed taking images before snacking would be easy. After additional training, the percent increased significantly to 32% (p<.0001). For taking images after snacking, there was also improvement (21% before training and 43% after, p<.0001). Adolescents readily adopt new technologies; however the mpFR design needs to accommodate the lifestyles of its users to ensure useful images and continuous use. Further, these results suggest that additional training in using a new technology may improve the accuracy among users.

Introduction

Dietary intake provides some of the most valuable insights for mounting intervention programs for the prevention of chronic diseases. However, accurate assessment of diet is problematic, especially in adolescents (1). Emerging technology in mobile telephones with higher resolution pictures, improved memory capacity, and faster processors allow these devices to process information not previously possible. Mobile telephones are widely used throughout the world and can provide a unique mechanism for collecting dietary information that reduces burden on record keepers. A dietary assessment application on a mobile telephone would be of value to practicing dietitians and researchers.

Previous results among adolescents showed that dietary assessment methods using technology, e.g., a Personal Digital Assistant with or without a camera or a disposable camera, were preferred over the traditional paper food record (2). This suggests that for adolescents, dietary methods that incorporate technology may improve cooperation and accuracy. Additionally, adolescents have expressed difficulty describing and estimating portions of foods and beverages consumed (2). Even with the use of portion size estimation aids, children's estimates are considered a source of error in quantifying food and energy intake (3–5). The development of a mobile telephone application for dietary assessment that fits into the lifestyle of young people may address many of the barriers identified above (6;7).

Most technology development emphasizes the requirements of the system as determined by the engineer or software programmer rather than the interaction of the user with the system (8;9). The study described here is part of the evidence-based development of a mobile telephone Food Record (mpFR) for adolescents (2;7;10). Evidence-based development of technology is a novel idea intended to focus on the effects of the system on its users and their lifestyles (9;11). Using this process will allow the design of the mpFR from the perspective of the user or the client rather than the perspective of the engineer or dietitian.

A form of evidence-based development is interaction design which is the discipline of defining the behavior of products that a user can interact with. This process of developing interactive products supports the way people communicate and interact in their everyday and working lives. The goals of interaction design are to develop usable products, in this case the mpFR, that are easy to learn, effective to use and provide an enjoyable experience. This is accomplished by involving users in the design process and receiving feedback about the product. Therefore, it is an iterative cycle of usability testing in which the user feedback is applied to the next version of the mpFR and tested again.

The objectives of this study informed the next development phase of the mpFR. The two objectives were: (1) to test whether participants' proficiency with the mpFR improved after training and repeated use, and (2) to measure changes in perceptions regarding use of the mpFR after training and repeated use. Proficiency with the mpFR was defined as capturing a useful image for image analysis and self-reported ease of use. Further, positive changes in perceptions regarding use of the mpFR were assumed to equate to potentially improved proficiency with the mpFR.

Methods

System Design

The client-server configuration for the mpFR is in Figure 1. In this system, a mobile telephone with an integrated camera can be used to capture images of food before and after eating and sent to the server (Figure 1, #1). A digital image is different from a photograph in that useful information, called metadata, is captured that is not visible, such as the time stamp and digital codes. Image analysis uses metadata to automatically identify characteristic features in food, such as color, texture, and intensity that are then used to identify a food. Methods for automatic identification of food using image analysis (Figure 1, #2) have been previously described (7;12).

Figure 1.

Current client server configuration of the mobile telephone food record (mpFR). 1: User acquires an image of the food before and after eating with the mpFR. The image along with metadata (time stamp and digital code) are sent to the server. 2–3: Image analysis occurs at the server. The food item(s) are identified (labeled) and the volume of each food estimated. 4: The results are sent to the user. 5: The user confirms and/or adjusts the information and sends it back to the server. 6: The data are indexed with a nutrient database, the Food and Nutrient Database for Dietary Surveys (FNDDS), to compute the nutrient content of the foods consumed. 7: The results are sent to researchers or dietitians.

An object of known dimensions and markings, referred to as a fiducial marker, must be included in the image as a size reference. The inclusion of a fiducial marker in the image (e.g., the checkerboard square in Figures 2A and 2B) aids in the reconstruction of a 3-dimensional environment which allows for the estimation of the volume of the foods and beverages (Figure 1, #3). The image is then sent back to the user (Figure 1, #4) for the user to confirm that the foods are identified correctly (Figure 1, #5). Together the information from image analysis and volume estimation can be linked to a nutrient database (Figure 1, #6) to estimate energy and nutrients consumed (10). In the final step, the nutrient analysis can be sent to researchers or dietitians.

Study Design and Recruitment

Data were collected from two samples of participants. For Sample 1 and Sample 2, the study methods were approved by the Blinded University Institutional Review Board, and informed assent and consent were obtained from the participants and their parents, respectively.

Sample 1

The first sample was drawn from summer camps for adolescents, ages 11–18 years, conducted by a variety of university departments and taking place on the university campus. Participants from these camps participated in one lunch (n=63) and 55 (87%, 55/63) returned for breakfast the next morning. Following lunch (i.e., their first meal after using the mpFR) all of the participants took part in an interactive session.

Sample 2

The second sample was a convenience sample drawn from the local community (n=15). Recruitment was limited to those individuals between 11–18 years. Participants received all meals and snacks for a 24-hour period while being monitored under controlled conditions. These participants took part in the interactive session after breakfast (i.e., their first meal after using the mpFR); and data from their first and second meals are included in this analysis.

Eating Occasions

A systematic method was used to design the menus using the most common foods reported during two 24-hour recalls administered among 162 adolescents between 10–18 years (13;14), as well as foods from local school lunch menus. For each meal, foods and beverages were weighed, plated, and arranged using a predetermined place setting (See Figure 2A). Three menus for breakfasts and three menus for lunches were cycled between the sessions. All participants received instruction for using the mpFR prior to their first meal. In order to obtain an image useful for image analysis, participants were asked to include two items in each image: 1) all food and beverages, and 2) the fiducial marker. Participants ate to satiation and they were served seconds, if requested. The procedures for capturing images were then repeated for any additional portions.

HTC p4351 mobile telephones running Windows Mobile 6.0 were used. The software described previously (7;10) guided the user to select the meal occasion and capture an image of foods and beverages (See Figure 2B). After capturing the image, the user was prompted to review the image and then given a choice to retake the image or save the image. Once the user was satisfied with the image, the mobile telephone prompted the user to eat before proceeding to the next screen. At the next screen, the user was prompted to take an image of the place setting regardless of whether food and beverages remained. The final screen showed the before and after images prior to exiting the program. The participants were assisted during the meal by trained dietetic students and for Sample 1 only, staff recorded the number of images taken by each participant, as well as position (sitting/standing) while taking an image.

Interactive Session

Eleven forced-choice questions with appropriate visuals regarding the interaction design of mpFR were shown using PowerPoint®. Response choices ranged from `strongly agree' to `strongly disagree' and responses were collected with the eInstruction™ Classroom Performance System (CPS) (Cincinnati, Ohio). Three open-ended questions were presented: 1) If the program, cannot identify a food, how would you want the program to tell you that?; 2) If you need to then label the food with the correct name, how would you want to do that?; and 3) Would you want to narrate into the telephone out loud the names of the food?. The oral responses from the participants were recorded by staff. The participants received additional training through activities where they practiced capturing images in potentially problematic snacking scenarios, e.g., on the school bus, in the movie theater.

Data Analysis

For analysis, participants were separated into early and late adolescent age groups to address cognitive development: 11–14 years old (n=44) and 15–18 years old (n=33). Descriptive analysis included frequencies and percents. Differences in responses by categorical characteristics, i.e., age group and gender, were examined using chi-square, and ANOVA for quantitative variables. Differences within participants were examined using McNemar (categorical variables), Wilcoxon rank-sum test (ordinal variables) or paired t-test (quantitative variables). The qualitative responses to the open-ended questions were examined using content analysis (15).

Results and Discussion

A total of 78 participants (26 males, 52 females) ages 11–18 years used the mpFR for a first meal and 70 of those also used the mpFR again for a second meal. Characteristics of the samples are in Table 1. The participants were either in middle school or high school, and the mean age was 14.2 years. The participants were of diverse ethnic backgrounds.

Table 1.

Characteristics of volunteers testing the usability of the mobile telephone food record (n=78)

Characteristic	n (%)1
Gender
Male	26 (33)
Female	52 (67)
Age, years
11–14	45 (58)
15–18	33 (42)
Grade in school
Middle school (6–8th grade)	30 (50)
High school (9–12th grade)	30 (50)
Ethnicity
Asian	1 (1)
Hispanic	7 (9)
Non-Hispanic white	55 (70)
Black	10 (13)
Multiple	5 (6)

¹Missing values due to non-response.

With regard to being able to acquire a useful image, the majority of the participants saved images that included all of the foods and beverages in both the before and after images for the first meal (56/70, 80%) and the second meal (59/70, 84%). A smaller proportion included the entire fiducial marker in both the before and after images for the first meal (50/70, 71%) and slightly more for the second meal (54/70, 77%). After repeated use of the mpFR, there was no significant change in the number of participants who included the entire fiducial marker in their images. There were numerous items (e.g., condiments, beverages) to include in each image (See Figure 2A). Participant comments during the meal sessions suggest that it was a challenge to include all items in the image. Height of the adolescents may also have contributed to the challenge of including all items in each image. In one instance, a short statured participant stood on a chair in order to capture the best image. A possible solution is to ask the user to take multiple images if they cannot capture all food items in a single image, or push the tableware closer together. The individuals that consumed all foods and beverages recommended the addition of an option, “ate all food and beverages”; rather than capturing an image of completely empty plates and glasses.

Repeated use allowed participants to familiarize themselves with the mpFR. Just under half of Sample 1 took more than 1 image before the first meal (21/50, 42%), however this declined significantly (p=.033) to 22% (11/50) after the second meal. In all cases, the likelihood of taking one image was positively associated with standing rather than sitting to take an image. A small fraction of participants changed the orientation of the camera between the before and after meals, i.e., portrait versus landscape. This indicated the need for a prompt to take images in landscape providing consistency for image analysis.

Responses to the questions regarding use of the mpFR are shown in Table 2. A larger proportion of the participants agreed it would be easy to use a credit card sized fiducial marker as opposed to using a USB flash drive sized fiducial marker. Although a fiducial marker sized like a USB flash drive would be smaller, the credit card sized fiducial marker might be more convenient since it would fit in a wallet. Most adolescents have standard sized school identification cards, thus a credit card sized fiducial marker may be easier to incorporate into their current lifestyles.

Table 2.

Responses from adolescents (11–18 y) after using a mobile telephone food record (n=78)

Statements, as presented	Agree1 n (%)	Neutral n (%)	Disagree n (%)	Mean response2
I think it would be easy to carry and use:
A credit card sized fiducial marker	55 (78)	10 (14)	6 (9)	2.0
A USB flash drive sized fiducial marker	30 (42)	19 (27)	22 (31)	2.8
The software was easy to use	55 (79)	9 (13)	6 (9)	1.9
I think it would be easy to remember to take an image (before additional training):
Before meals	26 (37)	22 (31)	22 (31)	2.9
After meals	29 (41)	27 (38)	15 (21)	2.8
Before snacks3	8 (11)	16 (23)	46 (66)	3.7
After snacks4	15 (21)	19 (27)	37 (52)	3.5
I think it would be easy to remember to take an image (after additional training):
Before snacks3	22 (32)	18 (26)	29 (42)	3.1
After snacks4	30 (43)	16 (23)	24 (34)	2.9
I prefer to sit while taking an image	25 (36)	21 (30)	23 (33)	2.9
I prefer to stand while taking an image	43 (63)	14 (21)	11 (16)	2.3

¹Missing values due to non-response. Percents do not always equal 100 due to rounding.

²Responses were coded as strongly agree=1, agree=2, neither agree nor disagree=3, disagree=4, strongly disagree=5.

³Paired sample analysis was performed comparing before snacks responses before and after training. P-value <.0001

⁴Paired sample analysis was performed comparing after snacks responses before and after training. P-value <.0001

The majority of participants agreed that the software was easy to use. Just over one third of the participants agreed that it would be easy to remember to take images before meals and a slightly higher proportion agreed that it would be easy to remember to take images after meals (See Table 2). Despite the ease of using the software, the perception of being less likely to remember to take an image before eating versus after eating is consistent with individuals being hungry and eager to eat. Responses before additional training reflected the lowest proportion of individuals indicating that taking an image before a snack would be easy to remember (See Table 2). Following additional training in which the participants practiced taking snack images in awkward situations, significantly more participants agreed that taking images before (t=3.78, df=67, p =<.0001) and after snacking (t=3.89, df=69, p <.0001) would be easy.

Adolescents are assumed to have a high level of technology readiness and most of the adolescents were participating in technology themed camps. The results support that improved use of the mpFR can be achieved with additional training activities and interaction design changes. Specifically, alterations were initiated to ensure speedier capture of images of foods and beverages prior to eating.

Over half of the participants agreed that they preferred to stand while taking their images. Many participants found it difficult to capture all food items and the fiducial marker while sitting. These results in a controlled environment need to be confirmed in a variety of free-living situations. Adolescents may not readily stand to take images of their meals in more realistic situations.

Overall, there were no significant differences in the respondents' perceptions of the mpFR and their proficiency with the mpFR by age group or gender. This suggests that the interaction design for the mpFR would be the same across the adolescent age span.

The open-ended questions addressed the interface issues of the automatic identification of the foods through image analysis (Figure 1, #2) and the subsequent confirmation with the user (Figure 1, #5). The participants stated their suggestions for notification to confirm the food and beverage items. Most preferred that a pop-up message appear on the screen if additional information was needed. Other ideas included the appearance of an “X” or an arrow on or by the food in question. Also, several suggested having a list appear or a space; each of which could be edited with the correct information. As a means of identifying a misidentified item, the majority preferred filling in a text box or selecting from a list of possible options. There was a predominance of negative responses for narrating the food identification into the mobile telephone. Participants noted social awkwardness (e.g., “weird”, “creepy”, “they will think you are an idiot”) and under development of voice recognition software (e.g., “voice recognition is too iffy”, “Voice recognition doesn't work. You say `call Bob,' and it doesn't even know who to call.”). These suggestions informed the next phase of the development of the mpFR.

Conclusions

Evidence-based development entails a shift from focusing on the product to focusing on the process and the interaction of the user with the product. Using this process will allow the design of the mpFR from the perspective of the user or the client rather than the perspective of the engineer or dietitian. Formative evaluation outcomes and user feedback are important components of evidence-based development used when designing applications (11). In the same way that evidence based guidelines help dietitians perform more effectively, we anticipate that developing a product that users can comfortably use throughout the day will lead to cooperation that will translate to an accurate and less burdensome record of dietary intake.

Results from this study suggest that additional training in using this new technology may improve user cooperation. Adolescents' proficiency for capturing all foods, beverages, and the fiducial marker with the mpFR improved after the first use. Adolescents readily adopt new technologies; however the design of the mpFR needs to accommodate the lifestyles of its users to ensure useful images and continuous use throughout the day or multiple days. The results from this study will be reflected in future versions of the mpFR, thus providing an accurate, inexpensive, easy-to-use method for dietary assessment in dietetic practice and research.