Automatic Food Documentation and Volume Computation using Digital Imaging and Electronic Transmission

Introduction

Capturing accurate food intake data from participants enrolled in nutrition studies is essential for understanding relationships between diet and chronic disease (1). Numerous methods are employed to assess dietary intake such as food records, 24-hour recalls, or food frequency questionnaires. While each of these techniques is valuable, the error associated with each is unique. The food record requires a motivated participant, is tedious for some, places attention on the act of eating thus altering intake and is difficult for subjects with low literacy skills (2). Interviewing subjects about the previous day’s intake avoids the reactivity involved when recording current intake, but also requires the individual reporting intake to have good recall skills, knowledge of food names and ability to estimate amounts eaten; and requires a well-trained interviewer which makes this a costly process (2, 3). Food frequency questionnaires are limited by food lists and lack of detail regarding food preparation, and require respondents to summarize past intake over many months or the past year. Such instruments are known to contain significant measurement error (4). While all these methods provide valuable information about dietary intake, improving methodology even modestly would advance our knowledge about the influence of food intake on health.

FIVR (Food Intake Visual and voice Recognizer), a subproject of the Genes, Environment, and Health Initiative from the National Institutes of Health (RFA-CA-07-032 at www.gei.nih.gov/index.asp), is designed to use new digital photographing technology to reduce measurement error associated with a food record. The intent is to create a tool that would both increase accuracy of intake records and reduce the recording burden for respondents. Using a mobile phone with a camera (Figure 1), the participant will photograph foods both before and after eating. In this way initial portion size is recorded as well as portions left uneaten. The photographs would be used to identify both the types and amounts of foods consumed. This paper briefly describes the technology and techniques involved.

Figure 1.

Typical Mobile Phone Interface showing (a). operator instruction screen, (b) menu of activities available and (c) camera poised to record meal.

Creating sufficiently detailed images

Capturing images of meals using a mobile phone presents its own unique challenges. Identifying foods from a picture requires a clear image; the automatic calculation of the amount eaten (volume) requires three or more clear images to be taken by the mobile phone user. Since a single image will not support estimation of food volume, rather 3-dimensional objects must be viewed at more than one angle (5, 6). The three images in Figure 2 are captured from 3 slightly different angles. A calibration object is also required in the images for determination of 3-dimensional size (see Figure 2). The calibration object (fiduciary marker) included in the images in Figure 2 is a card with black and white squares of known size. However, a standard credit card can be used to establish the relationship between size in image pixels and actual size of the object in milliliters. Images are also required before and after the meal is eaten to document the volume of food consumed.

Figure 2.

Three images captured by moving the camera using the FIVR mobile phone system.

Quality of the image hinges on several factors including resolution (roughly indicated by number of pixels per image). Higher resolution (more pixels per image) creates larger files, which makes transferring images slower and more subject to failure, thus testing and refinement of the image details is integral to developing a successful system. Camera focus is critical since the best volumetric estimation is obtained when the three images are in focus and taken with the plate at the same distance from the camera. With fixed focus cameras, the images will be blurred if not taken at the right distance (which is often too great). With auto-focus cameras, the focusing is assured but the distance still must be maintained by the user. Ways to adjust the image to correct for small variations in distance are still being explored.

Capturing intake data in real time

Using a mobile phone to document intake has several advantages. One worth mentioning is the ability to remind the user to provide intake information. The phone can be configured to issue queries about forgotten reports when the lag time between meals is greater than expected. The phone can also issue queries about what was consumed, asking the participant to describe the food intake orally, to be matched with the images when they are evaluated. Voice recognition software will be utilized to automate the food list capture and support queries to clarify food details. Timeliness is critical for improving accuracy of dietary reporting. Within a reasonable time the food scene should be translated into a written list of foods and amounts consumed, and the food list sent to the user for verification.

The final step will be matching the foods consumed to a food composition database. The USDA Food and Nutrient Database for Dietary Studies (FNDDS) (7) is the starting point used by FIVR. This database is used to evaluate the NHANES food intake studies, thus it broadly represents foods consumed in the United States. Each food will then be matched to the database of food names to identify a match. A human interface will resolve conflicts when the automatic program fails to find a match between spoken word and database. One important detail that is lacking in this automated system is the density of the foods. Density is required to convert volumes determined from images to weight, the unit of measure used in most databases. Most foods on the FNDDS provide the two components needed to calculate density (volume and weight), however some entries will require volume determinations to convert volume to weight. Higher level databases such as the USDA MyPyramid Equivalents Database (8) can be incorporated so intake of whole foods (in addition to nutrients) can be evaluated.

Conclusion

Since phone signals are routinely directed through a central service, the phone can be configured to forward all image data to a server managed by the nutrition researcher or service provider thus creating a permanent record available in real-time for monitoring data collection and clarifying ambiguities while the information is fresh in the memory of respondents. Real-time processing of dietary intake data has only been available to researchers in metabolic ward studies, but technologies are now available to support real-time dietary assessment in the field. FIVR will use images to identify food items and estimate food volume, automatically translate food intake to nutrients and other components and classify foods consumed by major groups. FIVR is being developed to improve accuracy in research. However the precision and timeliness of the method will be equally valuable in clinical practice. This technology can lead the way to new services that will advance the study and practice of dietetics.