Introduction
Dietary reporting by individuals is subject to error (1–3). Therefore, a research program has been initiated to develop a small electronic device to record food intake automatically. This device, which contains a miniature camera, a microphone, and several other sensors, can be worn on a lanyard around the neck. It collects visual data immediately in front of the participant and stores them on a memory card in the device. The data are transferred regularly to the dietitian’s computer for further processing and analysis. The device is designed to be almost completely passive to the participant, and thus hopefully will not intrude on or alter the participant’s eating activities. In addition to this function, in the future the device will have other functions, such as the measurement of physical activity, human behavior, and environmental exposure (e.g., pollutants).
Hardware Design
The electronic system contains two major components, a unit worn by the participant and a data analysis software package installed in a computer at the dietitian’s office. The prototype of the wearable unit is shown in Fig. 1. The circular device presently looks like a wearable MP3 player, suitable for both men and women. The appearance of the device may be tailored according to individual’s preference. Currently, the device has a diameter of 62 mm, which will be reduced substantially in the future. A large-capacity (2700 mAh), rechargeable, lithium-ion battery (rectangular object in Fig. 1) is placed in back of the neck connected to the circular device using adjustable cables. Two separate smaller devices, connected to the front device by the same cables, can reach locations on the upper body and head. These devices can be earphones, microphones, accelerometers, or skin-surface electrodes for various applications, such as user interface and physiological measurements. The front device contains a number of sensors and data processing/storage components, including a miniature camera for video recording, reference lights for food dimension measurement in the visual field, an accelerometer for physical activity monitoring, a microphone for the identification of eating episodes using ambient sound, a Global Positioning System (GPS) for location identification, a central processor for data processing, and an 8 GB MicroSDHC flash memory for storing data. All electronic components are installed on an eight-layer printed circuit board. Currently, the hardware design has been completed, and the embedded software (routines that provide interface to hardware modules) is being developed.
Figure 1.
Prototype of wearable electronic device
Software Design
The recorded data in the flash memory are readily transferred to the dietitian’s computer during regular office visits or by Internet uploading. The software package to process data, aiming at minimizing dietitian’s workload and maximizing data processing accuracy, is currently being developed. The functions of the software are as follows.
Privacy Protection
The data in the flash memory card are intentionally scrambled for security protection. Once the data are received by the dietitian’s computer, they are disassembled into individual types, and the video portion of the data is automatically scanned for human faces which are then blurred to prevent identification. After these automatic procedures, the “de-identified” data are converted to standard image formats that can be observed by the dietitian. For privacy protection, the audio data cannot be replayed by the dietitian. They are only used internally by the computer to perform automatic video segmentation
Video Segmentation
The first data processing step is the identification of the eating episodes in the video. This is performed by software, based on the following features computed from the available records: motion patterns, presence of circular food plates, similarities of contents in the video, recorded audio signal of surrounding sound, and time of the day. The computer-identified episodes are presented on the computer screen as icons. When an icon is clicked, the corresponding eating episode is replayed with the replay rate controlled by the dietitian.
Food Identification
For each selected eating episode, the dietitian views the image sequence within the episode to determine the food items being consumed. When necessary, the dietitian browses both the video and sensor data for additional food information, such as the cooking procedure and the location of a restaurant based on recorded GPS coordinates. In certain cases, some foods may still be unknown. In these cases, once all data collection has been completed, the participant will be asked to identify such foods in the images.
Portion Size Analysis
Although digital pictures provide an excellent tool to identify food, these pictures, by themselves, do not provide sufficient information to estimate portion size (i.e., dimensions), unless a dimensional referent is present in the field of view. Accordingly, several approaches have been studied to provide such a dimensional referent.
Before a participant begins to wear the device in a study, he/she will be asked to take pictures of their commonly-used plates and measure their diameters. In addition, the participant will be provided with a checkered tablecloth for their dining table. When eating episodes occur in the participant’s home, food portion size can be determined computationally when one of these referents is present in food pictures. In cases where food consumption occurs outside of the home, lights on the device will be used to project a dimensional referent into the field of view. Presently, two types of light patterns are being evaluated. The first consists of three small beams produced by laser diodes on the device that form an equilateral triangle in the visual field; the second is a light-emitting diode (LED) on the device that projects a circular spotlight into the field of view (Fig. 2). Each of these light patterns produces a dimensional referent that permits calculation of food portion size.
Figure 2.
A food image with a spotlight projected by the device
Mathematical Model of Portion Size Measurement
In each approach, a mathematical camera model, called the pin-hole model (4), relates each point in the digital image (called a pixel) to a corresponding point in the real-world space. The circular plate, square checkerboard, equilateral triangle formed by the three laser beams, or circular spotlight produced by the LED usually has a distorted shape when observed in the image. The degree of distortion can be evaluated by an algorithm to obtain information about object orientation and distance, based on a set of known camera parameters (5–7). For example, as shown in Fig. 2, the size of the food and beverage can be determined using the referent provided by the light image. Currently, these approaches are being evaluated and optimized.
Determination of Nutrients and Calories
Once the food and portion size are determined, the system uses a food database, such as the United States Department of Agriculture Food and Nutrient Database for Dietary Studies (8), to convert food volumes to values for calories and nutrients. The information obtained is accumulated over the entire period of study to produce a comprehensive dietary report.
Conclusions
Recent technological advances in microelectronics offer unique opportunities to develop electronic devices and methods for objective dietary assessment. Our group is applying some of these advances to design and evaluate new systems to minimize intrusion into the participant’s life. Although many of these approaches are still under development, we believe that, with further technological development, it will become possible to conduct accurate and objective dietary assessments.
Acknowledgments
This work was supported by National Institutes of Health grant U01 HL91736. The authors are thankful to Amy Subar, Catherine Loria, and Phyllis Stumbo for their valuable suggestions.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain
Contributor Information
Mingui Sun, Email: drsun@pitt.edu, Departments of Neurosurgery, Electrical & Computer Engineering, and Bioengineering, University of Pittsburgh, 3520 Forbes Ave., Suite 202, Pittsburgh, PA 15213; Phone: 412-802-6481; Fax: 412-802-6785.
John D. Fernstrom, Email: fernstromjd@upmc.edu, Departments of Psychiatry and Pharmacology, University of Pittsburgh, Research Director, UPMC Weight Management Center, Rm. 1620 WPIC, 3811 O’Hara St., Pittsburgh, PA 15213, Phone: 412-246-5297; Fax: 412-246-6470.
Wenyan Jia, Email: wej6@pitt.edu, Department of Neurosurgery, University of Pittsburgh, 3520 Forbes Ave., Suite 202, Pittsburgh, PA 15213; Phone: 412-802-6475; Fax: 412-802-6785.
Steven A. Hackworth, Email: stevenahackworth@gmail.com, Department of Electrical & Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15260; Phone: 412-802-6481; Fax: 412-802-6785.
Ning Yao, Email: aniesyao@gmail.com, Department of Electrical & Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15260; Phone: 412-802-6481; Fax: 412-802-6785.
Yuecheng Li, Email: yclee0202@gmail.com, 1Department of Neurosurgery, University of Pittsburgh, Pittsburgh, PA 15260; 2Image Processing Center, Beihang University, Beijing, China 100083; Phone: 412-802-6481; Fax: 412-802-6785.
Chengliu Li, Email: chengliu.li@gmail.com, Department of Electrical & Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15260; Phone: 412-802-6481; Fax: 412-802-6785.
Madelyn H. Fernstrom, Email: fernstrommh@upmc.edu, Departments of Psychiatry and Surgery, University of Pittsburgh, Director, UPMC Weight Management Center, Rm. 1615 WPIC, 3811 O’Hara St., Pittsburgh, PA 15213, Phone: 412-246-5297; Fax: 412-246-6470.
Robert J. Sclabassi, Email: bobs@cdi.com, Departments of Neurosurgery, Electrical & Computer Engineering, and Bioenginering, University of Pittsburgh, 3520 Forbes Ave., Suite 202, Pittsburgh, PA 15213; Phone: 412-802-6473; Fax: 412-802-6785.
References
- 1.Trabulsi J, Schoeller DA. Evaluation of dietary assessment instruments against doubly labeled water, a biomarker of habitual energy intake. Am J Physiol. 2001;281:E891–E899. doi: 10.1152/ajpendo.2001.281.5.E891. [DOI] [PubMed] [Google Scholar]
- 2.Goris AH, Westerterp-Plantenga MS, Westerterp KR. Undereating and underrecording of habitual food intake in obese men: selective underreporting of fat intake. Am J Clin Nutr. 2000;71:130–134. doi: 10.1093/ajcn/71.1.130. [DOI] [PubMed] [Google Scholar]
- 3.Goris AH, Westerterp KR. Underreporting of habitual food intake is explained by undereating in highly motivated lean women. J Nutr. 1999;129:878–882. doi: 10.1093/jn/129.4.878. [DOI] [PubMed] [Google Scholar]
- 4.Hartley R, Zisserman A. Multiple view geometry in computer vision. 2. Cambridge, UK: Cambridge University Press; 2003. [Google Scholar]
- 5.Sun M, Liu Q, Schmidt K, Yang J, Yao N, Fernstrom JD, Fernstrom MH, DeLany JP, Sclabassi RJ. Determination of food portion size by image processing. Proc. IEEE Engineering in Medicine and Biology; August 20–25, 2008; Vancouver, Canada.. [DOI] [PubMed] [Google Scholar]
- 6.Yao N, Sclabassi RJ, Zhao R, Zhang H, Sun M. A laser based depth measurement method for digital imaging of close-up objects. Proc. 16th International Conference on Mechanics in Medicine and Biology; July 22–25, 2008; Pittsburgh, PA, USA. [Google Scholar]
- 7.Jia W, Zhao R, Fernstrom JD, Fernstrom MH, Sclabassi RJ, Sun M. A food portion size measurement system for image-based dietary assessment. Proc. 35th Northeast Biomedical Engineering Conference; April 1–4, 2009; Cambridge, MA, USA.. [Google Scholar]
- 8.USDA Food and Nutrient Database for Dietary Studies. Beltsville, MD: Beltsville Human Nutrition Research Center, Agricultural Research Service, US Department of Agriculture; 2004. [Google Scholar]