THE USE OF CO-OCCURRENCE PATTERNS IN SINGLE IMAGE BASED FOOD PORTION ESTIMATION

Abstract

Measuring accurate dietary intake is considered to be an open research problem in the nutrition and health fields. Food portions estimation is a challenging problem as food preparation and consumption process pose large variations on food shapes and appearances. We use geometric model based technique to estimate food portions and further improve estimation accuracy using co-occurrence patterns. We estimate the food portion co-occurrence patterns from food images we collected from dietary studies using the mobile Food Record (mFR) system we developed. Co-occurrence patterns is used as prior knowledge to refine portion estimation results. We show that the portion estimation accuracy has been improved when incorporating the co-occurrence patterns as contextual information.

1. INTRODUCTION

Six of the ten leading causes of death in the United States, including cancer, diabetes, and heart heart disease can be directly linked to diet. Due to the growing concern of chronic diseases and other health problems related to diet, there is a need to develop accurate methods to estimate individual’s food and energy intake. Dietary assessment, the process of determining what someone eats during the course of the day, provides valuable insights for mounting intervention programs for prevention of many of the above chronic diseases. Measuring accurate dietary intake is considered to be an open research in the nutrition and health fields. Developing methods for dietary assessment and evaluation has continued to be a challenging task. Traditional dietary assessment technique, such as dietary record, requires individuals to keep detailed written reports for 3–7 days of all food or drink consumed [1, 2], hence it is a time consuming and tedious process. With smartphone quickly gaining popularity in the recent years, the use of smartphones can provide a unique mechanism for collecting dietary information of users. Mobile dietary assessment systems have been developed such as the Technology Assisted Dietary Assessment (TADA) system [3, 4, 5], FoodLog [6], FoodCam [7], DietCam [8], Tuingle [9], Im2Calories [10] to automatically determine the food types and energy consumed by a user using image analysis techniques. In the mobile dietary assessment system that we developed, we focus on the use of image analysis techniques to automatically analyze and determine the food types and energy consumed by a user [11, 5, 12] from a single food image.

Food volume estimation (also known as portion size estimation or portion estimation) is a challenging problem as food preparation and consumption process can pose large variations in food shape and appearance. To date several images analysis based techniques have been developed such as those based on single view image [13, 14, 15, 12, 10, 16, 17], multiple images [18, 8, 19], video [20], 3D range finding [21] and RGB-D image [22]. We feel that either modifying the mobile device or acquiring multiple images/videos of the eating scene is not desirable for users. We focus on food volume estimation using a single-view food image as shown in [23] which reduces user burden.

Estimating the volume of an object from a single-view image is an ill posed problem. Most of the 3D information has been lost during the projection process from 3D world coordinates onto the 2D camera sensor plane. The use of the priori information is required to estimate the food portions. In [15] food portion estimation was converted into pre-determined serving size classification hence the technique could not be generalized. In [14, 24] pre-defined 3D template matching was used however it required manual tuning hence scaling with many foods became a problem. Food portion estimation using geometric models [12] and the approach based on predicted depth map using a Convolutional Neural Network (CNN) [10, 25] overcame the scaling issue with many foods. We compared the food portion size estimation accuracy using both geometric models and depth images [26]. We showed that geometric model based approach achieved higher accuracy compared to that using high quality depth images obtained by structured light technique [27]. In addition, the quality of depth map obtained using commercial level portable devices lead to even worse performance.

In this paper we use contextual information to further improve food portion estimation using geometric models based approach [12]. We define contextual dietary information as the data that is not directly produced by the visual appearance of an object in the image, but yields information about a user’s diet or can be used for diet planning [28]. Food portion co-occurrence pattern is one type of contextual information. We estimate the patterns from food images we collected for dietary studies. The patterns we estimated provide valuable insights about a user’s eating behavior. Such contextual information can not be determined by examining a single food image alone. In this work we develop a method to model the food portion co-occurrence patterns. We use the co-occurrence models to further refine the portion estimation results. We are able to obtain more accurate estimates of food portion sizes.

2. ESTIMATING FOOD PORTION CO-OCCURRENCE PATTERNS FOR PORTION ESTIMATION REFINEMENT

Volume estimation based on a single-view image is an ill-posed problem and the 3D structure of the scene can not be fully reconstructed. The correct food classification label and segmentation mask in the image alone is insufficient for 3D reconstruction of a food item. The use of geometric models will allow for volume estimation where food label is used to index into a proper class of a food type [12]. In this work we focus on the food classes that have varying shapes and appearances. We use prism model [12] to the food classes as the prism model is designed for food classes with varying shapes and appearances. We have designed a checkerboard pattern fiducial marker to be placed in the eating scene shown in Figure 1. The fiducial marker serves as a reference for both image rectification and food area sizes in world coordinates (in cm²). We designed the fiducial marker to a credit card size for users to conveniently carry. The small size of the fiducial marker causes errors in the rectified image using computer vision techniques [29]. For example, if a food item is placed far away from the fiducial marker in the eating scene, the estimated volume for such food item may be less accurate. To improve the accuracy of portion size estimation, we rely on a user’s eating behavior modeled from food images of dietary studies. By proper modeling and incorporating the food portion co-occurrence patterns into portion estimation, we are able to improve the accuracy of portion estimation. Food portion co-occurrence patterns consist of the distributions of portion sizes and the associated weighting factor. We use Gaussian distributions as they best represent the characteristics of portion sizes distributions. We then refine the food portion estimates based on the models of food portion co-occurrence patterns.

Fig. 1.

Sample food images collected by users using mFR with fiducial markers placed in the scenes.

2.1. Food Portion Estimation Using Prism Model

The prism model is an area-based volume estimation technique based on the assumption that the height is the same for the entire horizontal cross-section of the food item. The 5 × 4 blocks color cherkerboard pattern fiducial marker is used as a reference for corner correspondences and the absolute size in world coordinates. The corners on the checkerboard pattern marker can be estimated using [30]. We obtain the 3 × 3 homography matrix H using Direct Linear Transform (DLT) [29]. Assume I is the original food image (as in Figure 1), the rectified image $\widehat{I}$ can then be obtained by: $\widehat{I}=H^{-1}I$. The segmentation mask S_j associated with food j in the original image can be projected from the pixel coordinates to rectified image coordinates. The area of segmentation mask ${\widehat{S}}_j$ from the rectified image can then be estimated. We assume the height h_j for the entire horizontal cross-section. We use median height as the height of the same food class in our food image dataset. The volume of a food item V_j associated with segmentation mask S_j is then estimated: $V_j={\widehat{S}}_j\times h_j$.

2.2. Food Combination Patterns

Food combination pattern describes the frequencies of various food pairs present in the eating scenes. We use conditional probability of food items appearing in the same eating scene as the food combination patterns [28]. We estimate the food combination patterns from our food images collected from dietary studies. We define c_j,k as the conditional probability of food category j appeared given that food category k is present as:

$$c_{j,k}=\frac{p(j,k)}{p(k)}=p(j|k)$$ (1)

The food combination patterns only indicate whether two food items are likely to present in the same food image, hence it is insufficient to refine portion size estimation. We need to develop a technique to model the food portion co-occurrence patterns to refine portion estimation.

2.3. The Use of Food Portion Co-occurrence Patterns for Portion Estimation Refinement

The food portion co-occurrence patterns can help refining the portion estimates as they represent the insights reflected by the entire food image dataset rather than a single image. For example, if we know that food items j and k (e.g. fries and ketchup) usually appear in the same eating scene and the distribution of food portions j and k, we are able to refine the portion estimates based on such prior knowledge.

We use $x_i^j,x_i^k$ to denote the food portions for food classes j, k estimated from food image with index i, where i ∈ {1, 2, 3, ⋯, N}. N is the size of our food image dataset, and j, k ∈ {1, 2, 3, ⋯, M} where M is the number of the food classes we use. ${\mathcal{C}}_{j,k}$ is a combination pair that represents food items j and k are present in the same image. For the combination ${\mathcal{C}}_{j,k}$, the associated conditional probability is always c_j,k = 1. We denote ${\mathcal{S}}_j$ as the set containing all the combination pairs ${\mathcal{C}}_{j,k}$ exist in food image i. We use 2D Gaussian to model the distributions of portion sizes $\left\{x_i^j,x_i^k\right\}$ from our user food image data:

$$g_{j,k}\left(x_i^j,x_i^k\right)~N\left({\mu }_{j,k},{\sigma }_{j,k}\right)$$ (2)

Similarly, we use 1D Gaussian to model the distribution of portions $x_i^j$ estimated for food class j:

$$g_{j,j}\left(x_i^j,x_i^j\right)=g_j\left(x_i^j\right)~N\left({\mu }_j,{\sigma }_j\right)$$ (3)

where μ_j,k, μ_j are the means and σ_j,k, σ_j are the standard deviations of the food estimates we obtained from our user food image data.

As the frequency of combination ${\mathcal{C}}_{j,k}$ appearing in our user food image data is different, we assign different weighting factors. For example, for the combination ${\mathcal{C}}_{j,k}$ that appears often across in food image dataset, we assign a heavier weight as it contributes more to the refinement of the portion estimates. Otherwise, we assign a lighter weight for combination ${\mathcal{C}}_{j,k}$. Furthermore, as the ${\mathcal{S}}_j$ is different for each food image i, the same ${\mathcal{C}}_{j,k}$ can carry different weight in different food image. We define the weighting factor $w_i^{j,k}$ of the combination ${\mathcal{C}}_{j,k}$ in image i as:

$$w_i^{j,k}=\frac{c_{j,k}}{{\sum }_{\forall {\mathcal{C}}_{j,k}\in {\mathcal{S}}_i}{c}_{j,k}}$$ (4)

Note that for each image i the weighting factor $w_i^{j,k}$ is different depending on the food combinations present. The food portion co-occurrence patterns consist of both the weighting factor $w_i^{j,k}$ and the distribution of the portion size estimates as shown in Equation 2 and 3. To refine the portion estimation results obtained using geometric models, we introduce a cost function in which we weight the probability of portion estimates: $\left(x_i^j,x_i^k\right)$ in the image i based on co-occurrence patterns. The cost function is defined as:

$$f\left(x_i^j\right)=1-\sum _{\forall C_{j,k}\in {\mathcal{S}}_i}{w}_i^{j,k}\cdot g_{j,k}\left(x_i^j,x_i^k\right)$$ (5)

Our goal is to minimize the cost such that the refined portion size best reflect the co-occurrence patterns we estimate from our food images collected in dietary studies. The refined food portion ${\widehat{x}}_i^j$ in the eating scene can then be obtained by:

$${\widehat{x}}_i^j=\underset{x_i^j}{\arg \min }\left\{f\left(x_i^j\right)\right\}$$ (6)

3. EXPERIMENTAL RESULTS

We divide our food image data into testing and training subsets. We tested on a total of 40 food classes. To reduce the errors propagate from the automatic classification and segmentation, we use ground truth food labels and segmentations masks. We use a subset of our food images for testing and leave the rest food images for training. Geometric model-based technique [12] is used to estimate the portion sizes from our food images. The median height of each food class is estimated from the training dataset. We model the food portion co-occurrence patterns based on the training subset for portion estimation refinement.

We refine the portion estimation results using food portion co-occurrence patterns. We compare the refined portion estimation error of each food class obtained using geometric models to the portion size errors without refinement. The errors in portion size estimation in Figure 2 are defined as:

$$\mathit{Error}=\frac{|\mathrm{Estimated}\mathrm{Portion}\mathrm{Size}-\mathrm{Ground}\mathrm{Truth}\mathrm{Portion}\mathrm{Size}|}{\mathrm{Ground}\mathrm{Truth}\mathrm{Portion}\mathrm{Size}}$$ (7)

The ground truth portion sizes for each food item are provided by nutrient professionals.

Fig. 2.

Average original errors vs. refined errors for portion estimates by food class.

The average errors per food class are obtained based on average error of 20 trials. In each trial we randomly sample 5% of our food images as testing subset. We use sampling with replacement technique so that the sizes of the training and testing subsets are the same for each trial. The original error in Figure 2 is the error of food portion estimates obtained using geometric model based [12] approach where the refined error is obtained by incorporating food portion co-occurrence patterns. For most food classes, we are able to improve portion estimation accuracy significantly using refinement technique, such as turkey meal. For some food classes the refinement technique was not sufficient to improve the estimation accuracy significantly (such as grapes). For a few food classes (garlic bread and rice krispy bar) the portion estimates become less accurate with refinement. This is due to the co-occurrence patterns we estimated from training dataset do not generalize well for these specific food classes. Such issue can be addressed by increasing the size of the food image dataset collected from future dietary studies as refinement is fundamentally adding biasness to our system based on past observations. If the past observations include variety of scenarios for most user behavior patterns, we can further improve the estimation accuracy. Our geometric model based portion estimation technique becomes less sensitive to noise by incorporating co-occurrence patterns. It has been shown in Figure 2 that the co-occurrence patterns we estimated generalize well for most of the food classes in our dietary studies. We define the improvement rate as:

$$Improvement=\frac{\mathrm{Original}\mathrm{Error}-\mathrm{Refined}\mathrm{Error}}{\mathrm{Ground}\mathrm{Truth}\mathrm{Portion}\mathrm{Size}}$$ (8)

The overall improvement rate for our dataset is 36.9%.

4. CONCLUSION AND FUTURE WORK

In this work we model the food portion co-occurrence patterns based on the food images we collected in dietary studies. The food portion estimation is refined by incorporating the portion co-occurrence patterns. We have shown that with the refinement we significantly improve the estimation accuracy for most of the food classes. In the future we are interested in developing predictive models that extend to food classes that we do not currently have sufficient knowledge on their co-occurrence patterns.