Automatic identification of the number of food items in a meal using clustering techniques based on the monitoring of swallowing and chewing

Abstract

The number of distinct foods consumed in a meal is of significant clinical concern in the study of obesity and other eating disorders. This paper proposes the use of information contained in chewing and swallowing sequences for meal segmentation by food types. Data collected from experiments of 17 volunteers were analyzed using two different clustering techniques. First, an unsupervised clustering technique, Affinity Propagation (AP), was used to automatically identify the number of segments within a meal. Second, performance of the unsupervised AP method was compared to a supervised learning approach based on Agglomerative Hierarchical Clustering (AHC). While the AP method was able to obtain 90% accuracy in predicting the number of food items, the AHC achieved an accuracy >95%. Experimental results suggest that the proposed models of automatic meal segmentation may be utilized as part of an integral application for objective Monitoring of Ingestive Behavior in free living conditions.

1. Introduction

Dietary self-report methods commonly used for Monitoring of Ingestive Behavior (MIB) like observation, weighed food records, estimated records, diet history food-frequency questionnaires, food recall methods and others have been used intensively in research. However, the majority of them suffer from subject’s underreporting (ratio of estimate over actual intake) down to 0.84 [1]. The reason for subject’s underreporting related to these methods is mainly due to two main factors: the change in eating behavior that occurs when subjects are asked to record their intake (observation effect), and the subject’s tendency to misreport their changed eating behavior (reporting effect) [2–4]. Observation effect can lead to an overall decrease of 5% in intake estimation accuracy, while reporting effect is much more variable and results in decreases ranging from 5% to 20%, depending on the intake measurement method used [4].

Additionally, underreporting does not only result from a systematical underestimation of portion sizes for all food items, but is also related to avoiding the reporting of certain intake behaviors like snacking [5,6] and consumption of some specific food items [7]. Overall, there is a current need for developing more objective means for MIB that eliminate the burden of subject’s active participation in reporting of not only the food size, but also the food periods during the day and different types of food items ingested [7].

In previous studies, we developed a sensor system for non-invasive monitoring of chewing and swallowing and validated the reliability of the produced manual scores [8–10], established a methodology of automatic detection of swallowing instances by acoustical means [11], and proposed and validated several methods for detection and characterization of food intake based on chewing and swallowing [10,12]. In this study, we propose a methodology to identify the number of distinct food items within a meal using information extracted from deglutition and mastication patterns. Our methodology is a follow-up on our ongoing research that comprises different stages for automatic objective MIB.

Successful identification of food items not only simplifies understanding of eating behaviors associated with obesity and other eating disorders, but could also be very helpful when monitoring patients with certain diseases. For example, the rate of relapse in patients with anorexia nervosa is estimated to be as high as 50% [13]. One study showed that the selection of a diet with low energy density and limited variety of food items was associated with a poor treatment outcome [14]. Also, a controlled diet consisting in very-low calorie food items in obese type 2 subjects promotes long-term glycemic control [15]. This study showed that such type of diet produced greater decreases in fasting glucose, with the additional benefit of weight loss. Development of an objective method for monitoring of variety of food items consumed by subjects with diseases like the ones described could potentially lead specialists to a better clinical treatment and recovery control.

Several methodologies have been proposed, urging the need of automatic objective methods for MIB and classification of food items consumed. A study proposed the use of a semi-automatic method to classify the food items from recorded shopping receipts and keep track of their nutritional content to generate suggestions about healthier food items that could help to supplement missing nutrients [16]. However, this method relies in subject’s active participation to swipe receipts into a handheld scanner. Also, the shopping suggestions of the system are based on purchases and not necessarily consumed foods. Another study proposed the use of a dining table instrumented with weight sensors and radio frequency identification (RFID) of food containers to detect and distinguish how and what people eat [17]. While this sensor device is still under development, several limitations were mentioned by the authors when reporting their results: external food sources are not allowed different than those contained in RFID containers; weight, nutrition, ingredients of foods, as well as weight and owners of food containers are known a priori; leaning hands and elbows over the table is avoided. These limitations make this device at the moment impractical for free-living conditions. A different approach was used in another study where multi-sensor based monitoring is used to identify intake gestures, chewing and swallowing to provide timing and food category information [18]. This method successfully discriminates different events by the gesture behavior such as eating with fork and knife, drink from a glass, eat with a spoon, or the use of hands to approach food to the mouth. The motion sensor jacket developed was a research prototype and less complex sensors are being developed. Recently, the use of mobile phones has been proposed in which subjects capture images of a meal being consumed to be transferred for further analysis [19,20]. As with the food monitor using shopping receipts [16], this type of monitoring still presents a burden to the subject under evaluation that might change an ad libitum eating behavior. A wearable device almost completely passive to the subject and still under development is proposed in [21] with a camera collecting images of the meal and a dietician later evaluating the image sequence to determine the food items.

The methodology we propose here is based on the use of features extracted from the monitoring of swallowing and chewing to segment a meal into the different food items it consists of. We believe that monitoring of chews and swallows over time provide a useful measure for food intake pattern identification of different categories of food since both activities are directly related to the food intake process in healthy humans.

2. Methods

2.1. Data collection

The data for the development of the proposed models described in this work were collected in a study reported in previous published work [8]. Data from 17 human subjects (11 males, 6 females) were used. The average body mass index across the sample population was equal to 28.81 ± 6.61 kg/m², age of subjects varied from 18 to 48 years old. The subjects reported no dental problems that would deficit their normal masticatory ability to process food. An attempt to recruit a diverse population was made in terms of gender, ethnicity, age, and body size. Institutional Review Board approval was obtained for this study. Subjects read and signed the informed consent form document. Each subject performed from two to four meal experiments on separate days. In each experiment, subjects were asked to eat a meal consisting of a slice of cheese pizza, a can of 1% fat yogurt, an apple, a peanut butter sandwich (PB) and a cup of water consumed in order. The selection of foods represented different food properties that may impact chewing and swallowing, such as crispiness, moistness, softness/hardness and tackiness, and focused the experiment on inter-subject variability of chewing and swallowing pattern rather than inter-food variability. In half of the meal experiments, subjects were asked to remain silent, and in the other half they were involved into conversation with the experiment’s operator.

During a meal, the subjects were monitored by a number of sensors capturing and characterizing the process of chewing and swallowing. Chewing was monitored by a strain sensor located in the area immediately below the ear. Swallowing was monitored by a microphone located over the laryngopharynx. In addition to these sensors the subjects were also videotaped in profile. Time-synchronous recordings from the sensors and video were reviewed in specially designed software and used for manual scoring process [8].

The produced manual scores contain timetable and duration of each swallow and chew as well as the number of chews in each mastication period [8]. Manual scores of the swallows and chews provided the initial data for the meal segmentation models. In total, 66 meals were scored, with around 17.5 h of meal recordings containing 5634 swallows and 44,038 chews.

2.2. Features extraction

Information from swallowing and chewing scores was analyzed to extract features which were used for meal segmentation into different food items. Three types of features were defined that characterize the timing and frequency of chewing and swallowing. The first feature analyzed was the location in time t_i of each swallow for i = 2, …, k total number of swallows in a particular subject’s meal. This feature was obtained as the center point between the beginning and end, defined by the manual scores, of the ith swallow. The underlying assumption for using this feature is that most individuals consume foods bite by bite at a time during a meal, thus making occurrence time of each swallow contained in each bite an important feature in grouping the swallows related to a certain food.

The second feature was the Time to Preceding Swallow for the ith swallow expressed in seconds and defined as TPS_i = t_i − t_i−1, which is the difference in occurrence time between a swallow located at moment t_i with respect to the previous swallow located at moment t_i−1. The third feature was the number of chews between two consecutive swallows, t_i and t_i−1, defined as Chews Preceding a Swallow ${\mathit{CPS}}_i=N_i{|}_{t_{i-1}}^{t_i}$.

Although eating behavior is influenced by a large number of external factors like palatability, time of day, day of the week, social environment, etc. [22], the main hypothesis of the experiment was that different foods generate different chew and swallow sequences due to different properties of the foods being consumed in a meal [23]. Fig. 1 shows an example of the TPS and CPS signals generated for a complete meal of one subject. It can be seen that different foods in the experimental meal create distinct patterns in terms of TPS and CPS metrics over time. As an example, moist semi-solid food (yogurt) exhibits low chewing duration and high frequency of swallowing, while a tacky food (peanut butter sandwich) exhibits low swallowing frequency with prolonged periods of chewing. Thus, transitions between different foods in a meal can potentially be discovered by monitoring changes in features of the time sequence of chews and swallows.

Fig. 1.

Example of a complete meal of a subject. The figure on top shows the TPS signal, and the one on the bottom shows the CPS signal. Vertical dashed lines represent the time boundaries of the different segments or food items. Swallow locations in time are represented as dot marks.

Each swallow during consumption of given food was assigned a class label. Class labels were defined as C_i ∈ {1, 2, 3, 4, 5}, representing swallows of pizza, yogurt, apple, PB and water respectively. Feature vectors T_i were built based on different combinations of the described features as:

$$T_i=\{x_i,C_i\}{|}_{i=1}^k$$ (1)

where the combination of features for different space dimensionality was studied to observe the impact of t_i, TPS and CPS, extracted from the data, in the performance of the implemented models. Each of these feature combinations were defined as:

• x_i ∈ ℜ¹ = {t_i}

• x_i ∈ ℜ² = {CPS_i, TPS_i}

• x_i ∈ ℜ³ = {t_i, CPS_i, TPS_i}

• x_i ∈ ℜ⁷ = {t_i, TPS_i−1, CPS_i−1, TPS_i, CPS_i, TPS_i+1, CPS_i+1},

for i = 1, 2, …, k and k being the total number of swallows in a subject’s meal. The first feature set (ℜ¹) investigated the effect of sequence in food consumption. The second feature set (ℜ²) focused just on properties of chewing and swallowing associated with a given swallow, regardless of where they reside in time sequence of food intake. The third (ℜ³) and fourth feature (ℜ⁷) sets studied if the accuracy of meal segmentation can be further improved by combining information from (ℜ¹) and (ℜ²), and introducing information from the nearest swallows, respectively.

2.3. Meal segmentation

In order to identify groups of food within a meal, a clustering approach was used based on two different techniques. The first one is a relatively new unsupervised method named Affinity Propagation (AP); this method is a clustering algorithm that passes messages between data points to identify exemplars and form clusters around them [24]. It operates by simultaneously considering all data points as potential exemplars and exchanging messages between data points until a good set of exemplars and clusters emerges. One of the advantages of AP over other clustering techniques is the ability to determine the number of clusters from a given data set [24].

This approach was used in an attempt to automatically estimate the number of segments within the meal used in our experiments without specifying any a priori information about the number of food items consumed by the subjects using the feature vectors T_i ∈ ℜ^d as defined in Section 2.2.

The accuracy of the AP model in finding the correct number of food items was obtained based on an error measure across all subjects and all experiments as:

$$\mathit{Acc}\_\mathit{cluster}\%=(1-E)\times 100$$ (2)

where E is the error measure of finding exactly 5 known food items, defined as E = |n − 5|/5, with n being the number of estimated food items. Additionally, to compare accuracy between supervised and unsupervised approaches, a second model was implemented using Agglomerative Hierarchical Clustering (AHC) [25–27]. First, each feature vector T_i ∈ ℜ^d was considered a singleton cluster and the proximity matrix was calculated for the k initial clusters, using unweighted average distance algorithm (UGPGMA) with Euclidian distance as a measure between points [27]. This linkage distance represents the extent in which clusters are proximal to each other. A threshold α was set in such a way that clusters with linkage distances lower than a certain value were merged into a new cluster. Representation of the linkage distances can be observed at the dendrogram shown in Fig. 2. Here, the linkage distance is normalized to range [0,1].

Fig. 2.

The dendrogram shows the hierarchical tree for all swallows with an optimal threshold α (dashed line) obtained using supervised learning to find the correct number of food items, described by the branches below the threshold.

With an α = 0, each swallow represented by a leaf on the dendrogram is interpreted as a cluster and, consequently, as a different food item. At the other extreme, with an α = 1 all swallows will be merged into one cluster, representing only one food item.

To find an optimal threshold α, half of the subjects were selected randomly to be used as the training data set, and the remaining subjects were used as the validation data set. For each meal in the training set, all possible α values that give 5 groups were used to build a probability density estimation (PDE).

Fig. 3 shows the distribution of all possible α values obtained from the training data set. The optimal α selected is the one with the highest probability of this obtained distribution. This α value is then applied as the threshold for the normalized linkage distances of each meal in the validation set.

Fig. 3.

Probability density estimation of α obtained using the training set. The α with the highest probability represents the optimal threshold of the linkage distances to group swallows into groups of the same food type.

In Fig. 2, the optimal α obtained after training is shown as the dash line that partitions the dendrogram, estimating in the given example five different food items.

The accuracy of the correct number of food items was calculated for the validation data set based on the error measure, as in Eq. (2).

Ten runs were performed using hold-out cross validation to reduce variability due to random selection of the subjects for the training data set, and overall accuracy was assessed as the average across these runs.

2.4. Classification of swallows

The capability of the models described above to classify swallows into groups corresponding to food items consumed separately by a subject was analyzed, thus evaluating the predictive power of features derived from chewing and swallowing in the meal segmentation. The association of swallows into different group types, 5 food items for the particular case of this study, is of significant importance for the development of our proposed MIB methodology, since it has been observed that bite size, hence swallow size, differs for different types of food, i.e. solids and semi-liquids [23], and that the number of swallows is proportional to consumed food mass [10].

Being able to classify swallows into groups of different foods may represent a viable measure for mass estimation of total food consumed.

Fig. 4(a) displays an example of $T_i=\left\{x_i{C}_i\right\}{|}_{i=1}^k$ for x_i ∈ ℜ³. It can be observed how each one of the swallows was grouped into one of the five clusters for this particular experiment using the AP model. Fig. 4(b) shows the projection of the clusters into the CPS-Time plane and Fig. 4(c) shows the projection into the TPS-Time plane, where separation of different clusters can be observed. The projection of the clusters into the plane CPS-TPS is shown in Fig. 4(d), showing no obvious separation of different clusters.

Fig. 4.

Grouping of swallows using the Affinity Propagation clustering method. In increasing order of time, the swallow groups belong to: pizza, yogurt, apple, PB and water.

In order to evaluate the ability of the segmentation models to associate all swallows correctly into their corresponding food item, an accuracy measure was defined as the ratio of correctly classified swallows P for a given food type class C over the real number of swallows R belonging to that same class:

$$\mathit{Acc}\_{Sw}_{jC}\%=\frac{P_{jC}}{R_{jC}}\times 100$$ (3)

for j = 1, …, 5 number of total subject’s consumed food items.

3. Results

Meal experiments from 17 subjects were used to analyze two different models to automatically perform meal segmentation. Swallow timing, TPS and CPS signals extracted from swallows and chews occurrence respectively were used as features for these two models.

The first model was based on AP. Using this technique, the number of exemplars emerged from the message-passing procedure [24], that is, AP was capable of finding the optimal number of exemplars of the data set without any a priori knowledge. The second model was a supervised method based on AHC that sets a threshold and merges clusters with low linkage distances.

Table 1 shows the accuracy results (from Eq. (2)) as the average and standard deviation (SD) across all subjects of the different combinations of features described in Section 2.2 for both AP and AHC.

Table 1.

Average accuracy and standard deviation of the models implemented on finding the correct number of 5 food items for different combinations of features.

	xi ∈ ℜ1	xi ∈ ℜ2	xi ∈ ℜ3	xi ∈ ℜ7
AP	89.7% (SD 13.2)	21.8% (SD 40.2)	88.8% (SD 14.1)	90.3% (SD 12.7)
AHC	91.3% (SD 9.9)	70.7% (SD 29.9)	92.9% (SD 9.6)	95.3% (SD 8.5)

Fig. 5 shows the distribution comparison between the two models, AP and AHC, for automatically finding the number of food items in a meal. Results are expressed as a percentage across all subjects for the best case obtained of each model, that is, T_i ∈ ℜ⁷ and optimal threshold α = 0.286 for the AHC model, which presents a significant increase from 91.3% to 95.3% (paired t-test, p < 0.0015).

Fig. 5.

Automatic meal segmentation of food items consumed in a meal using Affinity Propagation and Agglomerative Hierarchical Clustering. Results display the percentage of estimated number of food items over the a priori known number of 5 food items using T_i ∈ ℜ⁷ and an optimal threshold α = 0.286 for the AHC model.

In comparison of obtained meal segmentation, accuracies for visits where subjects remained silent during the meal with those where subjects were encouraged to participate in conversation no significant difference was observed (paired t-test, p > 0.05).

Accuracy of the classification of swallows (from Eq. (3)) is shown in Fig. 6, where both models represent the correct association of swallows into each food type.

Fig. 6.

Results obtained on the classification of swallows from the Affinity Propagation and the Agglomerative Hierarchical Clustering algorithm with specified 5 clusters.

4. Discussion

This study presents two different models to automatically obtain the number of food items involved in a subject’s meal. To our knowledge, this is the first attempt to automatically perform such segmentation which is of interest for studies of dietary activities, obesity and other eating disorders.

When performing automatic meal segmentation, AP found a range of 4–7 (Fig. 5) food items for a 5-food items meal with an accuracy of correct segmentation of 90.3% (Table 1). The supervised AHC model implemented found a range of 4–6 food items (Fig. 5) with 95.3% accuracy (Table 1) with training performed on half of the dataset with the rest of the dataset used for validation. These results show that the supervised model based on AHC has a higher performance compared to the AP model in segmenting a meal, which can be seen in the 75.76% of finding exactly 5 food items within the meal, compared to the 59% of the AP model (Fig. 5). This might be explained by the ability of the supervised learning methods to adjust the tradeoff between the bias (incorrect output prediction) and the variance (different output for different training sets) or the learned model [28].

Swallow location in time t_i appears to be the most efficient individual feature used by the implemented models. This is expected as the meals are typically consumed in a specific sequence (e.g. salad, entree, desert) and many consecutive chews and swallows will belong to the same food type and grouped in time. However, the accuracy is improved >4%by introducing the CPS and TPS features into the models and using ℜ⁷ neighbors results in the AHC model (Table 1). The highest accuracy obtained with the supervised model with a feature combination of x_i ∈ ℜ⁷ suggests that it is not only timing of the swallows which drives the clustering, but significant contribution is provided by CPS and TPS features characterizing chewing and swallowing process.

When performing the classification of swallows of different food items with a priori knowledge of the number of food items, AHC and AP achieved very similar accuracies (Fig. 6). Association of swallows into groups may result in an accurate mean to measure the total consumed mass of a subject, due to the constant bite size within similar foods [23], and the significant correlation between the number of swallows and the mass ingested by subject [10]. Water swallow classification resulted in the highest accuracy for both methods. The high frequency of swallowing in liquids, which results in lower TPS values, but particularly in the zero values of the CPS due to the absence of chewing, may explain why the highest accuracy was obtained in this case.

Also, fairly good accuracy was obtained for pizza and yogurt, since the transition from one to the other results in significant changes in the TPS and CPS features. On the other hand, distinction between apple and peanut butter swallows resulted in a lower accuracy which can be explained by similarity of their TPS and CPS characteristics across all subjects.

The high accuracy evidence for efficiently separating swallows of pizza from yogurt, and peanut butter from water, suggests that the methodologies described in this study may be used in more general cases under free-living conditions where differences between segments within a meal are significant. Additionally, the significant differences in the TPS and CPS signals may result useful in building models that identify the types of foods consumed, i.e. solid, semi-solid and/or liquid intake within a meal.

The features t_i, TPS and CPS used in this study were obtained from manual scores of sound recordings of swallows and strain sensor signal of chews, but they may potentially be obtained from automatic scores of swallowing and chewing detection methods that would enable creation of a wearable device [11]. This is part of our ongoing research goals.

One limitation of the reported study is that food items were consumed unmixed and in a specific order, which allowed to eliminate uncertainty in inter-food variation of food properties. In free-living conditions, the number and type of food items within a meal will vary according to subject’s behavior and preferences [29]. Further study is needed to test the clustering algorithms performance on unrestricted meals. However, the presented clustering approaches should be directly applicable to unrestricted meals as food intake is cumulative over time [30] and consumption normally happens sequentially in a bite-by-bite manner, similar to the meal in this study.

The proposed methodology is a first step toward the automatic meal segmentation that could eventually be used in the study of eating disorders, obesity, and in biofeedback applications aimed at managing and monitoring ingestive behavior with unconstraint conditions, i.e. time of food intake, number of food types consumed and the order of consumption.

5. Conclusion

Two models for automatic meal segmentation into distinct food items were proposed and evaluated on data collected from 17 subjects. An overall accuracy of >95% was obtained when estimating the number of food items in a meal using the implemented model based on supervised Agglomerative Hierarchical Clustering, compared to a 90% of the unsupervised Affinity Propagation model. Experimental results suggest that these features may be used as predictors of the composition of food items within a meal. The proposed models may be advantageous for development of an accurate, inexpensive, and non-intrusive methodology for objective monitoring of food intake in free living conditions.