Detection of Food Intake from Swallowing Sequences by Supervised and Unsupervised Methods

Abstract

Studies of food intake and ingestive behavior in free-living conditions most often rely on self-reporting-based methods that can be highly inaccurate. Methods of Monitoring of Ingestive Behavior (MIB) rely on objective measures derived from chewing and swallowing sequences and thus can be used for unbiased study of food intake with free-living conditions. Our previous study demonstrated accurate detection of food intake in simple models relying on observation of both chewing and swallowing. This article investigates methods that achieve comparable accuracy of food intake detection using only the time series of swallows and thus eliminating the need for the chewing sensor. The classification is performed for each individual swallow rather than for previously used time slices and thus will lead to higher accuracy in mass prediction models relying on counts of swallows. Performance of a group model based on a supervised method (SVM) is compared to performance of individual models based on an unsupervised method (K-means) with results indicating better performance of the unsupervised, self-adapting method. Overall, the results demonstrate that highly accurate detection of intake of foods with substantially different physical properties is possible by an unsupervised system that relies on the information provided by the swallowing alone.

INTRODUCTION

According to the National Health and Nutrition Examination Survey (NHANES), the prevalence of being overweight among the U.S. adults increased from 30.5% in 1960 to 32.0% in 1994 with an increase in obese individuals from 12.8 to 22.5%.⁹ By 2004, the overweight population reached 66.3%, and the percentage of obese individuals increased to an alarming 32.2%.¹⁸ The significant increase in the incidence of overweight and obesity constitutes a potential threat to life expectancy in the USA¹⁹ and has substantial economic consequences for the U.S. health care system. The total cost attributable to obesity was estimated to be $99 billion in 1995 (close to 12% of the total national health expenditure).³¹ By 2000, the total healthcare cost attributable to obesity rose to $117 billion with most significant contributions from Type 2 diabetes, coronary heart disease, and hypertension.²⁹

Obesity has been characterized as “a complex multifactorial chronic disease that develops from an interaction of genotype and the environment. Our understanding of how and why obesity develops is incomplete, but involves the integration of social, behavioral, cultural, physiological, metabolic and genetic factors.”²¹ However, the dramatic increase in obese population over the last two decades suggests that the modern lifestyle is the most probable cause of obesity epidemic,³ i.e., there has been an increase in the energy intake and a decrease in the energy expenditure leading to a positive energy balance reflected in body weight gain.

Different methods have been used to measure energy intake and associated eating behaviors. Dietary self-reporting-based methods like observation, weighed food records, estimated records, diet history food-frequency questionnaires, and food recall methods are some of the most commonly used. Most of these methods tend to underestimate energy intake.¹⁴ People tend to miscalculate and underreport their daily intake due to two main factors: change in eating behavior that occurs when subjects are asked to record their intake (observation effect), and the subject's tendency to misreport the changed eating behavior (reporting effect).^{4,6,7,16,23,30} Even when the usage of these self-report methods may provide fairly good estimates of portions of energy intake, they lack information on the specific patterns of food intake throughout the day. For example, two studies indicated that the subject-reported total daily energy intake was significantly lower than measured intake due to snacking. One of them stated that, while meals were accurately reported, energy from snack foods eaten between meals was significantly underreported.²² A similar effect was observed in an earlier study where subjects tended to misreport or admitted not reporting snacks at all.¹⁵ It is necessary to improve the detection of snacking activities to reduce bias due to underreporting.

New research considers the use of objective methods of Monitoring of Ingestive Behavior (MIB) for dietary assessment where accurate measurement of periods of food intake can provide useful information and bring a more explicit insight in the study of eating behaviors associated not only with overweight and obesity, but also for other eating disorders. For example, MIB might be helpful in diagnostics of such behaviors as night-eating or evening snacking.²⁴ A need for an accurate and objective MIB method has also been indicated for conditions where diet regimens and eating behaviors have to be very strict like in those of patients suffering from Chronic Kidney Disease (CKD) where food intake has to be scheduled according to dialysis times,²⁸ or to monitor diet in Type 2 diabetic patients, where patients may express feelings of dietary deprivation.³²

To date, methods of MIB are relatively poorly developed. In a study that involved patients with CKD,²⁸ a semi-automated MIB methodology was studied where subjects introduced data from food products using voice recordings and a PDA that scanned the bar code of products to be eaten. This methodology still relies heavily on self-report cooperation from the subjects. Another study proposed the use of sensors to detect chews, swallows, gestures, and movements, and the use of this information to identify dietary activities²; however, precision of these methodologies was not sufficient for practical applications. More research is needed to develop and evaluate MIB methods and technology such that it is suitable for free-living conditions with minimal cooperation from individuals. Our earlier study²⁶ reported a MIB method that achieved >95% accuracy in detection of food intake using information from chews and swallows counted within a fixed time window of 30 s. The goal of the food intake detection methodology proposed here is to improve detection over previous models by (1) monitoring swallow sequences alone, and thus eliminating the chewing sensor to simplify the implementation of a system that could be applied in free-living conditions; (2) classifying each swallow instance individually instead of labeling of all swallows within a time window and thus achieving more accurate models for mass prediction; (3) using supervised and unsupervised machine learning techniques to improve the model accuracy and reduce the inter-subject variability.

This article is organized as follows. “Data Collection” section consists of a brief description of the data set. “Methodology” section presents a description of the models for food intake detection: a group model based on a supervised learning technique (Support Vector Machines) and an individual model based on an unsupervised learning method (K-means clustering). Also described are feature extraction methods and techniques for assessing the accuracy of classification. “Results” section provides results obtained for all models implemented. Finally, “Discussion” section contains a discussion on the results, and “Conclusions” section the related conclusions.

DATA COLLECTION

Data from 18 human subjects were used (11 males, 7 females).²⁷ The average BMI across the sample population was 28.01 ± 6.35 km/m², age of the subjects varied from 18 to 57 years. Subjects participated in four visits performed at lunch time on different days. Each visit consisted of three different periods: an initial 20-min resting period, where the subjects remained silent for 10 min and spent 10 min reading aloud; a food intake period, where the subjects were asked to consume different food items in a specific order (pizza, yogurt, apple, peanut butter sandwich, and water); a final 20-min resting period, identical to the first one. The selection of used food types represented different physical properties such as crispiness, softness/hardness, and tackiness. This ensured applicability of the proposed methodology to various foods. Possible effects of talking during a meal on frequency of swallowing were accounted by engaging the subjects into a meal-time conversation during two of the visits. Possible effects of meal size on frequency of swallowing were accounted by serving two sizes: standard and large (50% larger). During the visit, subjects were monitored using a multi-modal sensor system that among other signals captured swallowing sounds and video of the food intake.²⁷ After the visit, the acquired sensor signals were visualized using a custom-designed software package and beginning and end of each swallow were marked (scored) manually by a human rater. Reliability of the manual score was confirmed by an inter-rater agreement study.²⁷ The collected sensor signals (including video recordings) and the resulting manual scores are being used as a gold standard for assessment of sensor design, development of automatic pattern recognition routines, and study of the relationship between swallowing/chewing and ingestive behavior. The wearable system being designed for use in free-living conditions will utilize a substantially reduced set of sensors. The swallowing scores were obtained on approximately 63 h of recorded visits, with a total of 4045 spontaneous swallows where no food was ingested, and 5811 food intake swallows. The swallowing scores represent the time sequence of swallowing events as they happen during the visit and serve as the primary data used by the food detection algorithms.

METHODOLOGY

The proposed methodology is based on the time sequence of swallows as the main predictor of food intake. Figure 1 shows the distribution of swallowing events for two different subjects in a 180-s window extracted from a resting period (Fig. 1a) and a food intake period (Fig. 1b). Figure 1 illustrates two fundamental differences: (1) the frequency of spontaneous swallowing during resting is lower than the frequency of swallowing during food intake and, (2) the spontaneous swallows are significantly more periodic than solid food intake swallows in which a relatively long period of chewing is typically followed by two more closely spaced swallows. These observations suggest that the absolute time difference between neighboring swallows can be used as a predictor. Another important observation is the significant difference in swallowing frequency between subjects which has to be considered when monitoring ingestive behavior. This suggests the modeling in two different approaches: (1) build a group model based on a sample representative of a population to overcome inter-subject variability and, (2) build an individual model that incorporates individual traits exhibited by subjects. The former requires the construction of a model obtained from a training process and a supervised learning method. Since training of individual models is impractical for real world applications, the latter model suggests the implementation of an unsupervised learning method, where self-adapting individual models are created automatically.

FIGURE 1.

Distribution of swallows over a 180-s window, extracted from the visits of two subjects, separated by two types: (a) spontaneous and, (b) food intake (pizza). It can be observed that the swallowing rate of food intake swallows is significantly higher than in resting periods and also, the occurrence of swallows is less periodic.

Feature Extraction

For each swallow located at time t_i, where i = 1, 2, ..., n is the total number of swallows in a subject's visit, the absolute difference in time is calculated for d neighboring swallows of t_i (d/2 previous and d/2 subsequent), resulting in a time feature vector ${\tau }_i\in {\Re }^d$, defined as,

$${\tau }_i=(t_{i-d∕2+1}-t_{i-d∕2},t_{i-d∕2+2}-t_{i-d∕2+1},\dots ,t_i-t_{i-1},t_{i+1}-t_i,\dots ,t_{i+d∕2-1}-t_{i+d∕2-2},t_{i+d∕2}-t_{i+d∕2-1})$$ (1)

Class labels for every t_i are defined as r_i = –1 if t_i ∈ C₁, and r_i = +1 if t_i ∈ C₂, where C₁ represents the spontaneous swallow label, and C₂ the food intake swallow label. Each visit is then represented as a set of time feature vectors $T_j={\{({\tau }_i,r_i)\mid {\tau }_i\in {\Re }^d,r_i\in \{-1,1\}\}}_{i=1}^n$, for the complete data set of j = 1, 2, ..., N visits. Since food intake swallows are more densely spaced than spontaneous swallows, scale equalization using a natural logarithm was applied to the time feature vector: ${\text{log}}_e\left({\tau }_i\right)\in {\Re }^d$, and compared to the linear scale models. In this study, two different models for periods of food intake classification were developed. The first one is a supervised learning method implemented as a group model, and the second one is an unsupervised learning method implemented as an individual model. The same time feature vectors ${\tau }_i\in {\Re }^d$ were used for both methods. Dimensionality d of these vectors was changed as d = {2, 4, ..., 10} to observe the models behavior. The difference in utilization of these feature vectors for two methods was the labeling of features for selection of the training data set for the supervised model. Unsupervised models do not need labeled feature vectors since no training process is involved.

Evaluating Accuracy of Food Intake Detection

In order to evaluate the models implemented, the manual scores are used as the gold standard. The accuracy is then calculated as the ratio between the number of correctly classified swallows, and the total number of swallows, as expressed in,

$$\mathrm{Acc}=\frac{T_{+}+T_{-}}{T_{+}+F_{-}+T_{-}+F_{+}}$$ (2)

where (T₊) is the number of correctly classified food intake swallows, and (T_–) is the number of correctly classified spontaneous swallows; (F₊) and (F_–) are the number of incorrectly classified food intake and the number of spontaneous swallows, respectively. Also, specificity and sensitivity are computed²⁰:

$$\text{Sensitivity}=\frac{T_{+}}{T_{+}+F_{-}},$$ (3)

$$\text{Specificity}=\frac{T_{-}}{T_{-}+F_{+}}$$ (4)

Group Model Based on Supervised Learning

The goal of a group model is to perform correct detection of food intake without being affected by the inter-subject variability. A group model is a population-based approach, wherein a significant sample of the population is used to define a representative model. This implies the use of supervised learning, where a model is built based on training data that consist of labeled examples defined by common descriptive features. The resulting model is then used to predict the labels of future data based on their known features.^1,8,12 There are many supervised learning techniques that can be used for classification tasks, one of them being Support Vector Machines (SVMs), which has proved to have excellent performance in classification problems.^17,25 SVM relies on the processing of the data to represent patterns in a higher dimension, with an appropriate mapping function into a new space, to be solved by a linear function, which would be the same as a nonlinear function in the original space.^1,8 Because of this particularity, SVM is capable of producing very complex decision functions. In order to implement a classifier using SVM technique, Lib-SVM software package is used.⁵ This SVM classifier utilized Radial Basis Functions with Gaussian kernel e^{(–γ·|U–V|2)}. The near optimal values of parameters C (cost of misclassification) and γ (width of Gaussian kernel) were established using a grid search procedure varying C as C = e^x for x = {0, ..., 5} and γ as γ = e^y for y = {–4, –3.8, ..., 3}. Using this approach, the time needed to find near optimal parameter values is approximately equal to the one needed by more advanced methods (for example, using the area under the ROC curve) since there are only two parameters.¹⁰ Furthermore, the grid search can be easily parallelized because values of C and γ are independent. Since in a practical application a non-detection of food intake (false negative) is as undesirable as a false detection of food intake (false positive), the analysis in the comparison of different classifiers was focused on average overall accuracy rather than the receiver operating characteristic curves. Hold-out cross-validation is used to train and validate the group model. From the 18 subjects, nine are selected randomly and used to train the group model; the remaining nine subjects are used as the validation data set. In order to reduce bias in results, ten randomized replicates are performed and the accuracy of the classifier is obtained as the average accuracy across the validation results.

Individual Model Based on Unsupervised Learning

Another way to overcome inter-subject variability is to build individual, subject-specific models. Since training of individual models is not feasible for practical applications, unsupervised, self-adapting techniques present a better alternative.^1,8,13 K-means clustering is one of the most popular unsupervised learning methods for classification.¹³ This technique initially chooses k cluster centers randomly, and each of the data points is assigned to the closest cluster center. Cluster centers are then recomputed using the data associated to them. This process continues until a convergence criterion is met.¹¹ Matlab® Statistical Toolbox (The MathWorks Inc.) is used to implement a classifier model based on the K-means algorithm. Utilized K-means algorithm from MATLAB® Statistical Toolbox performs a two-phase iterative algorithm to minimize the sum of point-to-centroid distances, summed over all k clusters. The first phase uses batch updates, wherein each iteration consists in reassigning points to their nearest cluster centroid, followed by recalculation of clusters centroid. The second phase uses online updates, where points are individually reassigned to reduce the sum of distances, and clusters centroid are recomputed after each reassignment. The problem of finding the global minimum of the sum of point-to-centroid distances can only be solved in general by an exhaustive selection of starting points; using several replicates with random starting points typically results in a solution that is a global minimum. Two clusters are defined by their centers, representing spontaneous swallows and food intake swallows. After the algorithm finds the corresponding cluster centers, the one with the lowest magnitude $(\stackrel{‒}{k_a}<\stackrel{‒}{k_b}\text{or}\stackrel{‒}{k_b}<\stackrel{‒}{k_a})$ is assigned to food intake since it has been observed in practice that the time feature vectors for food intake are closer to the origin in the vector space (Figs. 2a and 2b). Euclidian distance is used as the similarity measure,⁸ and each data point is associated to the closest cluster.

FIGURE 2.

A graphical representation of the methods implemented with a 2D time feature vector shows the differences of two different scales. The gray area represents the group model obtained from the training process. Data points from one experiment of the validation data set are displayed marked according to its corresponding type. Also the resulting two cluster centers of the individual model for that particular experiment: (a) using a linear scale of τ_i, (b) using natural logarithm scale log_e(τ_i).

In order to evaluate the individual model, accuracy is defined as in Eq. (2), together with sensitivity (3) and specificity (4). Ten replicates are performed for each visit to reduce variability in results due to random initialization of the cluster centers, and the accuracy average is calculated. The accuracy of the individual model is then obtained as the average across all visits.

RESULTS

Time feature vectors ${\tau }_i\in {\Re }^d$ with different dimensionality values, d = {2, 4, ..., 10}, are used to train a group model using SVM and to obtain individual models using K-means. The group model was able to achieve an accuracy in swallow classification of 83.2% using 2D τ_i (82.4% specificity and 82.7% sensitivity). Increasing the dimension of the time feature vector resulted in a decrease of the model accuracy to 64.36% accuracy (sensitivity of 79.6% and specificity of 63.0%) for the 10D τ_i (Table 1).

TABLE 1.

Results obtained for the food intake models using ${\tau }_i\in {\mathfrak{R}}^d$ as features.

	Group model			Individual model
Dimension d	Accuracy (%)	Sensitivity (%)	Specificity (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
2	83.2	82.5	82.7	80.0	97.2	54.1
4	82.5	78.7	85.0	83.3	97.0	62.5
6	78.0	68.4	93.0	84.6	96.9	66.4
8	67.9	78.5	69.2	85.2	96.8	68.1
10	64.4	79.6	63.0	85.7	96.7	70.2

On the other hand, the individual models based on K-means had an accuracy increase from 80.0 to 85.7% as the dimensionality of τ_i was increased from 2 to 10. No difference was observed on the sensitivity of the model across all dimensions of τ_i. The specificity increased from 54.1 to 70.2% for 2- and 10D τ_i respectively (Table 1).

The use of a natural logarithm scale on the time feature vector ${\text{log}}_e\left({\tau }_i\right)\in {\Re }^d$ resulted in a significant improvement on group and individual models. For the group model, accuracy increased to 89.4% using a 10D τ_i. Sensitivity also increased up to 89.6% and more significantly, specificity increased to 90.5% (Table 2). Similar improvement can be observed in individual models, where an accuracy of 93.9% was achieved with the 10D τ_i. Sensitivity and specificity for the same model were 92.4 and 95.7%, respectively (Table 2).

TABLE 2.

Results obtained for the food intake models implemented using ${\log }_e\left({\tau }_i\right){\mathfrak{R}}^d$.

	Group model			Individual model
Dimension d	Accuracy (%)	Sensitivity (%)	Specificity (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
2	83.5	85.2	84.5	89.6	86.4	93.5
4	88.7	86.8	92.9	92.5	89.6	96.0
6	86.9	85.1	90.7	93.4	91.2	96.2
8	90.0	88.3	93.5	93.8	92.1	96.0
10	89.4	89.6	90.5	93.9	92.4	95.7

Through the use of the model that resulted in the best accuracy, classification was observed for each food item in the meal period separately in terms of sensitivity. Results can be seen in Table 3, where the lowest sensitivity is observed for pizza with 78.2% of correct classification of swallows, and the highest was for water with 98.5% of correct classification.

TABLE 3.

Sensitivity results obtained for each type of food with a 10D individual model with natural logarithm scale of τ_i.

Food type	Pizza	Yogurt	Apple	Peanut Butter Sandwich	Water
Sensitivity	78.2%	96.3%	94.7%	88.7%	98.7%

DISCUSSION

This study demonstrates that swallowing events can be used to detect periods of food intake. Previous reported methods achieved a classification accuracy of 95% for detection of periods of food intake using swallows and chews instances as predictors.²⁶ The use of only swallows as predictors in methods proposed here eliminates the need for a chewing or any other additional sensors^2,26 while achieving comparable accuracy of 94%, These models are based on per-swallow classification as opposed to time epoch classification²⁶ which allow detection of short periods of food intake associated with snacking that may last only for several swallows. Also, the proposed unsupervised MIB method is automatically self-adapting which eliminates the need for extensive training to create a group model or for a training meal in the case of an individual model. This MIB method would simplify the implementation of a less intrusive and compliant device.

Supervised and unsupervised learning methods were used to classify a swallow as a food intake or a resting swallow. A supervised learning method was used to build a group model. It has been observed that this model is particularly affected by the high inter-subject variability. Figure 2a shows a tendency of the group model to overfit the training data when linear scale is used. This effect could explain why increasing dimensionality of τ_i results in a decrease of the performance of the model (Fig. 3a). The overfitted model is a direct consequence of high inter-subject variability. Figure 2b shows how the resulting group model overcomes this overfitting by using a natural logarithmic scale of τ_i. Having a more suitable group model allows the use of higher dimensions of τ_i that increase the accuracy of the model (Fig. 3a).

FIGURE 3.

Average overall accuracy across the validation data set when two different time feature vectors scales are used, linear τ_i and log_e(τ_i), and their dimensionality values are changed as d = {2, 4, ..., 10}. (a) Shows the results obtained from the group model implemented using SVM and (b) shows the results obtained from the individual model implemented using K-means.

In the unsupervised learning method, clusters are defined based on the data associated to a particular visit, and self-adapt to variability in the data. Increasing the dimensionality of τ_i for the individual model improves the accuracy in detecting swallows (Fig. 3b). However, low specificity obtained (Table 1) suggests that this model is affected by intra-class variability of spontaneous swallows. This can be seen in Fig. 2a, where spontaneous swallows have a wider distribution compared to the food intake swallows. The use of natural logarithmic scale reduces the intra-class variability significantly and gives better trade-off between specificity and sensitivity (Table 2), thus enabling more accurate prediction.

These results suggest that models implemented using an individual approach achieves a higher performance than those implemented with a group approach. Use of the unsupervised learning method provides a significant improvement from a practical point of view, since it eliminates the need for a training process, with the advantage of accounting for individual traits of each subject and having a more accurate classification model. A reason of a better performance of the unsupervised over the supervised learning method is most likely because the unsupervised model is developed for each subject and visit and therefore not impacted by inter-subject and inter-visit variability.

From the classification results of the individual model for each food item when analyzed separately, it is interesting to see that water has the highest accuracy and that pizza resulted in the lowest (78%) but still practically usable sensitivity. Yogurt, apple, and water had an accuracy >94%. A possible explanation is that these three food items present a higher swallowing frequency due to the high content of liquids in them. These results suggest that the proposed methodology may be usable for a variety of foods with significantly different physical properties, although a further investigation involving a larger number of foods would be necessary.

CONCLUSIONS

Two models to detect periods of food intake from swallowing sequences were implemented and compared. The use of swallowing sequences alone (with no additional information) significantly simplified the methodology to be implemented in a wearable device for use in free-living conditions. The group model, based on SVM implementation had an accuracy of 89.9% for detection of food intake. The individual model based on K-means clustering achieved an accuracy of 93.9%. Employing unsupervised learning to build an individual model for classification of swallows to identify periods of food intake has several significant practical improvements over the group model: it eliminates the need of a training process using labeled examples, it is not sensitive to inter-subject variability that is usually reflected in lower performance of group-based models, and it potentially enables the creation of a wearable device for MIB due to its simple implementation.