Smoking detection based on regularity analysis of hand to mouth gestures

Abstract

A number of studies have been introduced for the detection of smoking via a variety of features extracted from the wrist IMU data. However, none of the previous studies investigated gesture regularity as a way to detect smoking events. This study describes a novel method to detect smoking events by monitoring the regularity of hand gestures. Here, the regularity of hand gestures was estimated from a one axis accelerometer worn on the wrist of the dominant hand. To quantify the regularity score, this paper applied a novel approach of unbiased autocorrelation to process the temporal sequence of hand gestures. The comparison of regularity score of smoking events with other activities substantiated that hand-to-mouth gestures are highly regular during smoking events and have the potential to detect smoking from among a plethora of daily activities. This hypothesis was validated on a dataset of 140 cigarette smoking events generated by 35 regular smokers in a controlled setting. The regularity of gestures detected smoking events with an F1-score of 0.81. However, the accuracy dropped to 0.49 in the free-living study of same 35 smokers smoking 295 cigarettes. Nevertheless, regularity of gestures may be useful as a supportive tool for other detection methods. To validate that proposition, this paper further incorporated the regularity of gestures in an instrumented lighter based smoking detection algorithm and achieved an improvement in F1-score from 0.89 (lighter only) to 0.91 (lighter and regularity of hand gestures).

1. Introduction

According to The World Health Organization, smoking is the single most preventable cause of early death [1]. Cigarette smoking causes about twenty percentage of all annual deaths in the United States each year and also increases the chances of many serious illnesses. Smoking-related illness in United State cost $300 billion each year including direct medical care and lost productivity due to smoking generated illnesses [2,3]. The massive public health consequences of cigarette smoking illustrate the need for treatments for helping people to quit smoking.

An initial step in many cessation programs is the characterization of people’s smoking patterns over time. Although the predominant method of assessing smoking patterns comes through self-report [4], many different technologies to monitor smoking activities in daily life have been studied [5–7]. Nowadays wearable technologies, such as respiration sensors, proximity sensors, smart cigarette lighters, and inertial measurement unit (IMU) sensors are attractive because they provide a practical and powerful way to track smoking. Instrumented lighters can capture and record the lighter press and release events as a measure of cigarette consumption [8,9], however no details of smoking duration can be found from this information. IMUs have been widely used in wearable systems for assessing hand/arm movement. Their small size and affordability make them simple and effective tools to record the pattern of hand/arm movement. IMU-based studies have mainly focused on the hand-to-mouth gestures (HMGs) within the smoking event. The main challenge of hand-to-mouth gesture-based methods is to differentiate smoking-related gestures from other daily activities. Several different approaches and algorithms have been reported for recognition of smoking HMGs and smoking events.

In [10], a smoking gesture detection system using two 9-axis IMUs (accelerometer, gyroscope, and magnetometer) at the wrist and elbow was presented. Data were collected from 15 subjects performing different activities such as smoking, drinking, and eating. The relative position of the wrist to the elbow was calculated as a predictor of smoking. A number of features were extracted from the position and trajectory information of the wrist including speed, distance, duration, roll, and pitch angles. Random forest and conditional random field models were used to predict motions based on these feature sets. The model reached an F1-score of 0.85 for 10-fold cross-validation. The model was also applied to 4 users who wore two IMUs for 4 h each on 3 days in the field, achieving an accuracy (F1-score) of 0.83.

In [11], smoking events and puffs detection were performed by using four 3D accelerometers (at the dominant wrist, dominant upper arm, non-dominant wrist, and ankle). The evaluation involved 11.8 h of data from 6 subjects who performed different activities including eating, drinking, reading, using a phone, and using a computer. The whole raw data were segmented into fixed length segments of 25 s and 50% overlap. Mean, standard deviation, max, min, median, kurtosis, skewness, percentile, SNR, RMS, correlation coefficients, slope, MSE, and R-squared of each segment were used as features. An F1-score of 0.7 was reported for puff detection by a random forest classifier. For smoking detection, three specific history lengths were used to calculate puff frequency from puff classifier results. The smoking score for the current window was estimated by using Gamma distribution of puff frequency, resulting in F1-score of 0.79 for smoking detection.

In [12], the authors used a smartwatch to detect smoking activity. They collected 45 h from 11 subjects. Four features minimum, maximum, skewness, and kurtosis were extracted from accelerometer and gyroscope data over a window of 30 s. For smoking detection, the proposed method achieved an F1-score in the range from 0.83 to 0.94 in leave-one-subject-out validation.

In [13], 6 subjects wore four 6D inertial sensors on dominant arm. For each subject, 3.5 h of data were collected in a controlled laboratory setting. Mean, standard deviation, maximum, minimum, peak to peak, RMS and correlations between axes of the data were calculated within a window size of 10 s to extract features. A support vector machine model used to predict cigarette smoking based on these feature sets generated a false positive rate between 0.07 and 0.2 for different subjects.

In summary, a variety of features extracted from IMU data have been employed in previous research for the detection of smoking. However, none of the previous studies investigated gesture regularity as a way to detect smoking events. Regularity an important measure of periodicity and variations in periodicity, has been widely used in the studies of gait monitoring and analysis [14,15]. In this paper, we explore the hypothesis that periodicity of hand gestures during smoking may be an important predictor of smoking events.

The purpose of this study was to investigate the validity of smoking detection from the regularity and periodicity of hand-to-mouth gestures. In order to make the algorithm simple, lightweight, and computationally inexpensive, HMG detection was limited to a single axis of the inertial sensor. Also, the regularity of hand gestures was calculated by an unbiased autocorrelation procedure, which is commonly used in gait analysis studies [15–17].

2. Theoretical background

The autocorrelation procedure is a common technique to assess the repeating characteristics over a signal series having periodic and irregular forms. Consider a discrete-time signal series containing N points [x₁, x₂, … , x_N-1, x_N]. Eq. (1) calculates the unbiased autocorrelation coefficient a_m, which is the sum of the of x_i multiplied by another signal x_i+m at the given phase shift m. An autocorrelation sequence A=[a-_m,…,a₀,a₁, … ,a_m−1,a_m] can be presented by 2N-1 autocorrelation coefficients obtained at every phase shift m. Only the right segment a₀ to a_m of autocorrelation coefficients will be considered for simplicity.

$$a_m=\frac{1}{N-\left|m\right|}\sum _{i=1}^{N-\left|m\right|}{x}_i{x}_{i+m}$$ (1)

A signal sequence with a good repetitive pattern produces autocorrelation coefficients with peak values for intervals corresponding to the signal periodicity. Fig. 1 shows an example of an autocorrelation sequence computed from HMG gestures detected on the wrist during a smoking activity. The first coefficient peak D₁ indicates the first dominant period, the second peak D₂ the second dominant period, and so on.

Fig. 1.

Detected HMGs during smoking (top) and unbiased autocorrelation sequence of those data (bottom), the phase shift, m, is the average time difference between two smoking-HMGs. D₁ and D₂ are the first and second dominant periods respectively.

The amplitude of D₁ characterizes the regularity of hand gesture, similar to the regularity of gait [18]. The first dominant period shows the maximal similarity between the HMG sequence and its shifted version by m-point. The phase shift m shows the average time difference between two smoking-HMGs. This measure can be effectively used to quantify how frequently a smoker brings his/her hand to the mouth for puffs. The amplitude of D₁ indicates the similarity of neighboring HMGs from the viewpoint of HMGs duration and time difference between HMGs. A high regularity value shows that the duration of puffs and interpuff intervals are very similar across that smoking event.

3. Methods

3.1. Subjects

Thirty-five (male: 24, female: 11) medium to heavy smokers (Age: 25.1 ± 11.8 years, range: 19–62, body mass index: 24.6 ± 6.1 kg/m²,) with no chronic respiratory problems were recruited for the validation study. Each subject provided written informed consent prior to participation in the study, which was approved by the Institutional Review Board of the University of Alabama.

To qualify for the study, subjects had to report smoking at least 8 cigarettes per day, provide a breath carbon monoxide sample of >10 ppm. The self-reported cigarette consumption of the subjects was 9.1 ± 5.9 cigarettes per day. Average expired breath carbon monoxide (CO) measurement, collected at the beginning of the study, was 14.7 ± 6.1/8–33 ppm (using a BreathCO vitalograph device).

3.2. Instrumentation

The hand module of the PACT (Personal Automated Cigarette Tracker) v2.0 platform [19] was employed for capturing hand gestures associated with both smoking and non-smoking activities. This module contained a six-axial IMU (LSM6DS3) interfaced with an STM32L151RD processor. The accelerometer was configured to have a ± 8 g measurement range with 16bits of resolution. All IMU signals were sampled at the frequency of 100 Hz. The recorded data were stored on a 4-GB Micro-SD card for offline computer processing.

The instrumented lighter of PACT v2.0 was employed to record the time and duration of all lighting events. The lighter is a customized version of a commercial low-cost piezoelectric lighter. This commercial lighter was augmented to include a microcontroller, a battery, and a plastic strip glued one end to lighter trigger and another end to the magnet. When the trigger was pressed, the timestamp from the internal clock of the processor was read and stored in the memory. Fig. 2 shows the accelerometer orientation of hand module placed on the wrist and the instrumented lighter module.

Fig. 2.

Accelerometer axes (positive direction) for hand module and instrumented lighter.

3.3. Study protocol

The controlled portion of the study was conducted at the University of Alabama. The purpose and procedure of the study were told to the participants at the beginning of the controlled portion. Subjects were outfitted with a wrist device of the PACT 2.0 system and asked to use the instrumented lighter of PACT 2.0 for lighting their cigarettes. During the controlled portion, subjects were asked to perform 10 different activities in the lab (Table 1), including a lunch in the cafeteria.

Table 1.

Activity list for controlled and free-living portion of the study.

In controlled portion		Self-reported
order
1	Read aloud	Smoking
2	Slow walk (1.8 ± 0.3mph)	Eating
3	Fast walk(3.0 ± 0.45 mph)	Sedentary
4	Resting	Sleeping
5	Sitting + Smoking	Physical activity
6	Phone call
7	Eating
8	Walking + Talking + Smoking
9	Unconstrained activity
10	Walking + Smoking
11	Unconstrained activity
12	Standing + Smoking

Activities had a maximum duration of 5 min except for the eating and smoking activities. The laboratory session was videotaped by an i ON camera (Contour Action camera) time-synchronized with PACT2.0 sensors. The start and end timestamps of each activity were also marked in a freely available smartphone application (a Time Logger – Time Tracker). After completion of the controlled study, subjects left the laboratory and started their free-living portion. The activities during free-living portion were not restricted. The subject self-reported the start and stop time of their activities using the aTimeLogger application installed on their cellphone and returned to the laboratory after 24 h. Table 1 shows the activities in the controlled portion and in free-living of the study.

3.4. Dataset

A total 821 h of data (55 h controlled environment and 816 h of free-living) was recorded from 35 subjects. The electronic lighter recorded 455 lighting events (140 in the controlled environment and 315 from the free-living portion). Subjects registered 303 smoking events during free-living portion of the study. 18 subjects reported fewer than 7 smoking events, 13 subjects reported between 7 and 14, and 4 subjects reported more than 20 smoking events.

3.5. Overview of the proposed model

Fig. 3 shows an overview of the smoking detection algorithm from the raw accelerometer signal. First, hand-to-mouth gestures were detected from the time series of 1D (x-axis) accelerometer signal. Second, regularity scores of HMGs were calculated by using unbiased autocorrelation function and a sliding window. A smoothed regularity plot was obtained by using a moving average filter. Finally, smoking events and their boundaries were recognized via the detection of regularity peaks and the width of the peaks, respectively.

Fig. 3.

An overview of key steps for smoking detection in the proposed model.

3.6. Hand-to-mouth gesture detection

HMGs were detected by identifying the sequence of puff movement [20] in the raw accelerometer data (x-axis) collected from the axis parallel to the hand (Fig. 2). The change of x-axis accelerometer value from 0 or 1 g to − 1 g, due to the gravity, indicates the movement of hand toward the mouth. An opposite change in this axis indicates the movement of hand away from the mouth. Every falling and rising edge of the x-axis signal was considered as an HMG. For automatic identification, the following edge-detection algorithm was employed (Fig. 4).

Fig. 4.

Candidate HMG segments (a) x-axis accelerometer signal (b) wavelet filtered signal (c) derivate signal.

• The accelerometer x-axis signal was initially filtered by a wavelet filter (Haar wavelet with 9 decomposition levels).

• A time derivation of the wavelet filtered signal was done to obtain the rising and falling edge of a sequence of puff movement. This process created negative and positive spikes, which correspond to the start and end of HMGs respectively.

• The time segment between a minimum negative spike and the following maximum positive spike was defined and labeled as the HMG if its duration was longer than 2 s.

3.7. Regularity of HMG

HMG data were segmented into 3 min sliding windows with 10 s overlap. Within each segment, the unbiased autocorrelation coefficients from HMG data were calculated. Subsequently, the first dominant period (D₁) of autocorrelation coefficients was determined by finding local maxima. The amplitude of D₁ was recorded as the regularity score of that segment. Next, these regularity scores were smoothed by using a moving average filter with 18 data points (3 min/10 s) and a regularity plot was obtained.

Fig. 5 shows an example of x-axis accelerometer signal collected during the controlled portion of the study. The detected HMGs and their regularity scores (regularity plot) are also presented in Fig. 5.

Fig. 5.

(a) An example of the x-axis accelerometer signal collected during a control portion. (b) Detected HMGs. (c) The regularity of the HMGs.

3.8. Smoking event and its boundary

Significant changes on the HMG regularity plot can be used to detect smoking events and its boundaries. For this purpose, MAT-LAB’s findpeaks algorithm (by finding local maximum) was used to find all local maximum points on the HMG regularity plot. Each peak in the regularity plot was considered as a center of the smoking event if its value was greater than the predefined optimum threshold value (Section 3.9). The −6db bandwidth of each peak was considered as the start and the end of the smoking episode (Fig. 6). These detected smoking events were then compared with the ground truth information (smartphone registration) to evaluate the proposed method. The performance of the method was obtained by means of true positives (TPs), false positives (FPs), false negatives (FNs), accuracy (Acc), recall (Rec), precision (Prec), and F1-score (F₁) metrics;

$$Acc=TPs/\left(TPs+FPs+FNs\right)$$ (2)

$$Rec=TPs/\left(TPs+FNs\right)$$ (3)

$$Prec=TPs/\left(TPs+FPs\right)$$ (4)

$$F_1=\left(2\cdot Rec\cdot Prec\right)/\left(Rec+Prec\right)$$ (5)

Fig. 6.

Smoking event and boundary detection from regularity plot.

3.9. Optimal threshold level for smoking event detection

To guarantee person-independent validation, an optimum threshold level was determined by using Leave-One-Subject-Out (LOSO) cross-validation procedure instead of k-fold cross-validation. In the LOSO approach, training and testing datasets do not have the samples from the same subject, and this is a very common strategy to see the performance of the model for a new subject’s data. Having a data set of 35 subjects, 34 were chosen for the training of the threshold level selection procedure, and the remaining subject’s data were used for the test. For each training set, twenty different threshold levels from 100% to 5% of mean peak value were tested. The threshold level that provided the maximum F1-score was set as the optimal threshold of the training set. This procedure was repeated for 34 subsets and the average value of the optimal threshold levels, 60% of mean, was defined as the threshold level.

3.10. Regularity of hand gestures with an Instrumented lighter based approach

Data from an instrumented lighter were used to evaluate the applicability of the regularity of hand gestures as a supportive tool in multi-sensor based approach. The lighting event was considered as the start of a cigarette smoking event and 9 min (95% upper confidence duration of the ground smoking information in the controlled portion of the study) were considered as the duration of the smoking event. Regularity scores of HMGs were then calculated for each smoking event. If a regularity score was lower than the predefined threshold level, this detection was eliminated. The accuracy of two approaches i.e. employing only lighter and lighter and regularity of hand gestures was compared against the ground truth information (self-reported smoking event in the smartphone).

4. Results

4.1. Results for the controlled portion

Fig. 7 shows the activity-wise distribution of the regularity of HMGs and dominant period of HMGs across all subjects in the controlled portion of the study.

Fig. 7.

(a) Regularity of HMGs (b) dominant period of HMGs in each activity across all subjects in the controlled portion.

Table 2 shows the results of an algorithm for smoking event detection. F1-score and accuracy were 81% and 68% respectively in the controlled portion of the study.

Table 2.

Smoking detection results in the controlled portion.

Smoking event detection
TPS	FPS	FNS	TNS	Rec	Prec	F1	Acc
137	59	3	0	0.97	0.69	0.81	0.98

Fig. 8 shows the number of false predicted smoking events per activity collected during the controlled portion of the study. In this figure, the unconstrained activities refer to activities that were performed in inter-cigarette intervals. This figure indicates that eating and unconstrained activities were the main sources of FPs.

Fig. 8.

Number of false positive predictions per activity in the controlled portion.

4.2. Results for the free-living portion

The proposed algorithm applied to the free-living data of the study predicted 448 smoking events with 33% accuracy and a 49% F1-score. Using only lighter data, the accuracy and F1-score were 81% and 89%, respectively. With the combination of lighter data and HMG regularity, the performance metrics reached 83% accuracy and a 91% F1-score. Table 3 shows the smoking detection results from the free-living portion of the study.

Table 3.

Smoking detection results in the free-living portion.

	TPS	FPS	FNS	TNS	Rec	Prec	F1	Acc
HMG regularity	185	263	110	0	0.62	0.41	0.49	0.33
Only lighter data	273	42	22	0	0.92	0.86	0.89	0.81
Lighter + HMG regularity	272	30	23	0	0.92	0.90	0.91	0.83

5. Discussion

The primary purpose of this study was to examine the regularity of HMGs as an indicator of smoking during daily-life. The study is the first to demonstrate how HMGs data can be analyzed by unbiased autocorrelation procedures to obtain HMG regularity. The results indicated that, under controlled conditions, smoking-related HMGs were highly regular. Fig. 7a indicates that the regularity of HMGs during all four smoking activities was substantially higher than other activities, although the regularity value during eating was comparable to smoking while sitting. Also, Fig. 7b indicates that, during eating activities, the dominant period of HMGs was longer than smoking activities. This result suggests that the time difference between two eating and smoking HMGs could be used as another predictor of smoking activity.

Eating activity contributed most to smoking false positives (50.9%). The unconstrained activities between smoking events were the next largest source of false positives (35%). An intriguing result was that, when the subjects were smoking during a conversation, their HMGs were less regular than during other smoking activities (Fig. 7a).

Table 3 shows that the performance of the proposed method in during free-living conditions was low compared to the controlled portion of the study (F-score 49%, Accuracy 33%). It should be noted that these results were achieved with a very basic classifier that only uses the regularity of HMGs. According to the self-report data, 41 false smoking events were detected when subjects were eating, while 100 false smoking events were detected when they were doing some physical activity. These results suggests that, besides eating activities, physical activities and smoking activities have a similar pattern of HMG regularity. Overall, the data indicate that HMG regularity is not sufficient for the detection of smoking events with a high accuracy under free-living conditions. But the regularity of HMGs could be an additional feature or secondary tool for improving the performance of smoking detection methods. This was illustrated in the present study by the improvement in detection metrics by the combination of lighter event data and HMG regularity.

The main advantages of the proposed method are that it uses data from only one axis accelerometer to detect HMGs, and it uses the regularity value of HMGs as a single feature for the classifier. These two properties make the proposed method a suitable tool for real-time applications with low-cost hardware.

The results of present study show that regularities of HMGs are quite different for different activities. In future studies, machine learning methods can be applied to obtain better prediction results with more extracted features from both acceleration and autocorrelation signals.

6. Conclusion

This study describes the first implementation of an approach that uses regularity of HMGs for identifying smoking activities. HMGs were detected by using single-axis accelerometer sensor data from the PACT2.0 wrist module. The regularity of hand gestures was calculated using an unbiased autocorrelation procedure. The proposed approach achieved a very high accuracy for smoking detection in the controlled portion of the study, but only moderate accuracy with data collected during the free-living portion of the study. The accuracy of smoking event detection was improved during the free-living portion with the combination of data from an instrumented lighter and HMG regularity. This work showed that, under constrained conditions, HMGs exhibit a relatively high degree of regularity for smoking activities and can provide a useful feature for smoking detection. But, under unconstrained, free living conditions, HMG regularity it is not sufficient for the accurate detection of smoking events without using additional features or tools.