Posture and Activity Recognition and Energy Expenditure Prediction in a Wearable Platform

Abstract

The use of wearable sensors coupled with the processing power of mobile phones may be an attractive way to provide real-time feedback about physical activity and energy expenditure (EE). Here we describe the use of a shoe-based wearable sensor system (SmartShoe) with a mobile phone for real-time recognition of various postures/physical activities and the resulting EE. To deal with processing power and memory limitations of the phone, we compare use of Support Vector Machines (SVM), Multinomial Logistic Discrimination (MLD), and Multi-Layer Perceptrons (MLP) for posture and activity classification followed by activity-branched EE estimation. The algorithms were validated using data from 15 subjects who performed up to 15 different activities of daily living during a four-hour stay in a room calorimeter. MLD and MLP demonstrated activity classification accuracy virtually identical to SVM (~95%), while reducing the running time and the memory requirements by a factor of >10³. Comparison of perminute EE estimation using activity-branched models resulted in accurate EE prediction (RMSE=0.78 kcal/min for SVM and MLD activity classification, 0.77 kcal/min for MLP, vs. RMSE of 0.75 kcal/min for manual annotation). These results suggest that low-power computational algorithms can be successfully used for real-time physical activity monitoring and EE prediction on a wearable platform.

I. Introduction

Physical activity (PA), including the type, intensity and duration of activities, is an important component of programs designed to prevent/treat metabolic syndrome and obesity. Objectively monitoring physical activity and the associated estimates of instantaneous and cumulative energy expenditure (EE) can provide important feedback that would allow a person to regulate his/her PA and energy balance to maintain or achieve a healthy weight/lifestyle. Several physical activity monitors have been developed that provide continuous, real-time feedback of EE through heart rate [1], accelerometry [2], [3] and/or multi-sensor measurements [4]. While some of these products provide reasonably accurate assessments of EE, they have the disadvantage of being obtrusive and/or uncomfortable to wear continuously.

An attractive and practical opportunity for convenient, unobtrusive PA monitoring is to use mobile devices (e.g. smart phones) equipped with software that provides an instantaneous display of the amount of physical activity and estimated EE. Recent research has demonstrated the implementation of software for posture/activity recognition and EE prediction in cell phones using built-in 3-axis accelerometer [5]–[8]. However, the accuracy of posture and activity recognition and EE estimation is likely to be limited due to the wide variety of ways a cell phone can be worn or carried by a person.

We have recently developed a wireless shoe-based sensor system (SmartShoe) that records insole pressure and foot acceleration data. Several other studies also reported using shoe-based sensors [9]–[14] and a few commercial shoe-based products were developed [15], [16] for use in clinical or movement applications. However, these products are limited by only detecting gait characteristics and pressure distribution across the foot, and do not include activity classification or EE prediction. The attractiveness of using the SmartShoe for EE prediction is due to the fact that the wireless shoe-based sensor provides pressure monitoring of key body weight support points; is capable of differentiating static postures (such as sitting and standing), weight-bearing and non-weight bearing activities (such as walking and cycling); and is unobtrusive, lightweight and easy to use. The shoe sensor data can be used for posture/activity classification [17] to recogniz.e activities that may present challenges to other approaches (for example, differentiate Sit, Stand, Walk/Jog, Ascend Stairs, Descend Stairs or Cycle). The posture/activity classification can then be utilized in activity-branched EE prediction models [18], [19] that rely on the branching techniques similar to those developed for accelerometers and heart rate monitors [20]–[26].

Methods for Posture and Activity Classification and EE prediction (PAC/EE) reported in [17], [18] are not well suited for use on a smartphone due to computational intensity and high memory requirements. A combination of SmartShoe sensors, efficient PAC/EE algorithms with the audio/visual capabilities of a modern smartphone can potentially lead to the development of biofeedback-based interventions for increasing physical activity and weight management. Therefore, it is highly desirable to execute PAC/EE algorithms on a mobile computing platform in real time.

This study investigates the implementation of PAC/EE models that considerably reduce computational intensity and memory requirements and can execute on a modern smart phone and provide proactive, device-initiated feedback in real time. The execution time constraints define the need to leave a substantial portion of running time left to the cell phone system processes to reduce the battery drain. Memory constraints demand minimization of storage of both the coefficients for classification and regression models, and the sensor data. Several computational models are considered. First, a Support Vector Machine (SVM) classifier is developed as the baseline measurement for computational and space complexity. Next, Multinomial Logistic Discrimination (MLD) and Multi-Layer Perceptron (MLP) models are developed as lightweight generalized linear and nonlinear alternatives to SVM, respectively. The impact of the classification model on the accuracy of activity-branched EE estimation is investigated by comparing the EE prediction from activity annotations performed by SVM, MLD, MLP models and a human observer. The proposed PAC/EE models are validated in a room calorimeter study with 15 subjects performing various activities of daily living.

The paper is organized as follows. Section II describes the study design and the data, followed by the methods used for model training and evaluation. Section III provides results using data from 15 subjects. Sections IV-V present a discussion and conclusions.

II. Methods

A. Subjects and protocol

Nineteen subjects (10 male, 9 female) were recruited to participate in this study. The desired sample size of 20 was estimated based on two primary outcomes, recognition rate and EE agreement, with the goal of achieving 90% power in detecting a difference of approximately 2 percentage points. The protocol was approved by the Colorado State University Institutional Review Board. Subjects completed a physical activity and health history questionnaire [27], and were determined to be in good health by a physician. Based on self-report, subjects were physically inactive to moderately active (less than six hours of physical exercise per week), not taking any medications known to alter the metabolism, and weight stable over the past six months.

Each subject performed one 4-hour stay in a room calorimeter following a 4-hour fast. Metabolic data were collected while each individual performed a series of randomly assigned postures and activities that were recorded by a video camera (Table I). Subjects wore SmartShoe of appropriate size ranging from female size 7 to male size 12 (US). Data collection began with a 30-minute equilibration to assure that gas exchange within the calorimeter was representative of the current subject. This period of data collection was not used in the analysis. The remaining 3.5 hours of data collection started with a pre-determined period of acitvities including 20 minutes of lying quietly in the supine position, 20 minutes of sitting and watching TV, 20 minutes of sitting while using a computer, 10 minutes of quiet standing and 10 minutes of active standing. Subjects were then asked to perform six of eight randomly assigned activities for 10 minutes each from the set shown in Table I. The last hour of data collection consisted of free-living activities of the individual’s choice. Ten minutes of data collection were reserved for transitions between the various activities.

Table I.

Activities in the protocol

Activity	Description	Time	Class label
Equilibration	Quiet resting, data excluded	30 min	N/A
Supine	laying on bed, data excluded	20 min	N/A
Sitting	watching TV	20 min	Sit
	performing computer work	20 min	Sit
Standing	Quiet	10 min	Stand
	Active	10 min	Stand
Random assignment; 6 of 8 possible activities	Walking, 2.5mph	10 min each; 60 min total	Walk/Jog
	Walking, 3.5mph		Walk/Jog
	Walking uphill, 2.5% grade, 2.5mph		Walk/Jog
	Stepping		Walk/Jog
	Sweeping		Stand
	Cycling, 75W		Cycle
	Standing		Stand
	Sitting		Sit
Free-living	Any of the above activities, self-selected pace and posture	60 min, or until 4 hours total	Selected class

Sensor data from each experiment were manually annotated with every minute of each activity labeled from the set activity classes shown in Table I. The annotated data were later used for training and validation of PAC/EE algorithms as well as in the manually-branched EE estimation model.

Oxygen consumption and carbon dioxide production were recorded using the whole-room indirect calorimeter located in the Clinical Translational Research Center of the University of the Colorado Hospital [28]. EE and substrate oxidation were calculated using the non-protein RQ [29]. Measured oxygen consumption and associated EE was recorded on per minute basis. The average EE of the last 5 minutes of the supine period for each subject was used as an estimate of resting metabolic rate/EE in kcal/min [30]. The last minute of resting EE was used to perform conversion to METs.

Four subjects had incomplete data as a result of an error in the recording system, shoe sensors or power grid failures during the experiment and were excluded from the analysis. The anthropometric characteristics for the remaining 15 subjects are given in Table II. The resulting dataset used for development of the models described in this study contained approximately 52.5 hours of data.

Table II.

Subject characteristics

Characteristic	Males (n = 9)		Females (n = 6)
	Mean(SD)	Range	Mean(SD)	Range
Age, years	28.1(6.9)	(20-38)	23.7(3.1)	(20-29)
Height, cm	178.1(10.2)	(162.6-193)	167.5(8.5)	(157.5-180)
Weight, kg	79.3(16.7)	(57.6-98.5)	72.0(18.0)	(49.2-99)
BMI	24.8(3.6)	(19.6-31.1)	25.6(6.3)	(18.7-37.3)

B. Sensors

The sensor data were collected by a SmartShoe with embedded sensor system (Figure 1). Five force-sensitive resistors were integrated in a flexible insole and positioned under the critical points of contact: heel, metatarsal bones and the great toe (hallux), allowing for differentiation of static postures and weight bearing and non-weight bearing activities. The acceleration data were collected from a 3-dimensional MEMS accelerometer (ADXL335) positioned on the back of the shoe. Pressure and acceleration data were sampled at 400Hz, downsampled to 25Hz by averaging of 16 consecutive samples and sent over a Bluetooth link to a smartphone storing the signals for further processing.

Figure 1.

SmartShoe device: (left) Overall view of the shoe device with attached accelerometer, battery and power switch on the back; (right) Pressure-sensitive insole with 5 pressure sensors: heel (1), 3rd metatarsal head (2), 1st metatarsal head (3), 5th metatarsal head (4), hallux (5).

C. The information flow

The information flow in the proposed PAC/EE models is outlined in Figure 2. Algorithmically the processing of the signals consists of two major steps. First, sensor data from shoes are used to classify every 2-second time period into one of the four classes: Sit, Stand, Walk/Jog or Cycle. The choice of classes in the model was stipulated by the fact that the majority of a person’s time could be described as one of these posture/activities (except cycling, which presents a separate class due to its different EE requirement as compared with walking). Second, the same sensor data along with the result of the posture/activity classification are used to predict EE of 1-minute time intervals by selecting an appropriate branch model (Sit, Stand, Walk/Jog or Cycle) and applying it to the features extracted from the sensor data

Figure 2.

The information flow in the shoe-based system for EE prediction.

D. Posture and Activity classification

The posture/activity classification algorithms automatically recognize activities and assign class labels from the set of {Sit, Stand, Walk/Jog, Cycle} to 2-second periods of sensor data. The following features were derived from the raw 2-second signal (length 50 samples) for each sensor: mean value (mean), entropy (ent) and standard deviation (std). Entropy H of the distribution X of signal values was computed as H(X) = −Σp_k log p_k, where p_k is the relative frequency of values fallen into the k^th interval (out of 20 equally sized intervals) in the sample distribution of signal values. The features were normalized to the range [0,1]. The extraction of the above features provided a total of 24 possible features (8 sensors per shoe × 3 features per sensor) for the model. Out of these 24 features, the set of the best features was selected using the forward selection procedure applied to the MLD classification model with the accuracy of the validation data set as the criterion. The selected set of features was then used for all classification models, thus allowing us to investigate the dependency of the classification accuracy on classifier type and eliminating the potential inconsistencies due to possible selection of different feature sets by different classification models.

The Support Vector Machine classification model [31] with radial basis kernels was used as the baseline for comparison. SVM is considered as the most robust, but at the same time most computationally intensive and space consuming model of those considered. LibSVM for MATLAB [32] was used for training the SVM model. Parameters of the radial basis kernel SVM were found in a grid search.

To meet the time and space requirements of mobile devices, two classification approaches were considered that are less time and computationally intensive.

First is a multinomial logistic discrimination model [33] with three logit functions being the linear combinations of possible predictors (Cycle was selected as the baseline class):

$$Logit\left(p_i\right)=\text{log}\left(\frac{p_i}{p_{CYCLE}}\right)=X{\beta }_i,i=\{Sit,Stand,Walk∕Jog\}.$$

The MLD model presents the lowest computational complexity, but provides the simplest, generalized linear decision boundary in multi-class classification. The training of the MLD model was performed using MATLAB Statistics Toolbox.

Second is multi-layer perceptron model [34] which is a feedforward artificial neural network mapping a set of inputs to the desired output. The MLP model can be more computationally intensive than MLD, but in proper configurations can potentially map decision boundaries as complex as those of SVM. Stand-alone version of the Fast Artificial Neural Network Library (FANN) [35] was used for training of the MLP model. The network architecture of 1 hidden layer with hidden 4 neurons was established experimentally as the simplest network producing satisfactory results. The training parameters were autodetected by the FANN tool [36]: sigmoid activation functions on the hidden layer, linear activation functions on the output layer, simulated annealing as the training method.

Training of all classification models was done using leave-one-out procedure in which data from all subjects but one are used for training and determining of the model parameters, if necessary, and the remaining subject is used only for validation.

E. Energy Expenditure prediction

Four models for EE prediction were developed using linear regression branched by activity types. To match the time resolution of the room calorimeter data, the EE estimation was done in 1-minute intervals. The models varied in the source of activity branching: manual annotation by a human observer or SVM, MLD, or MLP classification of the sensor data acquired by SmartShoe. For the latter case, the sensor signals with 1 minute duration and time stamp matching the time of the room calorimeter were used in combination with SVM, MLD or MLP model to produce thirty posture/activity labels (one for each of the 2 second intervals in that minute). Out of these thirty predictions the activity with the highest frequency was selected to represent the activity for the entire 1 minute interval and choose the branch for EE model prediction.

All EE prediction models were trained in the identical manner. Every minute was classified into one of 4 activity branches (Sit, Stand, Walk/Jog and Cycle). Next, a linear regression for each of the four branches was trained using ordinary least squares. The following features derived from the sensor signals were used as predictors: (max) maximal value of the signal, used only for the pressure sensors, (zc) number of crossings of the median of the signal, (std) standard deviation of the signal, and (ent) entropy of the signal. To provide the robustness against the pressure sensor failures, the features for the 5 pressure sensors were combined by computing the medians of appropriate values: thus, the 5 max features for the pressure sensors were combined into a single predictor Sens_med(max), 5 zc features were combined into Sens_med(zc), 5 std and 5 ent features resulted in Sens_med(std) and Sens_med(ent) respectively. Anthropometric chracteristics such Weight, BMI and log(BMI), as well as Age were also included as predictors.

The selection of the best set of predictors was done using a forward selection procedure with the following criteria: the best model had to have a low value for the Akaike Information Criterion (AIC), a high adjusted coefficient determination ($R_{ADJ}^2$) and a low root-mean squared error.

Validation of EE prediction was done using the leave-one-out approach: for each subject all epochs corresponding to that subject were excluded from the training set and the resulting trained model was used to predict EE for every epoch of the left out subject.

F. Model comparison

The developed PAC/EE models were compared in terms of computational burden, memory requirements, and classification and EE estimation accuracy. All models were coded in Visual C++, compiled using Windows Phone SDK, and benchmarked on an entry-level BLU Win Jr phone running Windows 8.1 Phone OS on a 1.2GHz quad-core Cortex-A7 processor.

Computational time and space requirements of different models were compared using one or more of the following metrics: 1) space complexity (number of elementary objects needed to be stored during execution) for the model’s testing stage, 2) space requirements for model storage (number of elementary floating point objects to be stored between execution steps), 3) time complexity (number of elementary operations needed to be performed during execution) for the model’s testing stage, 4) time (ms) to run the model on the Windows smart phone, measured as an average over 100000 iterations of the code. The MLD and MLP models were benchmarked as actual predictive models implemented in C++. The libSVM code had compatibility issues with compilation on Windows Phone platform, therefore, SVM running time was evaluated on a simulated C++ code with the same type and number of iterations performed on the same type of variables.

Activity classification accuracy was assessed by comparing automatic classification by SVM, MLD and MLP with classification by a human observer who manually annotated the experiment and expressed as a per-minute cumulative confusion matrix pooled from the subject population with overall classification accuracy being the ratio of the sum of diagonal elements in the confusion matrix to the sum of all elements in the matrix.

Accuracy of EE prediction was assessed through comparison with the EE measurements of the room calorimeter. The models branched by SVM, MLD, MLP classifiers and manual annotation were compared. The EE was expressed in kcal/min and Metabolic Equivalents (METs) that were computed as the multiples of measured resting EE. The following characteristics were used: a) total error of prediction, computed as the average total error of prediction across all subjects: $totERR=\frac{1}{N}{\sum }_{i=1}^N\frac{\mid totE{E}_i-totE{E}_i^{pred}\mid }{totE{E}_i}$, where N is the number of subjects, $totE{E}_i={\sum }_{t=1}^{T}E{E}_{ti}$ is total EE measured by the room calorimeter and $totE{E}_i^{pred}={\sum }_{t=1}^{T}E{E}_{ti}^{pred}$ is total EE predicted by a model, and where T is the number of 1 minute EE values in an experiment, b) per-minute root-mean squared error (RMSE): $RMSE=\sqrt{\frac{1}{2}{\sum }_{t=1}^T{(E{E}_t-E{E}_t^{pred})}^2}$, where EE_t is EE measured by room calorimeter for a given 1-minute interval; $E{E}_t^{pred}$ is EE estimated by a model for the same 1-minute interval.

III. Results

The feature selection for posture and activity classification resulted in the selection of 12 features, 6 from the accelerometer and 6 from the pressure sensors: {PSens1_mean, PSens2_mean, PSens5_mean, Acc1_ent, Acc2_ent, Acc3_ent,, Acc1_std, Acc2_std, Acc3_std, PSens1_std, PSens2_std, PSens5_std } where PSensX is pressure sensor X and AccY is Y-s dimension of the accelerometer. Running time for feature computation on the smartphone was 1.56ms.

Comparison of the time and space requirement of the SVM, MLD and MLP modes is shown in Table III. The space required to store MLD model is equivalent to storing 13×3 element matrix of floating point numbers. MLP requires storing 13×4 matrix of weight for the connections between inputs and hidden layers and a 4×5 matrix for the connections between the hidden layer and the output layer. The SVM model needs to store 9234×12 matrix of support vectors (as determined during model training) and a 9234×4 element matrix of discrimination function coefficients. Thus, the relative difference in the model’s storage space requirements is at least 10³-fold for the SVM model compared to the MLD and MLP models.

Table III.

Time and space complexity for the posture/activity classification models

Posture/ activity classification method	Space complexity	Space requirement, floating point numbers	Recognition Time Complexity	Actual running time, ms
SVM	O(ns)	147744	O(ns)	66.1
MLD	O(n)	39	O(n)	0.022
MLP1	O(n)	72	O(n)	0.045

The accuracies of posture and activity classification for the SVM, MLD and MLP models are presented in Table IV - Table VI. The overall accuracy for all classifiers were virtually identical, ≥95%.

Table IV.

Population-cumulative confusion matrix of SVM classifier.

Predicted class
		Sit	Stand	Walk/Jog	Cycle	Class- specific recall
Actual class	Sit	1232	35	4	1	0.97
	Stand	31	477	7	4	0.92
	Walk/Jog	5	2	447	3	0.98
	Cycle	1	4	0	106	0.95
	Class- specific precision	0.97	0.92	0.98	0.93	0.96

Table VI.

Population-cumulative confusion matrix of MLP classifier.

Predicted class
		Sit	Stand	Walk/Jog	Cycle	Class- specific recall
Actual class	Sit	1228	28	2	2	0.97
	Stand	34	484	5	4	0.92
	Walk/Jog	5	2	451	10	0.96
	Cycle	2	4	0	98	0.94
	Class- specific precision	0.97	0.93	0.98	0.86	0.96

The comparison of accuracy of EE prediction by branched EE is shown in Table VII. As an example, Table VIII shows the best set of predictors and equations for the EE prediction model branched by the activity by MLP. Running time of the EE regression on the smartphone (including additional feature computation as required by the EE models) was 2.54ms.

Table VII.

Comparison of EE prediction performance

Branching Classifier	Total Error, %	RMSE,kcal/min(METs)		Limits of agreement, kcal/min
Manual activity annotation		Sit model:	0.58 (0.40)	−1.14,1.17
		Stand model:	0.63 (0.45)	−1.24,1.29
		Walk/Jog model:	1.06 (0.76)	−2.08,2.16
		Cycle model:	1.39 (0.98)	−2.95,2.60
	5.0	Weighted average:	0.75 (0.54)	−1.58,1.60
SVM		Sit model:	0.58 (0.42)	−1.17,1.17
		Stand model:	0.70 (0.50)	−1.38,1.43
		Walk/Jog model:	1.06 (0.77)	−2.09,2.17
		Cycle model:	1.49 (1.05)	−3.16,2.76
	5.0	Weighted average:	0.78 (0.56)	−1.64,1.65
MLD		Sit model:	0.60 (0.44)	−1.21,1.20
		Stand model:	0.69 (0.49)	−1.37,1.40
		Walk/Jog model:	1.08 (0.78)	−2.14,2.19
		Cycle model:	1.36 (0.89)	−2.79,2.65
	5.5	Weighted average:	0.78 (0.55)	−1.62,1.63
MLP		Sit model:	0.58 (0.42)	−1.16,1.18
		Stand model:	0.69 (0.49)	−1.36,1.40
		Walk/Jog model:	1.07 (0.77)	−2.12,2.17
		Cycle model:	1.40 (0.91)	−2.87,2.74
	5.1	Weighted average:	0.77 (0.55)	−1.61,1.63

Table VIII.

EE prediction model branched by the MLP classifier

Branch	EE regression (kcal/min)
Sit	EE = 0.22 +6.1·10−3Weight +9.2·10−2 -log(BMI) + 32.4· Sensmed(sld) +0.68·Acc3zc +10−3 ·Acc3sld+0.59·Acc1zc +1.8·10−3 ·Acc1std−9.1 · 10−4 · Acc2std
Stand	EE = 2.8+2.6·10−2·Weight – 1.24· log(BMI) + 5.2·10−3· Acc1std +14.9· Sensmed(std)+ 8.2·10−4· Sensmed(frg) +1.84· Acc1zc
Walk/ Jog	EE = −3.1 + 63·10−2·Weight + 0.08·log(BMI) + 43.1· Sensmed(sld)+ 6.64·Acc3zc
Cycle	EE = 11.2 + 3.0·10−2·Weight–1.7·log(BMI) −0.06·Age −5.0·10−4·Sensmed(max)

Figure 3 shows correlations and Bland-Altman plots from all 1-min instances from all 15 subjects for the EE prediction model branched by MLP classifier. These plots demonstrate high correlation of predicted EE to actual EE (R² = 0.88) and that the errors of prediction do not follow any specific pattern, and are not biased to either side (under- or over-prediction) with average bias being negligibly small (−0.01kcal/min).

Figure 3.

Correlation (top) and Bland-Altman (bottom) plots for the EE model branched by the MLP classifier.

IV. Discussion

The main goal of this study is the development of computationally efficient PAC/EE algorithms allowing real-time execution on a mobile phone. Given that mobile devices have limits on computational power imposed by the battery life of the device, the computational and memory demands of the algorithms need to be taken into consideration along with the accuracy of activity classification and EE estimation.

The MLD-based and MLP-based algorithms for posture and activity classification were considered as alternatives to SVM classification. Both of these methods achieved a substantial improvement, reducing the run time by a factor >10³ (from 66.1ms for SVM to 0.022ms for MLD and 0.045ms for MLP), and produced a very efficient and accurate classification. Very high processor load (O(ns)) with SVM implementation would be detrimental to the latency of the output, computing resources available to other programs and battery life. MLD and MLP substantially reduce the impact of real-time posture and activity classification on computational resources available on a mobile device, thus increasing battery life. Assuming 24-hour operation, and classification of every 2s interval, the MLD/MLP would consume only 0.95s and 1.94s of processor time per day, compared to 2855.5s (0.8 hour) consumed by SVM. The potential savings in battery life achieved by the MLD and MLP can be estimated as 20%-10%, assuming 4-8 hours of battery life under full processor load.

Use of MLD/MLP also substantially reduced the memory requirements as the amount of memory to store scalar discriminant coefficients and the weights of the connections between neurons is far less than the amount of memory required to store hundreds of multi-dimensional support vectors needed by SVM: the space storage due to the introduction of the MLD in place of the SVM was also reduced >10³ times. The reduction in memory requirements could be especially beneficial if classification is implemented directly on the wearable sensor, which has substantially less memory than a modern smartphone.

The accuracy of posture and activity classification (95%) obtained by MLD and MLP classifiers was comparable both to SVM trained on the same dataset (96%) and to previously reported 95%-98% [17], [37] that included community studies in a neurologically impaired population. The 2-second classification interval used in the shoe sensors allows to finely resolve posture transitions and short bursts of activity, however, in this study it had to be aligned with 1-minute EE estimates produced by the room calorimeter. The somewhat lower accuracy of posture and activity classification demonstrated in this study could be explained by transitions between activities where a class label is assigned to a whole minute although several different activities may happen within this minute. With at least 12 transitions in any given visit, this may generate a substantial difference between manual annotation and automatic class labels. Another notable contributor to the error, appearing in all classification approaches, is confusion between standing and sitting. The potential reason is that the classification algorithms are trained on population data and thus do not require individual calibration. This feature is convenient, however, it may potentially create some confusion for light-weight subjects that exhibit less pressure on the shoe sensors. Simple individual calibration of sitting vs. standing should largely eliminate this source of error.

It should be noted that cycling was differentiated from walking with high degree of precision. Such differentiation is typically considered difficult for accelerometer-based activity monitor and highlights the benefits of utilizing of pressure sensors and in-shoe sensor locations for differentiation of weight-bearing and non-weight-bearing activities.

Use of the activity–branched EE estimation produced consistent results across all models tested. The manually annotated activity labels presented the best case scenario in selecting the appropriate branch of the EE model and served as the gold standard. As expected, the manually branched EE model produced the best results with 5.0% of total error, and RMSE of 0.75 kcal/min and 0.54 METs. The SVM-branched EE model was the next most accurate (5.0%, 0.78 kcal/min and 0.56METs), followed by the MLP (5.1%, 0.77 kcal/min and 0.55METs) and MLD (5.5%, 0.78kcal/min and 0.55METs). It is clear that more complex models produce slightly better results, however, the differences between approaches are practically negligible, thus allowing for application of computationally lightweight MLD/MLP without loss of fidelity.

The overall high accuracy of EE estimation in all of the tested EE models also highlights the benefits of the proposed wearable system relying on in-shoe sensors. Our previous research [18] has clearly demonstrated an increase in estimation accuracy by models utilizing pressure sensor data which most likely is result of accurate branching between EE models between weight-bearing and non-weight-bearing activities.

The sensor data in this study were acquired by real-time streaming via Bluetooth. Modern methods of wireless communication, such as Bluetooth Smart, can maintain a sensor operational and streaming data for several days on miniature batteries with small capacities [38]. The continuous streaming, combined with on-phone computations provides the ability to generate real-time feedback in behavioral intervention studies while preserving the raw sensor data for research applications and further off-line processing. Data streaming also allows for higher-accuracy calculations by utilizing the floating point arithmetics on the phone, as almost all sensor-class microcontrollers lack an FPU. A consumer-grade sensor may gain a longer battery life by rebalancing the computation between the sensor and the phone to minimize wireless traffic. The potential cost, however, may be a reduction in accuracy due to use of fixed point arithmetic on the sensor.

A potential limitation of this study is that EE of the subjects was measured under a specific (although randomized) activity protocol. The number of activities in free living is much greater and not so clear cut, thus the performance of the EE estimation models will need to be validated in community-living individuals, which is the goal of our future research. The miniaturization and integration of sensors and electronics in user-friendly insoles (SmartStep, [38]) should improve social acceptance of the device and enable community validation. The majority of the day for many of us is time spent outside of the home and most of this time is spent wearing the shoes. The inexpensive insole sensors can be used to instrument multiple shoes worn in different life situations (formal shoes, running shoes, outdoor shoes) and accurately estimate the activities and energy expenditure during the most active part of the day. The major benefit of the shoe monitoring system is that the sensors are integrated into an item that most of us wear anyway, thus virtually eliminating the wear burden typical of other monitoring devices. The downside is that in most cultures the shoes are not worn at home, and these periods should be approximated by models of sedentary behavior or measured by alternative means.

Another limitation is that the study was conducted on a population mostly consisting of young adults. Age and health factors are potentially important predictors and further studies are needed to study the performance of the proposed system in a wider population, including elderly and adolescents, morbidly obese and individuals with gait impairments.

Overall, the proposed wearable sensor system and the methodologies to classify postures and activities, and to estimate EE are sufficiently accurate and computationally lightweight to be utilized in modern mobile devices and provide real-time, proactive feedback on activity levels and energy expenditure. With increased penetration of mobile devices into all aspects of life and emergence of low-power wireless technologies (Bluetooth Smart), real-time feedback capabilities present new opportunities for behavioral interventions aimed at maintaining energy balance or achieving a healthy lifestyle. Further work may include consideration of additional lightweight classifiers (e.g. decision trees) for performing classification directly on the sensor module, investigation of energy costs of “compute vs. transmit” on the sensor nodes, consideration of introducing additional activities into the branched model and practical implementation of the real-time feedback mechanisms based on behavioral science.

V. Conclusion

This paper introduced a wearable system consisting of the SmartShoe sensor system and a mobile phone for signal processing, pattern recognition and real-time user feedback of expended calories and other physical activity information. Use of logistic discrimination or multi-level perceptron instead of previously reported support vector machine reduced the execution time and memory requirements a factor of >10³ while maintaining comparable accuracy of classification and energy expenditure estimation. The proposed algorithms were validated on over 50 hours of data from 15 subjects performing a wide range of activities in a room calorimeter and indicated accurate EE reduction. These results pave the way for an implementation of physical activity monitoring and EE prediction system with real-time biofeedback on a wearable smart phone based system which could potentially be used in physical activity interventions.

Table V.

Population-cumulative confusion matrix of MLD classifier.

Predicted class
		Sit	Stand	Walk/Jog	Cycle	Class- specific recall
Actual class	Sit	1235	33	12	4	0.96
	Stand	28	477	12	4	0.92
	Walk/Jog	5	4	434	9	0.96
	Cycle	1	4	0	97	0.95
	Class- specific precision	0.97	0.92	0.95	0.85	0.95