A contactless monitoring system for accurately predicting energy expenditure during treadmill walking based on an ensemble neural network

Summary

The monitoring of treadmill walking energy expenditure (EE) plays an important role in health evaluations and management, particularly in older individuals and those with chronic diseases. However, universal and highly accurate prediction methods for walking EE are still lacking. In this paper, we propose an ensemble neural network (ENN) model that predicts the treadmill walking EE of younger and older adults and stroke survivors with high precision based on easy-to-obtain features. Compared with previous studies, the proposed model reduced the estimation error by 13.95% and 66.20% for stroke survivors and younger adults, respectively. Furthermore, a contactless monitoring system was developed based on Kinect, mm-wave radar, and ENN algorithms, and the treadmill walking EE was monitored in real time. This ENN model and monitoring system can be combined with smart devices and treadmill, making them suitable for evaluating, monitoring, and tracking changes in health during exercise and in rehabilitation environments.

Graphical abstract

Highlights

• •Proposed an ensemble neural network to predict treadmill walking energy expenditure
• •The model enables more precise and universal prediction than existing methods
• •Developed contactless system for monitoring treadmill walking energy expenditure

Applied sciences; Health sciences; Natural sciences

Introduction

The World Health Organization (WHO) recommends that people of all ages engage in regular and adequate physical activity (PA) to overcome inactivity, which is the fourth-largest cause of mortality.^1,2 PA improves musculoskeletal and cardiopulmonary health,¹ reduces mortality,³ and provides an effective method to prevent and rehabilitate chronic diseases.⁴ A useful tool is requested to objectively monitor PA using factors such as energy expenditure (EE).⁵ Exercise intensity is the key factor for ensuring safety and obtaining health benefits^6,7 and is often expressed in terms of EE.⁸ A recent study reported a U-shaped relationship between exercise intensity and adverse cardiovascular events,⁹ which further emphasizes the importance of monitoring EE in PA. Walking is the most common moderate-intensity PA reported among adults,¹⁰ and it is strongly associated with good health and quality of life.¹¹ Treadmill walking is a common and important method for individuals with sedentary behavior, elderly, and stroke patients to engage in PA and rehabilitation training indoors. Monitoring the treadmill walking EE in those individuals can provide an effective tool for active health management, evaluating the body’s health and physiological status, tracking changes in aerobic fitness, and investigating dose-response relationships.^2,12,13

Walking EE can be accurately estimated using indirect calorimetry (e.g., respirometry).^14,15 However, this method is not applicable for monitoring EE in the exercise and rehabilitation training environment because it requires breath-by-breath measurements from uncomfortable wearing masks and expensive equipment.^16,17 Note that respirometry offers an accurate estimate reference for prediction models of walking EE.¹⁸ The unsupervised problem of walking EE has prompted extensive efforts to develop predictive equations. Despite numerous efforts,^{15,16,18,19,20,21} prediction methods with broad applicability and high accuracy for walking EE remain lacking. Developing prediction models with broad applicability and high accuracy for walking EE is challenging because of the complex time-varying and nonlinear nature of the human body, the complex and diverse factors affecting EE, and the different prediction factors and function relationships among different populations.^22,23

Over the last few decades, several predictive equations have been proposed through investigating the functional relationship between prediction factors (i.e., anthropometry data and motion information) and EE.^24,25,26 Generally, predictive equations are developed for homogeneous populations using best-fit approaches.^18,23,27 For example, a linear function of walking speed and EE was established based on data from healthy adults,²⁸ and an exponential function of walking speed and oxygen cost was proposed based on data from stroke survivors.²¹ The prediction equation for walking EE developed using this method is only applicable to a certain population and is not universal. Specifically, the prediction equation based on the data of healthy adults is not applicable to individuals with chronic diseases because previous studies have reported that stroke increases the EE of treadmill walking by 2-fold compared with normal controls.^29,30 Even for the same population, the form of prediction equation also has heterogeneity; the American College of Sports Medicine established a linear function to predict walking EE in healthy adults,²⁸ whereas Pandolf et al.³¹ established a polynomial function containing exponential terms.^27,31 More importantly, computations that input identical prediction factor values for different theoretical constant-speed trials for the same population identified differences between the equations as large as 37% during walking.²⁴ Therefore, the current predictive algorithms for walking EE are largely unavailable.

With the development of effective wearable sensors and the incorporation of predictive algorithms, the importance and potential of these algorithms for health monitoring are greater.²⁷ Estimating walking EE using wearable devices is a convenient option, considering the relationship between steps count, speed and heart rate (HR), and EE.^12,16,32 However, these wearable devices exhibit errors ranging from 27% to 93% when evaluated using new subjects.^2,33 Furthermore, the overall correlation with indirect calorimetry was observed to be 0.34 for all monitors.²⁵ To improve prediction accuracy, researchers have studied new approaches involving machine learning (ML) and deep learning algorithms to develop regression models for monitoring walking EE. Beltrame et al.¹⁸ predicted the walking EE of healthy younger adults based on HR and treadmill ergometer inputs by artificial neural network (ANN); however, they relied on small-sample subject-specific data to train their models and did not evaluate the accuracy for new subjects. Unlike the manual design and selection of features in ML, deep learning algorithms can automatically extract deep features without professional knowledge.^15,22 Although convolutional neural network (CNN) and long short-term memory (LSTM) algorithms have been used to increase the prediction accuracy of walking EE in previous studies,^15,22,34 they rely on complex time-series signals such as electrocardiography (ECG) and electromyography (EMG) and cannot achieve convenient and large-scale monitoring of walking EE.

Therefore, the aim of this study was to develop an algorithm model with broad applicability, high accuracy, and sufficient utility for monitoring treadmill walking EE based on common and easily available information. Considering the potential burden of additional wearable devices for mental load and EE,³⁵ we further developed a completely contactless monitoring system based on cameras to monitor treadmill walking EE. This system can be combined with smart devices and/or treadmill equipment, which makes it suitable for evaluating physical health, monitoring safety, and tracking changes in aerobic fitness during exercise and rehabilitation training indoors.

Results

ML algorithm selection through comparison of predicted results

Figure 1 presents the selection process (Figures 1A and 1B) and prediction accuracy (Figure 1C) of ML algorithms. A detailed description of the ML model development and validation is presented in STAR Methods, and the information related to the dataset is presented in Table 1 and Figure 2. As shown in Figure 1C, the ANN model had the best predictive performance (R² = 0.90). Additionally, the root-mean-square error (RMSE) and mean square error (MSE) of the different ML algorithms for predicting walking EE are presented in the supplementary material (Table S1).

Figure 1.

Selection of machine learning algorithm

(A) Flowchart of machine learning model development and validation.

(B) Importance score and ranking of features based on max-relevance and min-redundancy algorithm (mRMR).

(C) Comparison of determinant coefficients (R²) of different ML models for predicting walking energy expenditure. BSA, body surface area; BMI, body mass index; NIHSS, National Institute of Health Stroke Scale; FMA, Fugl-Meyer Assessment Scale; LR, linear regression; GPR, Gaussian process regression; SVM, support vector machine; ANN, artificial neural network; DT, decision tree; BT, boosting tree.

Table 1.

Basic characteristics of study subjects

Characteristics	Healthy younger (n = 11)	Healthy older (n = 10)	Stroke (n = 14)
Age (years)	24.36 ± 2.42	64.6 ± 4.45	62.57 ± 9.86
Height (m)	1.69 ± 0.05	1.62 ± 0.07	1.66 ± 0.08
Mass (kg)	61.27 ± 7.20	61.39 ± 9.78	73.57 ± 7.91
Sex (female/male)	7/4	8/2	10/4
BSA (m2)	1.70 ± 0.12	1.65 ± 0.14	1.80 ± 0.12
BMI (kg/m2)	21.31 ± 1.83	23.35 ± 3.47	26.56 ± 2.72
Type of stroke (Isc/Hem)	–	–	9/5
Time post-stroke (months)	–	–	12.43 ± 8.32
NIHSS (score)	–	–	9.07 ± 3.97
FMA (score)	–	–	24.21 ± 6.33

Note: BSA, Body Surface Area; BMI, Body Mass Index; Isc, Ischemic stroke; Hem, Hemorrhagic stroke; NIHSS, National Institute of Health Stroke Scale; FMA, Fugl-Meyer Assessment Scale.

Figure 2.

Framework of data collection and analysis of energy expenditure during walking

(A) Overall diagram of data collection; wearable system consisting of a respirometer, inertial measurement units, and heart rate monitor; (B) protocol of data collection; the participants were instructed to stand for 2 min and then walk at three different constant speeds for 10 min; SSS: self-selected speed.

(C) Data process and extraction.

Development and performance of ensemble neural network model

Model architecture

An ensemble neural network (ENN) model is proposed to address the trade-off between population diversity and prediction accuracy within an evolutionary multi-objective framework and to achieve high-precision prediction of walking EE for various populations. Figure 3 shows the framework and basic block of the proposed ENN model, which includes modular for feature input, data processing, decision-making, ENN model, and output. The ENN model consists of three independent neural networks, each of which was developed based on the data from stroke survivors, older adults, and younger adults. The feature input consists of anthropometric data (i.e., age, height, mass, BMI, and body surface area [BSA]), functional disorder assessment (i.e., time post-stroke, National Institute of Health Stroke Scale, and Fugl-Meyer Assessment Scale), physiological data (HR), and waking speed. The data processing module consists of moving average filter and normalization function; the moving average filter was used to reduce random noise while maximizing the major signal for the data of HR and EE, and the normalization method was used to reduce adverse effects of outliers. The decision-making module determines whether there is a stroke and age label and guides the preprocessed input features into the corresponding sub-model. The output modular with a built-in linear transfer function was used to output and display the predicted walking EE. More detailed descriptions are provided in the STAR Methods.

Figure 3.

Framework and basic block of the proposed ensemble neural network (ENN) model

The overall framework consists of five modules: feature input, data processing, decision-making, ENN model, and output. The ENN model consists of three independent neural networks, trained using the Levenberg-Marquardt algorithm based on the data of stroke survivors and older and younger adults, respectively. The decision-making will read the label of subjects and input the data into the corresponding neural network model. ENN, ensemble neural network.

Model performance analysis using leave-one-subject-out cross-validation

The overall performance of our proposed model is presented using the correlation analysis and the Bland-Altman plots of the results of the leave-one-subject-out cross-validation (LOSOCV) (Figure 4). In the correlation plots, most of the points lay close to the dashed black line, indicating a significantly strong positive R value between the estimated and measured EE for each population (stroke survivors: r = 0.9651; older adults: r = 0.9763; younger adults: r = 0.9607; Figures 4A–4C). In the Bland-Altman plot, more than 95% of the points lay within the limit of agreement in EE estimation, and the bias was very small (stroke survivors: −0.007 kcal/min; older adults: −0.002 kcal/min; younger adults: −0.004 kcal/min; Figures 4D–4F), suggesting that the proposed model is highly accurate for EE estimation.

Figure 4.

Regression analysis and Bland-Altman plots of the predicted and measured energy expenditure during walking

(A–C) show the results of regression analysis of stroke survivors, older adults, and younger adults, respectively.

(D–F) show the Bland-Altman plots of predicted energy expenditure of stroke survivors, older adults, and younger adults, respectively. Solid horizontal line shows the bias of the prediction, and dashed horizontal lines represent the limit of agreement. Pre_EE, predicted EE by model; mse, mean squared error; CI, confidence interval.

Model performance analysis in testing data

Figure 5 shows the prediction performance of the proposed model in the testing data (15% of the samples were randomly selected and did not participate in model training and validation). The numbers of testing data points for stroke survivors and older and younger adults were 51, 40, and 47, respectively. As shown in Figures 5A–5C, the model exhibited high accuracy for predicting walking EE, which was indicated by the strong correlation between the predicted and measured EE (stroke survivors: r = 0.9302; older adults: r = 0.9521; younger adults: r = 0.9506). Additionally, Bland-Altman analysis indicated a small bias (stroke survivors: −0.049 kcal/min; older adults: 0.002 kcal/min; younger adults: 0.048 kcal/min; Figures 5D–5F).

Figure 5.

Performance assessment of the proposed model in testing data through regression and Bland-Altman analysis

(A–C) show the results of regression analysis of stroke survivors, older adults, and younger adults, respectively.

(D–F) show the 95% CI and bias of the difference EE of stroke survivors, older adults, and younger adults, respectively. The solid horizontal line shows the bias of the prediction, and the dashed horizontal lines represent the limit of agreement. Pre_EE, predicted EE by model; mse, mean squared error; CI, confidence interval.

Comparison with previous related studies and algorithms

Because previous related studies did not involve universal algorithms for three different populations, it is difficult to directly compare them with our study. To verify the advantage of our proposed ENN model in predicting the walking EE, we compared the prediction accuracies for stroke survivors and younger adults in previous studies.

Table 2 presents the results of the comparison with other regression algorithms for predicting the walking EE of stroke survivors, which included different regression equations (logarithm, polynomial, exponential) and deep learning algorithms. For comparison, the oxygen cost of walking (Cw, ml.kg⁻¹.m⁻¹) was converted to EE (kcal/min) by the Equation 1. Furthermore, Lopes et al.¹⁵ predicted the walking EE of stroke survivors using LSTM and CNN model, and the deep learning model relied on the time-sequence data processing and thus could not be reproduced on our dataset. Therefore, we converted only the results for comparison. Table 2 shows that the prediction accuracy of the ML algorithm was higher than taht of the traditional regression equation. Compared with the LSTM and CNN, the proposed ENN model reduced the EE estimation error by 13.95% in terms of RMSE.

$$EE\left(kcal.{\mathit{min}}^{-1}\right)=\frac{Cw\times {Sfree}^{-1}\times mass}{200}$$ (Equation 1)

Note: EE, energy expenditure; Sfree, spontaneous walking speed, m.s⁻¹; Cw, oxygen cost of walking at Sfree, ml.kg⁻¹.m⁻¹.

Table 2.

Comparison with other regression algorithms for predicting walking EE of stroke survivors

Reference	Methods	Equation	RMSE (kcal/min)
Zamparo et al.36	regression (logarithm)	$\ln \left(Cw\right)=1.27-0.718\times \ln \left(Sfree\right)$	7.67
Polese et al.37	regression (polynomial)	$Cw=0.95-1.28\times Sfree+0.47\times {Sfree}^2$	6.26
Compagnat et al.21	regression (exponential)	$Cw=0.2109\times {Sfree}^{-0.877}$	8.66
Lopes et al.15	deep learning	LSTM and CNN	0.43
This work	machine learning	ENN	0.37

Note: Sfree, spontaneous walking speed; Cw, oxygen cost of walking at Sfree; LSTM, long short-term memory network; CNN, convolutional neural network; ENN, ensemble neural network; RMSE, root-mean-square error.

Table 3 presents the results of the comparison with other ML algorithms for predicting the walking EE of younger adults. Among these algorithms, the deep multi-branch two-stage regression network (DMTRN) exhibited the best performance (RMSE = 0.71 kcal/min). The proposed model reduced the RMSE of the EE estimation by 66.20% compared with DMTRN.

Table 3.

Comparison with other machine learning algorithms for predicting walking EE of younger adults

Reference	Feature type	Method	RMSE (kcal/min)
Ni et al.22	anthropometric, IMU and ECG	DMTRN	0.71
Park et al.38	IMU and ECG	LR	1.05
Catal & Akbulut39	breathe and hearth rate	BDTR	1.22
Zhu et al.40	anthropometric, HR, accelerometer	CNN	1.12
This work	anthropometric, HR, speed	ENN	0.24

Note: IMU, inertial measurement unit; ECG, electrocardiogram; HR, heart rate; DMTRN, deep multi-branch two-stage regression network; LR, linear regression; BDTR, boosted decision tree regression; CNN, convolutional neural network; ENN, ensemble neural network; RMSE, root-mean-square error.

Contactless monitoring system for treadmill walking EE

Advancements in computers and artificial intelligence have enabled various camera-based measurements, which have improved contact-based monitoring solutions in various scenarios such as sports health and PA monitoring. In this study, we further established a contactless monitoring system using the ENN model for real-time measurement of walking EE. As shown in Figure 6, this monitoring system included a Kinect camera, mm-wave radar, computer, and treadmill. MediaPipe’s pose estimation algorithm was used to track the Kinect image and obtain the estimated speed information (Figures 6B and 6C). The mm-wave radar was used to estimate the HR, and standardized methods and signal processing were recommended by the official website and are shown in Figures 6B and 6D. The speed information and HR were combined with other input features according to the results of decision-making, entered into different neural network models, and used to predict walking EE in real time (Figure 3).

Figure 6.

Schematic of contactless monitoring system

(A) Contactless monitoring system includes treadmill, Kinect camera, mm-wave radar, and computer.

(B) Kinect camera used to obtain the walking speed, and mm-wave radar used to acquire the heart rate.

(C and D) show the flowchart of data processing of the Kinect camera and mm-wave radar, respectively.

(E) shows the software interface. FFT, fast Fourier transform.

To evaluate the performance of the proposed model and monitoring system in tracking walking EE changes, we provided the comparison results of measured EE and predicted EE of three subjects (stroke survivors, older adults, and younger adults) during 30-min treadmill walking (Figure 7). As shown in Figure 7, the proposed system can accurately track the changes of treadmill walking EE, which was indicated by the strong correlation and low MSE between the predicted and measured EE (stroke survivors: r = 0.9061, MSE = 0.0298 kcal/min; older adults: r = 0.9205, MSE = 0.0132 kcal/min; younger adults: r = 0.9264, MSE = 0.004 kcal/min; Figure 7).

Figure 7.

Tracking performance of contactless monitoring system during 30-min treadmill walking

(A) Stroke survivors.

(B) Older adults.

(C) Younger adults. EE, energy expenditure.

Discussion

Prediction algorithms for treadmill walking EE have been developed using small and homogeneous populations in previous studies, resulting in a lack of universality and potentially not meeting the requirements for monitoring.^18,23 This paper proposes an ENN model that addresses the trade-off between the prediction accuracy of walking EE and broad applicability in different population and can be used to predict treadmill walking EE with high precision for younger and older adults and stroke survivors. Additionally, this study further developed a completely contactless monitoring system for non-interference monitoring treadmill walking EE. This system provides a contactless assessment tool for walking EE that can be used to monitor exercise intensity and safety, evaluate health status, and track capability changes.

The complex and diverse factors that affected EE make real-time and accurate estimation challenging, particularly for prediction tasks involving different populations.^22,23 In this study, we observed that the single ML algorithm did not achieve satisfactory overall prediction results, including NN algorithms with nonlinear approximation capabilities (R² = 0.90; Figure 1C). A potential explanation is that the strong correlation relationship between the factors and EE in a single population disappears or is neutralized in most samples. Previous studies demonstrated that the walking speed is a strong predictor of EE,^36,37 but the healthy younger adults have significantly lower EE (3.50 ± 0.57 kcal/min) and faster walking speed (0.86 ± 0.13 m/s) compared with the stroke survivors (3.82 ± 0.20 kcal/min; 0.20 ± 0.08 m/s). This suggests that walking speed may not contribute significantly to overall EE prediction. The important score calculated using the max-relevance and min-redundancy (mRMR) algorithm proved this conjecture, and the importance score of walking speed was only 0.3043 (Figure 1B). In addition, the NN model had a relatively better prediction performance than other ML algorithms of different types. This may be related to the strong nonlinear function approximation and fault tolerance of the neural network, which further implied the nonlinearity of EE prediction.⁴¹

The concept of ensemble learning, which has better performance and generalization ability than the single ML, was proposed to address the trade-off between prediction accuracy and multi-objective learning.^42,43 In this study, we developed an ENN model for predicting walking EE that combines the outputs of three independently trained neural networks through stacking. The results of LOSOCV and testing data showed that the accuracy of the ENN model was indicated by a high linear correlation and low bias (Figures 4 and 5). We also conducted separate comparisons between the predictive accuracy of our proposed model in stroke survivors and young individuals and that of previous study, as previous studies did not utilize universally applicable algorithms for all three populations. The deep learning models proposed by Lopes et al.¹⁵ and Ni et al.²² (i.e., LSTM and CNN; DMTRN) achieved remarkable accuracy in predicting walking EE for stroke survivors and young individuals, respectively, by utilizing wearable sensors such as inertial measurement units (IMUs) and EMG for Lopes and IMUs and ECG for Ni. In comparison, our proposed model reduced the estimation error by 39.53% for stroke survivors and 73.24% for younger adults (Tables 2 and 3). In contrast to previous studies that involve processing complicated timing signals of sensors, the ENN model presented in this study achieves accurate prediction of walking EE for all three populations using only simple real-time inputs, such as walking speed and HR. This emphasizes the lightweight and user-friendly nature of our model in future application. It is important to note that the comparison of accuracy may not be entirely fair as our dataset does not include sensor data such as EMG and ECG. These types of data cannot be reproduced and can only be compared to our results through existing literature after normalization. In order to further compare the advantages of different algorithms, future research should include more real-time sensor sequence data.

Effective input feature selection may be key to the precision of the ENN model. For example, features related to disease characteristics and dysfunction were inputted into the model when predicting the walking EE of stroke survivors. A previous study reported that the oxygen cost of walking is related to the disability level in stroke survivors, with oxygen cost increasing sharply as disability became severe.^23,37 This provides inspiration for future research; that is, disease and functional information must be considered when predicting EE. In addition, an algorithm with the ability to capture nonlinear relationships is also a key factor affecting prediction accuracy.⁴¹ As presented in Table 2, heterogeneity existed in the predicted curves used (polynomial, exponential, and logarithmic) and in the criterion method for equation construction. More importantly, these regression equations based on the relationship between walking speed and oxygen cost performed poorly in terms of prediction. A recent system review demonstrated that these regression models may not be suitable for walking at low speed (0.3 m/s), which appear more like "trampling" and are strongly influenced by balance control strategies.²³ The self-selected walking speed of stroke survivors in the present study is average at 0.2 m/s, which may be the reason for the poor prediction performance of regression models in our dataset. However, the optimal function relationship is not clear, and various factors limit the practical application of traditional regression equations. Compared with traditional logistic regression, neural networks have prominent advantages and good performance in solving unknown and/or nonlinear questions. With technological advancements in wearable sensors, these algorithmic models will offer smarter solutions for walking EE in complex environment and have potential value for practical applications in health science and rehabilitation medicine.

A contactless monitoring system was developed based on Kinect and mm-wave radar equipped with ENN algorithms, which was used to efficiently monitor walking EE in real time. Our results of tracking performance demonstrated that the proposed monitoring system can accurately track changes in treadmill walking EE. It should be noted that the predictive performance of the entire monitoring system is lower than that of single ENN model (Figure 5 vs. Figure 7), which may include errors of the single in real-time acquisition (i.e., Kinect, mm-wave radar). This system has the advantages of non-interference, low cost, and high prediction accuracy and can be used as an alternative method to EE measurement. First, the acquisition of important features does not rely on complex wearable sensors and measurement systems, such as the remote estimation of HR through contactless mm-wave radar.⁴⁴ Second, the data acquisition hardware is low cost and highly convenient, with an estimated total cost of no more than $1000, which is far below the cost of respirometry system. Finally, the system accuracy is reflected in accurate estimation of physiological signals and high-precision prediction by the ENN model, and the accuracy of predicting the walking EE during steady state can be achieved based on easy-to-obtain inputs. In summary, the ENN algorithm and contactless system proposed in this paper can accurately and conveniently predict walking EE and can be combined with other training, treadmill, and rehabilitation systems, and we expect it to be widely used.

Limitations of the study

This study had several limitations. First, this model may not be suitable for stroke patients with cerebellar ataxia or bilateral paralysis because model training was based solely on hemiplegic data. Second, the present model also suffers from the “black box” problem and lacks interpretability. Although we can compute weights for each feature, the nonlinear mapping makes interpretability challenging.⁴¹ Third, we verify the overall tracking performance of the contactless monitoring system for walking EE but did not verify the estimation accuracy of HR and walking speed; this is because previous studies had fully demonstrated the accuracy for mm-radar in estimating HR and Kinect in capturing speed information.^44,45 However, this has a negative impact on the prediction accuracy of the walking EE. Last, while we have successfully achieved the study goal of providing a high-precision ENN model and contactless system for monitoring EE during indoor walking, it is important to note that the dataset is derived from laboratory treadmill settings with wearable sensors. Several factors may affect the ground-truth measures of walking EE, including treadmill walking and tight configuration of wearable sensors, and this model and system may not be directly applicable to free-living environment. Future studies are required to develop prediction models based on large samples and interpretable optimization algorithms and identify the optimal sensor information for predicting walking EE by comparing the differences of algorithm frameworks and model performance based on the wearables and contactless sensors, which may further promote intelligent monitoring of walking EE during exercise, rehabilitation training, and free-living environment.

Conclusions

This paper proposed an ENN model that achieves a high-precision prediction of EE during treadmill walking for younger and older adults and stroke survivors based on easy-to-obtain features. Compared with previous studies, the proposed model reduced the estimation error by 13.95% and 66.20% for stroke survivors and younger adults, respectively. Furthermore, a contactless monitoring system was developed based on Kinect and mm-wave radar, equipped with ENN algorithms, and the treadmill walking EE was monitored in real time.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
Matlab R2022a	MathWorks	https://ww2.mathworks.cn/products/matlab.html
Statistics and Machine Learning Toolbox v12.3	MathWorks	https://ww2.mathworks.cn/products/statistics.html
PyCharm v2021.1	PyCharm	https://www.jetbrains.com/pycharm/download/#section=windows
PyQt5 v5.15.9	PyQt5	https://pypi.org/project/PyQt5/#files
Python v3.9	Python	https://www.python.org/
Numpy v1.21.6	Python	https://numpy.org/
Pyqtgraph v0.12.4	PyQt	http://www.pyqtgraph.org
Opencv2 v4.6.0.66	OpenCV	https://opencv.org/releases/
MediaPipe Pose v0.8.10	Google	https://github.com/google/mediapipe
DriverVitalSignsLab	Texas Instruments	https://www.ti.com/tool/MMWAVE-SDK#related-design-resources
Other
Dataset, machine learning model and evaluation results	This paper	https://data.mendeley.com/datasets/mrmfyt2mv5/draft?a=c58f5a8d-7ba4-4d32-8944-002a540b3ace
Portable metabolic system K5	COSMED	https://www.cosmed.com/en/products/indirect-calorimetry
MTw Awinda	Xsens	https://www.xsens.cn/mtw-awinda/
Split-belt treadmill	Bertec	https://www.bertec.com/products/instrumented-treadmills
Azure Kinect	Microsoft	https://azure.microsoft.com/zh-cn/products/kinect-dk/
AWR1642BOOST	Texas Instruments	https://www.ti.com.cn/tool/cn/AWR1642BOOST

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Wenxin Niu (niu@tongji.edu.cn).

Materials availability

The study did not generate new materials.

Data and code availability

• •The dataset, machine learning model and all the other results have been deposited at Mendeley Data and are publicly available as of the date of publication. Links are listed in the key resources table.
• •All original codes involved in this paper will be shared by the lead contact upon request.
• •Any additional information required to reanalyze the data reported in this protocol is available from the lead contact upon request.

Experimental model and study participant details

All experiments and procedures were approved by the institutional review board of Tongji University (SBKT-2022-010), and informed consent was obtained from all enrolled participants. Asian healthy younger adults (n = 11, 7 women), healthy older adults (n = 10, 8 women), and stroke survivors (n = 14, 10 women) were recruited from the Shanghai University of Sport, surrounding communities, and Shanghai Seventh People’s Hospital, respectively. The participants’ basic characteristics are listed in Table 1.

Healthy participants without disabilities or chronic diseases were required to walk on a treadmill for sufficient time to complete the metabolic analysis. The inclusion criteria of the participants with stroke were as follows: (1) clinical diagnosis of cerebral hemorrhage or infarction through CT/MRI; (2) 60–80 years of age; (3) ≥ six months post stroke, and first stroke; (4) without muscle spasticity (the score of modified Ashworth scale < 1);⁴⁶ (5) ability to walk on a treadmill independently without assistance for 30 min to complete the metabolic analysis; (6) without cognitive disorder (Mini-Mental State Examination, MMSE >24).⁴⁷

Method details

Data collection and preprocess

The anthropometric data (e.g., height, mass) of each participant were collected before biomechanics measurements. Additionally, the stroke severity (National Institute of Health Stroke Scale, NIHSS) and lower limb function (Fugl-Meyer Assessment, FMA) of the participants with stroke were evaluated by an experienced physiotherapist.^48,49 The results of clinical assessments are shown in Table 1.

Figure 2A presents the overall schematic diagram of the participants wearing experimental devices, including portable metabolic system, HR monitor and IMUs. Participants wore loose-fitting shorts and walking shoes. The portable metabolic system (K5, COSMED, Rome, Italy) was used to measure the oxygen consumption (VO₂) and carbon dioxide production (VCO₂) per breath and calculated gross EE during walking using the Brockway equation;⁵⁰ The HR monitor (REF A-661-200-001, 2.4 GHz, Garmin, China) fixed in the chest was used to continuously monitor the HR in beats per minute at the same intervals as the respirometry measurements.⁵⁰ The full-body inertial motion tracking system (MTw Awinda, Xsens Technologies, Enschede, The Netherlands) was used to collect the kinematics of walking at a sampling frequency of 100 Hz⁵¹ According to manufacturer’s instructions of sensor placement, 17 IMU sensors placing head, both shoulder, sternum, upper arms, forearms, wrist, pelvis, upper leg, lower leg, and feet, respectively.⁵² The orientation of each sensor will align with the orientation of each segment of the Xsens biomechanical model when performing the calibration. Additionally, the experimental devices and sensor were performed by the same researchers, ensuring repeatability in the instrumentation procedure.

Figure 2B shows the test protocol of data collection. To begin, Martin's suggested approach was used to ascertain the preferred walking speed (PWS) for each participant.⁵³ Subsequently, slow and fast walking speeds were determined by adjusting by ±10%. Previous study suggested that under the condition of non-fatigue, the longer the duration, the more representative it is of the energy expenditure level in a steady state.^17,23 During participant recruitment, the assessor has determined that the majority of participants were able to complete a 30-min walk, by the asked the participants, "How long can you usually walk?". Therefore, the walking time for the participants in this study was set at 30-min. During formal testing, the participants were instructed to stand statically for 2 min, then walk on a split-belt treadmill (Bertec Corporation, Columbus, OH) at each walking speed for 10-min, a total of 30-min. The testing order for the three walking speeds of the participants was randomly determined. There was a 20-minute break between each trial to allow for recovery of EE and HR to baseline before proceeding with the next speed test, in order to prevent participant fatigue. During the laboratory experiment, the conditions of the environmental temperature and relative humidity were similar: 22.06±0.96, 53.62±2.96, respectively.

Figure 2C shows a diagram of the data processing and extraction for model training proposed in this paper. Data processing was performed using a custom code based on MATLAB R2022a (MathWorks, Natick, MS, USA). All data sources (e.g., EE, HR, and walking speed) were aggregated to the minute level using the sliding window methods because previous studies demonstrated that the walking EE fluctuated slightly in one minute, and the longer the sliding window used, the smaller the EE estimation error.^22,34 If data points were missing, the single minute was excluded from the analysis, and a total of 812 single-minute samples were obtained. Data preprocessing is a necessary step before commencing the development of machine learning algorithm models. The moving average filter with window length 3 was used to reduce random noise while maximizing retain the major signal for the data of HR and EE.⁵⁴ Then, map the features minimum and maximum values to [-1 1] using the normalization method to reduce adverse effects of outliers.⁵⁵

Machine learning algorithm selection

In this section, we describe the process of selecting the ML algorithm (Figure 1). This is because previous studies have observed that the choice of ML algorithm is important for estimating the EE, particularly for prediction tasks with unknown optimal and complex function relationship. The ML model was developed and evaluated using Statistics and Machine Learning Toolbox 12.3 provided by MATLAB R2022a (MathWorks, Natick, MS, USA). The following algorithms that were used in this study are well-known and representative of different types of ML techniques: linear regression (LR), gaussian process regression (GPR), support vector machines (SVMs), ANN, decision trees (DT) and boosting trees (BT).

The mRMR method, which maintains features with high relevance to class labels and low redundancies to other features, was adopted for feature selection.⁵⁶ Figure 1A shows the alternative feature inputs, the importance scores and list of features filtered by the mRMR algorithm are present in Figure 1B. We reserved the features with an importance score greater than 0.3 as final data input, consists of age, height, mass, BMI, BSA, time post-stroke, NIHSS score, FMA score, walking speed and HR. The stratified sampling method was employed to split dataset, considering the group and sex factor, removing 30% of the total samples to be used as testing data, while the remaining 70% were designated as training data.⁵⁷ All developed algorithm models were trained using a training set with LOSOCV.¹⁸ To compare the prediction accuracy of each model, we calculated the RMSE, MSE, and coefficient of determination (R²) of the testing data, and the best model was selected based on the maximal R². All ML model and performance results were provided in the Supplementary material (Table S1).

ENN model development and evaluation

In the ML algorithm section, the R² of the neural network model for EE prediction based on 812 samples from three groups (i.e., younger, older and stroke survivors) was only 0.90. The low prediction accuracy may be related to the different and complex relationships between the input features and EE in each group. The utilization of an ensemble obtains higher accuracy than other neural network approaches and can address the trade-off between prediction diversity and accuracy within an evolutionary multi-objective framework.^42,43 Therefore, an ENN model is proposed in this paper to achieve a high-precision prediction of walking EE for different populations.

The ENN model combines the outputs of independently trained neural networks. In this study, the proposed ENN model comprises three independent neural networks, and each neural network was developed based on data from stroke survivors (291 samples), older adults (238 samples), and younger adults (283 samples). The multilayer perceptron regressor (MLPR) was selected as the basic form of the model because it is a single-hidden-layer neural network that can generate simpler algorithms with greater applicability for embedded systems.⁴² In order to maximize the consistency of data distribution while taking into account variables such as sex, BMI, and functional level, the stratified sampling method was employed to split data for each population. This involved removing 15% of the total samples to be used as testing data, while the remaining 85% were designated as training data to construct the MLPR. The data preprocessing and models’ input as same as described in the section of “Data collection and preprocess” and “Machine learning algorithm selection”, respectively. Each of these independent neural networks with Sigmoid activation function was trained using the Levenberg–Marquardt algorithm. The optimization process started with a single neuron, and then one neuron was cyclically added into the neural networks. Additionally, to enable the selection of the best neural network, 10000 epochs, L2 regularization 0.001, Validation checks 6 and an MSE of 10^-5 were set to guarantee that the model had a favorable fit. The parameters and initial conditions of model training were provided in the Supplementary material (Table S2). The model's performance (MSE) was evaluated using the LOSOCV method, which helped determine the NN model with the optimal number of hidden neurons and the lowest MSE.⁵⁸ Finally, each model with the best performance was combined into an ENN model via stacking.

Development protocol of contactless monitoring system

The Azure Kinect is an RGB-D camera that combines a best-in-class 1 MP depth camera, a 12 MP RGB camera, and an orientation sensor for building advanced computer vision. A recent study confirmed the Microsoft officially stated values, namely standard deviation ≤ 17 mm; and distance error < 11 mm in up to 3.5 meters distance from the Azure Kinect.^45,59 MediaPipe Pose is an ML solution for high-fidelity body pose tracking that infers 33 3D landmarks and a background segmentation mask for the entire body from RGB video frames.⁶⁰ In this study, we applied MediaPipe Pose estimation and tracked the Kinect RGB image to obtain the estimated joint coordinates. The Kinect camera was operated at 1 fps and was placed 2 m high in front of the treadmill. The experimental setup used in this study is shown in Figure 6A.

Radars are widely used for the remote sensing of physiological signals, such as breathing and HR, in an easier and more comfortable manner than contact-based and wearable devices.⁶¹ The millimeters-wave frequency-modulated wave radars produced by Texas Instruments (AWR1642), which have been shown to measure HR with high accuracy.⁴⁴ Compared with contact-based sensors (e.g., airflow sensor, smart bracelet Mi 3) and electrocardiograms, the accuracy of HR measurement via radar is up to 93% and > 80%, respectively.⁶² The AWR1642BOOST integrates RF/analog system, radio processor subsystem, data processing system, and main control system, operates in 76-81 GHz band, and can detect heart rate waveforms with a sampling rate of 20 Hz. A micro-USB cable was included in the AWR1642 package to communicate with the computer and a 5 V/2.5 A power cable was used to power the sensor. As shown in Figure 6, the signal processing included four main steps: target detection, phase extraction, signal extraction, and HR estimation. Standardized methods and codes for HR estimation are recommended by the official Texas Instruments website.

The contactless monitoring system developed in this study is based on Microsoft Windows 10 operating system. Prior to developing the software of contactless monitoring system, it is important to configure the necessary software and SDK for the hardware (i.e., Azure Kinect, AWR1642) in accordance with the official recommendations. Once the debugging process is complete, proceed with configuring the system development environment. PyCharm Community was used to development, due to its convenience in deploying and managing virtual environments, it also provides various integrated development environments (IDE) and complete set of tools for code development. PyQt5 v5.15.9 is an IDE that utilizes a Python implementation, and used to develop Graphical User Interface (GUI). The essential Python packages for this developing environment including Python v3.8, Opencv2 v4.6.0.66 (package for image processing), Numpy v1.21.6 (package for scientific computing), DriverVitalSignsLab (package for radar data), mediapipe v0.8.10 (package for recognition of human body movements and postures) and pyqtgraph v0.12.4 (package for visualization). Detailed instructions on resource package and identifier are provided in the key resources table.

Three participants (stroke survivors, older and younger adults) were re-recruited, and used to evaluate the tracking performance of the proposed monitoring system. The participants’ basic characteristics are listed in Table S3. The participants were required to walking treadmill at self-selected speed for 30-min, the portable metabolic system and our proposed contactless monitoring system was used to collect walking EE at the same time. The coefficient (r) and MSE between the predicted EE by contactless monitoring system and the measured EE by portable metabolic system was used to evaluate the tracking performance.

Quantification and statistical analysis

To assess the model performance, the Matlab custom code was used to calculate the correlation coefficient (r) between the estimated EE of the ML and ENN model and the measured EE.

Published: February 2, 2024